2025-03-14T05:31:28.8769305Z Current runner version: '2.322.0' 2025-03-14T05:31:28.8775597Z Runner name: 'i-0166a710cfefd3e7e' 2025-03-14T05:31:28.8776506Z Runner group name: 'Default' 2025-03-14T05:31:28.8777552Z Machine name: 'ip-10-0-17-58' 2025-03-14T05:31:28.8782538Z ##[group]GITHUB_TOKEN Permissions 2025-03-14T05:31:28.8785226Z Actions: read 2025-03-14T05:31:28.8785904Z Attestations: read 2025-03-14T05:31:28.8786563Z Checks: read 2025-03-14T05:31:28.8787344Z Contents: read 2025-03-14T05:31:28.8787992Z Deployments: read 2025-03-14T05:31:28.8788654Z Discussions: read 2025-03-14T05:31:28.8789293Z Issues: read 2025-03-14T05:31:28.8789922Z Metadata: read 2025-03-14T05:31:28.8790565Z Packages: read 2025-03-14T05:31:28.8791209Z Pages: read 2025-03-14T05:31:28.8791853Z PullRequests: read 2025-03-14T05:31:28.8792546Z RepositoryProjects: read 2025-03-14T05:31:28.8793287Z SecurityEvents: read 2025-03-14T05:31:28.8793981Z Statuses: read 2025-03-14T05:31:28.8794630Z ##[endgroup] 2025-03-14T05:31:28.8797862Z Secret source: Actions 2025-03-14T05:31:28.8798733Z Prepare workflow directory 2025-03-14T05:31:28.9447406Z Prepare all required actions 2025-03-14T05:31:28.9486285Z Getting action download info 2025-03-14T05:31:29.2209809Z Download action repository 'pytorch/test-infra@main' (SHA:de00dac6adc071cb2f9861380a0ed3947b93e5cc) 2025-03-14T05:31:30.8254250Z Download action repository 'pytorch/pytorch@main' (SHA:d4496346b901e9a4c3993bf6b2054014c7c0b731) 2025-03-14T05:31:45.0073081Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2025-03-14T05:31:45.1975109Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-03-14T05:31:45.4974062Z Getting action download info 2025-03-14T05:31:45.6208049Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683) 2025-03-14T05:31:45.8645109Z Getting action download info 2025-03-14T05:31:45.9595039Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-03-14T05:31:46.1553818Z Getting action download info 2025-03-14T05:31:46.2648017Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-03-14T05:31:46.4304488Z Getting action download info 2025-03-14T05:31:46.5539663Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (aed0b7a742a2d7b7901790622829cbd2135049a4) 2025-03-14T05:31:46.5541971Z ##[group] Inputs 2025-03-14T05:31:46.5542360Z build-environment: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:31:46.5544235Z test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2025-03-14T05:31:46.5546482Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:31:46.5547423Z sync-tag: 2025-03-14T05:31:46.5548231Z timeout-minutes: 240 2025-03-14T05:31:46.5548513Z use-gha: 2025-03-14T05:31:46.5548753Z dashboard-tag: 2025-03-14T05:31:46.5549018Z s3-bucket: gha-artifacts 2025-03-14T05:31:46.5549323Z aws-role-to-assume: 2025-03-14T05:31:46.5550029Z disable-monitor: false 2025-03-14T05:31:46.5550343Z ##[endgroup] 2025-03-14T05:31:46.5550887Z Complete job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:31:46.6112123Z A job started hook has been configured by the self-hosted runner administrator 2025-03-14T05:31:46.6210799Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-03-14T05:31:46.6222654Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:31:46.6223393Z ##[endgroup] 2025-03-14T05:31:48.8565734Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2025-03-14T05:31:48.8566249Z Instance Type: g5.4xlarge 2025-03-14T05:31:48.8566537Z AMI Name: unknown 2025-03-14T05:31:48.8604247Z AMI ID: ami-08b5b3a93ed654d19 2025-03-14T05:31:54.2295740Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-03-14T05:31:54.2296176Z with: 2025-03-14T05:31:54.2296785Z github-secret: *** 2025-03-14T05:31:54.2297470Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-03-14T05:31:54.2298196Z activate-with-label: false 2025-03-14T05:31:54.2298481Z label: with-ssh 2025-03-14T05:31:54.2298750Z remove-existing-keys: true 2025-03-14T05:31:54.2299045Z fail-silently: true 2025-03-14T05:31:54.2299295Z env: 2025-03-14T05:31:54.2299562Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:31:54.2299868Z ##[endgroup] 2025-03-14T05:31:54.3509641Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-03-14T05:31:54.3511329Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-03-14T05:31:54.3682482Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-03-14T05:31:54.3682926Z with: 2025-03-14T05:31:54.3683161Z no-sudo: true 2025-03-14T05:31:54.3683415Z submodules: recursive 2025-03-14T05:31:54.3683704Z fetch-depth: 0 2025-03-14T05:31:54.3683939Z env: 2025-03-14T05:31:54.3684178Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:31:54.3684469Z ##[endgroup] 2025-03-14T05:31:54.3788598Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:31:54.3789541Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:31:54.3800129Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:31:54.3800523Z env: 2025-03-14T05:31:54.3800763Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:31:54.3801051Z ##[endgroup] 2025-03-14T05:31:54.3903501Z ##[group]Run # Use all available CPUs for fetching 2025-03-14T05:31:54.3903957Z # Use all available CPUs for fetching 2025-03-14T05:31:54.3904350Z cd "${GITHUB_WORKSPACE}" 2025-03-14T05:31:54.3904748Z git config --global fetch.parallel 0 2025-03-14T05:31:54.3905159Z git config --global submodule.fetchJobs 0 2025-03-14T05:31:54.3905520Z  2025-03-14T05:31:54.3905910Z # Clean workspace. The default checkout action should also do this, but 2025-03-14T05:31:54.3906498Z # do it here as well just in case 2025-03-14T05:31:54.3906870Z if [[ -d .git ]]; then 2025-03-14T05:31:54.3907186Z  if [ -z "${NO_SUDO}" ]; then 2025-03-14T05:31:54.3907537Z  sudo git clean -ffdx 2025-03-14T05:31:54.3907856Z  else 2025-03-14T05:31:54.3908116Z  git clean -ffdx 2025-03-14T05:31:54.3908408Z  fi 2025-03-14T05:31:54.3908651Z fi 2025-03-14T05:31:54.3917408Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:31:54.3917798Z env: 2025-03-14T05:31:54.3918109Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:31:54.3918411Z NO_SUDO: true 2025-03-14T05:31:54.3918660Z ##[endgroup] 2025-03-14T05:31:54.6984995Z Removing .additional_ci_files/ 2025-03-14T05:31:54.6985367Z Removing .pytest_cache/ 2025-03-14T05:31:54.6985639Z Removing build/ 2025-03-14T05:31:54.6985887Z Removing dist/ 2025-03-14T05:31:54.6986304Z Removing logs-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_38751144889.zip 2025-03-14T05:31:54.6987045Z Removing test-jsons-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_38751144889.zip 2025-03-14T05:31:54.6987747Z Removing test-reports-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_38751144889.zip 2025-03-14T05:31:54.6988859Z Removing test/.pytorch-disabled-tests.json 2025-03-14T05:31:54.6989238Z Removing test/__pycache__/ 2025-03-14T05:31:54.6989557Z Removing test/inductor/__pycache__/ 2025-03-14T05:31:54.6989901Z Removing test/test-reports/ 2025-03-14T05:31:54.6990237Z Removing test/torch_compile_debug/ 2025-03-14T05:31:54.6990576Z Removing tools/__pycache__/ 2025-03-14T05:31:54.6990888Z Removing tools/stats/__pycache__/ 2025-03-14T05:31:54.6991321Z Removing tools/stats/upload_utilization_stats/__pycache__/ 2025-03-14T05:31:54.6991768Z Removing tools/testing/__pycache__/ 2025-03-14T05:31:54.6992191Z Removing tools/testing/target_determination/__pycache__/ 2025-03-14T05:31:54.6992749Z Removing tools/testing/target_determination/heuristics/__pycache__/ 2025-03-14T05:31:54.6993225Z Removing usage_log.txt 2025-03-14T05:31:54.7070257Z ##[group]Run actions/checkout@v4 2025-03-14T05:31:54.7070569Z with: 2025-03-14T05:31:54.7070837Z ref: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:31:54.7071195Z fetch-depth: 0 2025-03-14T05:31:54.7071456Z submodules: recursive 2025-03-14T05:31:54.7071741Z show-progress: false 2025-03-14T05:31:54.7072032Z repository: pytorch/pytorch 2025-03-14T05:31:54.7072420Z token: *** 2025-03-14T05:31:54.7072663Z ssh-strict: true 2025-03-14T05:31:54.7073145Z ssh-user: git 2025-03-14T05:31:54.7073412Z persist-credentials: true 2025-03-14T05:31:54.7073699Z clean: true 2025-03-14T05:31:54.7073967Z sparse-checkout-cone-mode: true 2025-03-14T05:31:54.7074270Z fetch-tags: false 2025-03-14T05:31:54.7074522Z lfs: false 2025-03-14T05:31:54.7074773Z set-safe-directory: true 2025-03-14T05:31:54.7075064Z env: 2025-03-14T05:31:54.7075299Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:31:54.7075581Z ##[endgroup] 2025-03-14T05:31:54.8194389Z Syncing repository: pytorch/pytorch 2025-03-14T05:31:54.8195598Z ##[group]Getting Git version info 2025-03-14T05:31:54.8196070Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-03-14T05:31:54.8196755Z [command]/usr/bin/git version 2025-03-14T05:31:54.8197084Z git version 2.47.1 2025-03-14T05:31:54.8220488Z ##[endgroup] 2025-03-14T05:31:54.8231338Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/b893d3fd-c9a4-4ab9-a5a0-224d5a7d531d/.gitconfig' 2025-03-14T05:31:54.8252367Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b893d3fd-c9a4-4ab9-a5a0-224d5a7d531d' before making global git config changes 2025-03-14T05:31:54.8253293Z Adding repository directory to the temporary git global config as a safe directory 2025-03-14T05:31:54.8257063Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T05:31:54.8302543Z [command]/usr/bin/git config --local --get remote.origin.url 2025-03-14T05:31:54.8327420Z https://github.com/pytorch/pytorch 2025-03-14T05:31:54.8345599Z ##[group]Removing previously created refs, to avoid conflicts 2025-03-14T05:31:54.8349477Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD 2025-03-14T05:31:54.8375698Z HEAD 2025-03-14T05:31:54.8419648Z ##[endgroup] 2025-03-14T05:31:54.8421735Z [command]/usr/bin/git submodule status 2025-03-14T05:31:54.8826883Z 7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe) 2025-03-14T05:31:54.8922760Z 4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081) 2025-03-14T05:31:54.9019939Z b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327) 2025-03-14T05:31:54.9133433Z c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0) 2025-03-14T05:31:54.9162697Z e170594ac7cf1dac584da473d4ca9301087090c1 third_party/NVTX (v3.1.0) 2025-03-14T05:31:54.9276028Z a6bfc237255a6bac1513f7c1ebde6d8aed6b5191 third_party/VulkanMemoryAllocator (v2.1.0-705-ga6bfc23) 2025-03-14T05:31:54.9845139Z 51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a010365) 2025-03-14T05:31:54.9875912Z 0d98dba29d66e93259db7daa53a9327df767a415 third_party/benchmark (v1.6.1) 2025-03-14T05:31:54.9912779Z 8086bbe3a78d931eb96fe12fdc014082e18d18d3 third_party/composable_kernel (mock-tag-test-6-g8086bbe3a) 2025-03-14T05:31:55.0068151Z 3b6597bba913d51161383657829b7e644e59c006 third_party/cpp-httplib (v0.15.3-20-g3b6597b) 2025-03-14T05:31:55.0207025Z 1e83a2fdd3102f65c6f1fb602c1b320486218a99 third_party/cpuinfo (1e83a2f) 2025-03-14T05:31:55.0254661Z 91b7532f3386768bba4f444ee7672b497f34da8a third_party/cudnn_frontend (v0.5-44-g91b7532) 2025-03-14T05:31:55.0362801Z afa1772203677c5118fcd82537a9c8fefbcc7008 third_party/cutlass (v3.8.0) 2025-03-14T05:31:55.1159451Z 3147391d946bb4b6c68edd901f2add6ac1f31f8c third_party/eigen (3.4.0) 2025-03-14T05:31:55.1541414Z dbc3157bf256f1339b3fa1fef2be89ac4078be0e third_party/fbgemm (v0.4.1-446-gdbc3157b) 2025-03-14T05:31:55.1644133Z 979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4) 2025-03-14T05:31:55.1674713Z 01834de25e4bf3975a9a00e816292b1ad0fe184b third_party/flatbuffers (v23.3.3) 2025-03-14T05:31:55.2146525Z 123913715afeb8a437e6388b4473fcc4753e1c9a third_party/fmt (11.1.4) 2025-03-14T05:31:55.2277372Z 3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17) 2025-03-14T05:31:55.2415456Z 5354032ea08eadd7fc4456477f7f7c6308818509 third_party/gloo (5354032) 2025-03-14T05:31:55.2703054Z b514bdc898e2951020cbdca1304b75f5950d1f59 third_party/googletest (release-1.8.0-3484-gb514bdc8) 2025-03-14T05:31:55.2805818Z 719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1) 2025-03-14T05:31:55.2880966Z 5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42 third_party/ittapi (v3.23.0-14-g5b8a7d7) 2025-03-14T05:31:55.3172246Z 2859721fd9e73d3ca1c56f827dbc64e6d68f78a2 third_party/kineto (remotes/origin/sraikund/test-53-g2859721) 2025-03-14T05:31:55.3204542Z ef685a13cfbe8d418aa2ed34350e21e4938358b6 third_party/kleidiai (v1.3.0) 2025-03-14T05:31:55.3235784Z b66e3214d8a104669c2ec05ae91ebc26a8f5ab78 third_party/mimalloc (v1.8.2) 2025-03-14T05:31:55.3778526Z 87cda1d6646592ac5866dc703c8e1839046a6806 third_party/nlohmann (v3.10.1-113-g87cda1d6) 2025-03-14T05:31:55.4125456Z b8baa8446686496da4cc8fda09f2b6fe65c2a02c third_party/onnx (v1.17.0) 2025-03-14T05:31:55.4159220Z a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2) 2025-03-14T05:31:55.4190191Z 9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe third_party/pocketfft (release_for_eigen-11-g9d3ab05) 2025-03-14T05:31:55.4637862Z d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b) 2025-03-14T05:31:55.4740328Z 072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master) 2025-03-14T05:31:55.4815792Z 4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e) 2025-03-14T05:31:55.4877249Z a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e) 2025-03-14T05:31:55.4970544Z f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated) 2025-03-14T05:31:55.5099492Z 56e1f79cb140fb9326d612d0be06b5250565cade third_party/sleef (3.7-33-g56e1f79) 2025-03-14T05:31:55.5189436Z 52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master) 2025-03-14T05:31:55.5205593Z ##[group]Cleaning the repository 2025-03-14T05:31:55.5210150Z [command]/usr/bin/git clean -ffdx 2025-03-14T05:31:55.5445436Z [command]/usr/bin/git reset --hard HEAD 2025-03-14T05:31:56.6073089Z Updating files: 58% (9674/16540) 2025-03-14T05:31:56.6163668Z Updating files: 59% (9759/16540) 2025-03-14T05:31:56.6254853Z Updating files: 60% (9924/16540) 2025-03-14T05:31:56.6347235Z Updating files: 61% (10090/16540) 2025-03-14T05:31:56.6440225Z Updating files: 62% (10255/16540) 2025-03-14T05:31:56.6530984Z Updating files: 63% (10421/16540) 2025-03-14T05:31:56.6621053Z Updating files: 64% (10586/16540) 2025-03-14T05:31:56.6730544Z Updating files: 65% (10751/16540) 2025-03-14T05:31:56.7018264Z Updating files: 66% (10917/16540) 2025-03-14T05:31:56.7213953Z Updating files: 67% (11082/16540) 2025-03-14T05:31:56.7296259Z Updating files: 68% (11248/16540) 2025-03-14T05:31:56.7393231Z Updating files: 69% (11413/16540) 2025-03-14T05:31:56.7594721Z Updating files: 70% (11578/16540) 2025-03-14T05:31:56.7904328Z Updating files: 71% (11744/16540) 2025-03-14T05:31:56.8215828Z Updating files: 72% (11909/16540) 2025-03-14T05:31:56.8607026Z Updating files: 73% (12075/16540) 2025-03-14T05:31:56.8780013Z Updating files: 74% (12240/16540) 2025-03-14T05:31:56.8940768Z Updating files: 75% (12405/16540) 2025-03-14T05:31:56.9210059Z Updating files: 76% (12571/16540) 2025-03-14T05:31:56.9486494Z Updating files: 77% (12736/16540) 2025-03-14T05:31:56.9799736Z Updating files: 78% (12902/16540) 2025-03-14T05:31:57.0068013Z Updating files: 79% (13067/16540) 2025-03-14T05:31:57.0330985Z Updating files: 80% (13232/16540) 2025-03-14T05:31:57.0498735Z Updating files: 81% (13398/16540) 2025-03-14T05:31:57.0679364Z Updating files: 82% (13563/16540) 2025-03-14T05:31:57.0878414Z Updating files: 83% (13729/16540) 2025-03-14T05:31:57.1074496Z Updating files: 84% (13894/16540) 2025-03-14T05:31:57.1240883Z Updating files: 85% (14059/16540) 2025-03-14T05:31:57.1425710Z Updating files: 86% (14225/16540) 2025-03-14T05:31:57.1591839Z Updating files: 87% (14390/16540) 2025-03-14T05:31:57.1788138Z Updating files: 88% (14556/16540) 2025-03-14T05:31:57.2011674Z Updating files: 89% (14721/16540) 2025-03-14T05:31:57.2177130Z Updating files: 90% (14886/16540) 2025-03-14T05:31:57.2355758Z Updating files: 91% (15052/16540) 2025-03-14T05:31:57.2560336Z Updating files: 92% (15217/16540) 2025-03-14T05:31:57.2796183Z Updating files: 93% (15383/16540) 2025-03-14T05:31:57.3041129Z Updating files: 94% (15548/16540) 2025-03-14T05:31:57.3250142Z Updating files: 95% (15713/16540) 2025-03-14T05:31:57.3434632Z Updating files: 96% (15879/16540) 2025-03-14T05:31:57.3773813Z Updating files: 97% (16044/16540) 2025-03-14T05:31:57.3983115Z Updating files: 98% (16210/16540) 2025-03-14T05:31:57.4188800Z Updating files: 99% (16375/16540) 2025-03-14T05:31:57.4189252Z Updating files: 100% (16540/16540) 2025-03-14T05:31:57.4189667Z Updating files: 100% (16540/16540), done. 2025-03-14T05:31:57.4366494Z HEAD is now at 268fb634bb4 Update on "[dynamo] fix bug where non-recursive disable modifies the original function" 2025-03-14T05:31:57.4504495Z ##[endgroup] 2025-03-14T05:31:57.4506581Z ##[group]Disabling automatic garbage collection 2025-03-14T05:31:57.4511977Z [command]/usr/bin/git config --local gc.auto 0 2025-03-14T05:31:57.4547911Z ##[endgroup] 2025-03-14T05:31:57.4548681Z ##[group]Setting up auth 2025-03-14T05:31:57.4554133Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-03-14T05:31:57.4587741Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-03-14T05:31:57.4948531Z Entering 'android/libs/fbjni' 2025-03-14T05:31:57.5017015Z Entering 'third_party/FP16' 2025-03-14T05:31:57.5086317Z Entering 'third_party/FXdiv' 2025-03-14T05:31:57.5155756Z Entering 'third_party/NNPACK' 2025-03-14T05:31:57.5226109Z Entering 'third_party/NVTX' 2025-03-14T05:31:57.5294682Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:31:57.5363173Z Entering 'third_party/XNNPACK' 2025-03-14T05:31:57.5447352Z Entering 'third_party/benchmark' 2025-03-14T05:31:57.5515694Z Entering 'third_party/composable_kernel' 2025-03-14T05:31:57.5592855Z Entering 'third_party/cpp-httplib' 2025-03-14T05:31:57.5660886Z Entering 'third_party/cpuinfo' 2025-03-14T05:31:57.5733862Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:31:57.5803347Z Entering 'third_party/cutlass' 2025-03-14T05:31:57.5880316Z Entering 'third_party/eigen' 2025-03-14T05:31:57.5951189Z Entering 'third_party/fbgemm' 2025-03-14T05:31:57.6018874Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:31:57.6086367Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:31:57.6152702Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:31:57.6227182Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:31:57.6293598Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:31:57.6364437Z Entering 'third_party/flash-attention' 2025-03-14T05:31:57.6431896Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:31:57.6506445Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:31:57.6585758Z Entering 'third_party/flatbuffers' 2025-03-14T05:31:57.6658052Z Entering 'third_party/fmt' 2025-03-14T05:31:57.6726047Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:31:57.6796925Z Entering 'third_party/gloo' 2025-03-14T05:31:57.6868082Z Entering 'third_party/googletest' 2025-03-14T05:31:57.6937895Z Entering 'third_party/ideep' 2025-03-14T05:31:57.7004056Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:31:57.7080869Z Entering 'third_party/ittapi' 2025-03-14T05:31:57.7149783Z Entering 'third_party/kineto' 2025-03-14T05:31:57.7221301Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:31:57.7287231Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:31:57.7357368Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:31:57.7426865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:31:57.7496347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:31:57.7562153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:31:57.7633783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:31:57.7701710Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:31:57.7770173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:31:57.7845111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:31:57.7916206Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:31:57.7983016Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:31:57.8053433Z Entering 'third_party/kleidiai' 2025-03-14T05:31:57.8126610Z Entering 'third_party/mimalloc' 2025-03-14T05:31:57.8195098Z Entering 'third_party/nlohmann' 2025-03-14T05:31:57.8263336Z Entering 'third_party/onnx' 2025-03-14T05:31:57.8348896Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:31:57.8422662Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:31:57.8491007Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:31:57.8559663Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:31:57.8629969Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:31:57.8698271Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:31:57.8765973Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:31:57.8833728Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:31:57.8899893Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:31:57.8965453Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:31:57.9035396Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:31:57.9106686Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:31:57.9197232Z Entering 'third_party/pocketfft' 2025-03-14T05:31:57.9268188Z Entering 'third_party/protobuf' 2025-03-14T05:31:57.9345590Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:31:57.9414072Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:31:57.9486038Z Entering 'third_party/psimd' 2025-03-14T05:31:57.9555078Z Entering 'third_party/pthreadpool' 2025-03-14T05:31:57.9623290Z Entering 'third_party/pybind11' 2025-03-14T05:31:57.9692912Z Entering 'third_party/python-peachpy' 2025-03-14T05:31:57.9764195Z Entering 'third_party/sleef' 2025-03-14T05:31:57.9837585Z Entering 'third_party/tensorpipe' 2025-03-14T05:31:57.9907278Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:31:57.9975414Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:31:58.0042603Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:31:58.0108973Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:31:58.0173579Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:31:58.0270160Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-03-14T05:31:58.0302497Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-03-14T05:31:58.0662390Z Entering 'android/libs/fbjni' 2025-03-14T05:31:58.0732073Z Entering 'third_party/FP16' 2025-03-14T05:31:58.0802311Z Entering 'third_party/FXdiv' 2025-03-14T05:31:58.0874016Z Entering 'third_party/NNPACK' 2025-03-14T05:31:58.0944913Z Entering 'third_party/NVTX' 2025-03-14T05:31:58.1015212Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:31:58.1084927Z Entering 'third_party/XNNPACK' 2025-03-14T05:31:58.1170710Z Entering 'third_party/benchmark' 2025-03-14T05:31:58.1239295Z Entering 'third_party/composable_kernel' 2025-03-14T05:31:58.1315103Z Entering 'third_party/cpp-httplib' 2025-03-14T05:31:58.1385119Z Entering 'third_party/cpuinfo' 2025-03-14T05:31:58.1456299Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:31:58.1527222Z Entering 'third_party/cutlass' 2025-03-14T05:31:58.1607072Z Entering 'third_party/eigen' 2025-03-14T05:31:58.1678177Z Entering 'third_party/fbgemm' 2025-03-14T05:31:58.1747622Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:31:58.1815745Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:31:58.1885250Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:31:58.1959890Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:31:58.2026767Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:31:58.2096219Z Entering 'third_party/flash-attention' 2025-03-14T05:31:58.2166026Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:31:58.2239517Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:31:58.2316700Z Entering 'third_party/flatbuffers' 2025-03-14T05:31:58.2389719Z Entering 'third_party/fmt' 2025-03-14T05:31:58.2459814Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:31:58.2533083Z Entering 'third_party/gloo' 2025-03-14T05:31:58.2601929Z Entering 'third_party/googletest' 2025-03-14T05:31:58.2673570Z Entering 'third_party/ideep' 2025-03-14T05:31:58.2741352Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:31:58.2818249Z Entering 'third_party/ittapi' 2025-03-14T05:31:58.2892329Z Entering 'third_party/kineto' 2025-03-14T05:31:58.2959650Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:31:58.3024826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:31:58.3097306Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:31:58.3166430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:31:58.3238164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:31:58.3304808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:31:58.3378859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:31:58.3446800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:31:58.3514166Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:31:58.3583211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:31:58.3654136Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:31:58.3724818Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:31:58.3795568Z Entering 'third_party/kleidiai' 2025-03-14T05:31:58.3866236Z Entering 'third_party/mimalloc' 2025-03-14T05:31:58.3937461Z Entering 'third_party/nlohmann' 2025-03-14T05:31:58.4008514Z Entering 'third_party/onnx' 2025-03-14T05:31:58.4093692Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:31:58.4170541Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:31:58.4239847Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:31:58.4305890Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:31:58.4374237Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:31:58.4440926Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:31:58.4509281Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:31:58.4576427Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:31:58.4645117Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:31:58.4711297Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:31:58.4780596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:31:58.4856484Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:31:58.4949025Z Entering 'third_party/pocketfft' 2025-03-14T05:31:58.5019567Z Entering 'third_party/protobuf' 2025-03-14T05:31:58.5091733Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:31:58.5162891Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:31:58.5235214Z Entering 'third_party/psimd' 2025-03-14T05:31:58.5305767Z Entering 'third_party/pthreadpool' 2025-03-14T05:31:58.5376433Z Entering 'third_party/pybind11' 2025-03-14T05:31:58.5447738Z Entering 'third_party/python-peachpy' 2025-03-14T05:31:58.5516429Z Entering 'third_party/sleef' 2025-03-14T05:31:58.5588581Z Entering 'third_party/tensorpipe' 2025-03-14T05:31:58.5656633Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:31:58.5723097Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:31:58.5792911Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:31:58.5864469Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:31:58.5930769Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:31:58.6036400Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-03-14T05:31:58.6090254Z ##[endgroup] 2025-03-14T05:31:58.6090840Z ##[group]Fetching the repository 2025-03-14T05:31:58.6097455Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-03-14T05:31:58.9477813Z From https://github.com/pytorch/pytorch 2025-03-14T05:31:58.9478333Z - [deleted] (none) -> origin/angelayi/hf_version_update 2025-03-14T05:31:59.0138526Z - [deleted] (none) -> origin/atalman-patch-4 2025-03-14T05:31:59.0139069Z - [deleted] (none) -> origin/chenyang78/dyn-shape-ci-tmp 2025-03-14T05:31:59.0140033Z - [deleted] (none) -> origin/cleanup_vs_2019 2025-03-14T05:31:59.0140757Z - [deleted] (none) -> origin/csl/fflint 2025-03-14T05:31:59.0141893Z - [deleted] (none) -> origin/dataclass 2025-03-14T05:31:59.0143377Z - [deleted] (none) -> origin/gh/EikanWang/74/base 2025-03-14T05:31:59.0144356Z - [deleted] (none) -> origin/gh/EikanWang/74/head 2025-03-14T05:31:59.0145634Z - [deleted] (none) -> origin/gh/EikanWang/74/orig 2025-03-14T05:31:59.0146927Z - [deleted] (none) -> origin/gh/SamGinzburg/14/base 2025-03-14T05:31:59.0148122Z - [deleted] (none) -> origin/gh/SamGinzburg/14/head 2025-03-14T05:31:59.0150142Z - [deleted] (none) -> origin/gh/SamGinzburg/14/orig 2025-03-14T05:31:59.0150848Z - [deleted] (none) -> origin/gh/ZhiweiYan-96/37/base 2025-03-14T05:31:59.0151780Z - [deleted] (none) -> origin/gh/ZhiweiYan-96/37/head 2025-03-14T05:31:59.0152878Z - [deleted] (none) -> origin/gh/anijain2305/678/base 2025-03-14T05:31:59.0154008Z - [deleted] (none) -> origin/gh/anijain2305/678/head 2025-03-14T05:31:59.0155168Z - [deleted] (none) -> origin/gh/anijain2305/678/orig 2025-03-14T05:31:59.0156301Z - [deleted] (none) -> origin/gh/benjaminglass1/47/base 2025-03-14T05:31:59.0157406Z - [deleted] (none) -> origin/gh/benjaminglass1/47/head 2025-03-14T05:31:59.0158754Z - [deleted] (none) -> origin/gh/benjaminglass1/47/orig 2025-03-14T05:31:59.0159672Z - [deleted] (none) -> origin/gh/bobrenjc93/271/base 2025-03-14T05:31:59.0161003Z - [deleted] (none) -> origin/gh/bobrenjc93/271/head 2025-03-14T05:31:59.0162096Z - [deleted] (none) -> origin/gh/bobrenjc93/271/orig 2025-03-14T05:31:59.0163203Z - [deleted] (none) -> origin/gh/drisspg/124/base 2025-03-14T05:31:59.0164352Z - [deleted] (none) -> origin/gh/drisspg/124/head 2025-03-14T05:31:59.0176683Z - [deleted] (none) -> origin/gh/drisspg/124/orig 2025-03-14T05:31:59.0177365Z - [deleted] (none) -> origin/gh/int3/10/base 2025-03-14T05:31:59.0177981Z - [deleted] (none) -> origin/gh/int3/10/head 2025-03-14T05:31:59.0178625Z - [deleted] (none) -> origin/gh/int3/10/orig 2025-03-14T05:31:59.0179242Z - [deleted] (none) -> origin/gh/int3/11/base 2025-03-14T05:31:59.0179851Z - [deleted] (none) -> origin/gh/int3/11/head 2025-03-14T05:31:59.0180406Z - [deleted] (none) -> origin/gh/int3/11/orig 2025-03-14T05:31:59.0181055Z - [deleted] (none) -> origin/gh/int3/9/base 2025-03-14T05:31:59.0181701Z - [deleted] (none) -> origin/gh/int3/9/head 2025-03-14T05:31:59.0182318Z - [deleted] (none) -> origin/gh/int3/9/orig 2025-03-14T05:31:59.0182958Z - [deleted] (none) -> origin/gh/jamesjwu/106/base 2025-03-14T05:31:59.0183604Z - [deleted] (none) -> origin/gh/jamesjwu/106/head 2025-03-14T05:31:59.0184253Z - [deleted] (none) -> origin/gh/jamesjwu/106/orig 2025-03-14T05:31:59.0184903Z - [deleted] (none) -> origin/gh/kwen2501/124/base 2025-03-14T05:31:59.0185542Z - [deleted] (none) -> origin/gh/kwen2501/124/head 2025-03-14T05:31:59.0186180Z - [deleted] (none) -> origin/gh/kwen2501/124/orig 2025-03-14T05:31:59.0186922Z - [deleted] (none) -> origin/gh/laithsakka/102/base 2025-03-14T05:31:59.0187560Z - [deleted] (none) -> origin/gh/laithsakka/102/head 2025-03-14T05:31:59.0188206Z - [deleted] (none) -> origin/gh/laithsakka/102/orig 2025-03-14T05:31:59.0189073Z - [deleted] (none) -> origin/gh/masnesral/140/base 2025-03-14T05:31:59.0189674Z - [deleted] (none) -> origin/gh/masnesral/140/head 2025-03-14T05:31:59.0190332Z - [deleted] (none) -> origin/gh/masnesral/140/orig 2025-03-14T05:31:59.0190996Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/base 2025-03-14T05:31:59.0192061Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/head 2025-03-14T05:31:59.0193085Z - [deleted] (none) -> origin/gh/mikaylagawarecki/311/orig 2025-03-14T05:31:59.0194211Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/base 2025-03-14T05:31:59.0195354Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/head 2025-03-14T05:31:59.0196486Z - [deleted] (none) -> origin/gh/mikaylagawarecki/312/orig 2025-03-14T05:31:59.0197625Z - [deleted] (none) -> origin/gh/shunting314/198/base 2025-03-14T05:31:59.0198768Z - [deleted] (none) -> origin/gh/shunting314/198/head 2025-03-14T05:31:59.0200050Z - [deleted] (none) -> origin/gh/shunting314/198/orig 2025-03-14T05:31:59.0201083Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/base 2025-03-14T05:31:59.0202167Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/head 2025-03-14T05:31:59.0203305Z - [deleted] (none) -> origin/gh/tugsbayasgalan/288/orig 2025-03-14T05:31:59.0204447Z - [deleted] (none) -> origin/gh/voznesenskym/211/base 2025-03-14T05:31:59.0205577Z - [deleted] (none) -> origin/gh/voznesenskym/211/head 2025-03-14T05:31:59.0206708Z - [deleted] (none) -> origin/gh/voznesenskym/211/orig 2025-03-14T05:31:59.0207834Z - [deleted] (none) -> origin/gh/wconstab/382/base 2025-03-14T05:31:59.0208983Z - [deleted] (none) -> origin/gh/wconstab/383/base 2025-03-14T05:31:59.0210140Z - [deleted] (none) -> origin/gh/wconstab/384/base 2025-03-14T05:31:59.0211314Z - [deleted] (none) -> origin/gh/wconstab/385/base 2025-03-14T05:31:59.0212452Z - [deleted] (none) -> origin/gh/wconstab/385/head 2025-03-14T05:31:59.0213585Z - [deleted] (none) -> origin/gh/wconstab/386/base 2025-03-14T05:31:59.0214710Z - [deleted] (none) -> origin/gh/wconstab/391/base 2025-03-14T05:31:59.0215848Z - [deleted] (none) -> origin/gh/wconstab/391/head 2025-03-14T05:31:59.0216975Z - [deleted] (none) -> origin/gh/wconstab/391/orig 2025-03-14T05:31:59.0218125Z - [deleted] (none) -> origin/gh/yanboliang/63/base 2025-03-14T05:31:59.0219233Z - [deleted] (none) -> origin/gh/yanboliang/63/orig 2025-03-14T05:31:59.0220376Z - [deleted] (none) -> origin/gh/yanboliang/64/base 2025-03-14T05:31:59.0221550Z - [deleted] (none) -> origin/gh/yanboliang/64/orig 2025-03-14T05:31:59.0222705Z - [deleted] (none) -> origin/gh/yanboliang/65/base 2025-03-14T05:31:59.0223870Z - [deleted] (none) -> origin/gh/yanboliang/65/orig 2025-03-14T05:31:59.0225026Z - [deleted] (none) -> origin/gh/zou3519/1105/base 2025-03-14T05:31:59.0226444Z - [deleted] (none) -> origin/gh/zou3519/1105/head 2025-03-14T05:31:59.0227841Z - [deleted] (none) -> origin/gh/zou3519/1105/orig 2025-03-14T05:31:59.0228885Z - [deleted] (none) -> origin/gh/zou3519/1131/base 2025-03-14T05:31:59.0230049Z - [deleted] (none) -> origin/gh/zou3519/1131/head 2025-03-14T05:31:59.0231220Z - [deleted] (none) -> origin/gh/zou3519/1131/orig 2025-03-14T05:31:59.0232351Z - [deleted] (none) -> origin/gh/zou3519/1133/base 2025-03-14T05:31:59.0233533Z - [deleted] (none) -> origin/gh/zou3519/1133/head 2025-03-14T05:31:59.0234665Z - [deleted] (none) -> origin/gh/zou3519/1133/orig 2025-03-14T05:31:59.0235810Z - [deleted] (none) -> origin/malfet-patch-13 2025-03-14T05:31:59.0236957Z - [deleted] (none) -> origin/malfet-patch-14 2025-03-14T05:31:59.0238092Z - [deleted] (none) -> origin/malfet-patch-15 2025-03-14T05:31:59.0239247Z - [deleted] (none) -> origin/revert_commit_b57b38b 2025-03-14T05:31:59.0240401Z - [deleted] (none) -> origin/shunting-loaf-bm-6 2025-03-14T05:31:59.0241719Z - [deleted] (none) -> origin/zbv_algo 2025-03-14T05:31:59.0242708Z - [deleted] (none) -> ciflow/inductor/147552 2025-03-14T05:31:59.0243821Z - [deleted] (none) -> ciflow/inductor/149162 2025-03-14T05:31:59.0244948Z - [deleted] (none) -> ciflow/inductor/149167 2025-03-14T05:31:59.0246094Z - [deleted] (none) -> ciflow/inductor/149174 2025-03-14T05:31:59.0247226Z - [deleted] (none) -> ciflow/mps/149123 2025-03-14T05:31:59.0248381Z - [deleted] (none) -> ciflow/mps/149174 2025-03-14T05:31:59.0249496Z - [deleted] (none) -> ciflow/trunk/147552 2025-03-14T05:31:59.0250636Z - [deleted] (none) -> ciflow/trunk/148677 2025-03-14T05:31:59.0251804Z - [deleted] (none) -> ciflow/trunk/149142 2025-03-14T05:31:59.0252919Z - [deleted] (none) -> ciflow/trunk/149166 2025-03-14T05:31:59.0254054Z - [deleted] (none) -> ciflow/trunk/149171 2025-03-14T05:32:00.6508216Z 708a6d24ef4..a7fd123a105 gh/XuehaiPan/105/base -> origin/gh/XuehaiPan/105/base 2025-03-14T05:32:00.6514710Z cd35157939a..400fb6b954b gh/XuehaiPan/105/head -> origin/gh/XuehaiPan/105/head 2025-03-14T05:32:00.6521003Z + 8ff954ec831...dd091eb179e gh/XuehaiPan/105/orig -> origin/gh/XuehaiPan/105/orig (forced update) 2025-03-14T05:32:00.6529231Z c249567a846..ee65b27eaf4 gh/XuehaiPan/13/base -> origin/gh/XuehaiPan/13/base 2025-03-14T05:32:00.6537029Z e4931479838..43fb2e297e0 gh/XuehaiPan/13/head -> origin/gh/XuehaiPan/13/head 2025-03-14T05:32:00.6551955Z + 983bfc24d05...9b2bca889cb gh/XuehaiPan/13/orig -> origin/gh/XuehaiPan/13/orig (forced update) 2025-03-14T05:32:00.6562474Z 826899b19ca..eb39f4a7f8f gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-03-14T05:32:00.6569930Z cd7154ec08c..acff96a79ad gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-03-14T05:32:00.6578559Z + 97f66624f40...8fb2f7a06c2 gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig (forced update) 2025-03-14T05:32:00.6587869Z dec95a75ada..df9eb561f4f gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-03-14T05:32:00.6595966Z b797ea9517f..dae2fd762ed gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-03-14T05:32:00.6604124Z + a60f6c704be...a765a088f1a gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig (forced update) 2025-03-14T05:32:00.6612687Z 52e13e69219..87fe33ef964 gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-03-14T05:32:00.6623956Z da7c40c9dec..a2a924adf1d gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-03-14T05:32:00.6630532Z + 354700762d5...6bd873b4da7 gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig (forced update) 2025-03-14T05:32:00.6639739Z a8fb34cae53..af5bc4e801a gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-03-14T05:32:00.6646219Z 108d7f193af..4037b4fc227 gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-03-14T05:32:00.6652436Z + 7699702f5fd...38e92854787 gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig (forced update) 2025-03-14T05:32:00.6658951Z b3bb23708f6..ac684fae383 gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-03-14T05:32:00.6665035Z 877a79e8a68..8e0f02a7be5 gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-03-14T05:32:00.6671662Z + 8df0e95bf74...2d95342bd35 gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig (forced update) 2025-03-14T05:32:00.6677681Z ff84ad727c1..b0aee1f20c1 gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-03-14T05:32:00.6686580Z 715ac1dfa51..15cf0019cbe gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-03-14T05:32:00.6694838Z + edeff7c0064...05116cf0919 gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig (forced update) 2025-03-14T05:32:00.6702407Z 062bbcd4b05..cce45b10618 gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-03-14T05:32:00.6708986Z c382241afdf..fd94b3884b0 gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-03-14T05:32:00.6717011Z + cf79c58d031...6039683d77d gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig (forced update) 2025-03-14T05:32:00.6723010Z af3ed87405e..3867b41eddc gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-03-14T05:32:00.6729925Z e308c157fe3..b831702ab54 gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-03-14T05:32:00.6737743Z + 87852423f75...ac5f48532c0 gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig (forced update) 2025-03-14T05:32:00.6744875Z ca147cffda7..21f53568c63 gh/XuehaiPan/72/base -> origin/gh/XuehaiPan/72/base 2025-03-14T05:32:00.6753303Z ae400005561..fbe409a11ef gh/XuehaiPan/72/head -> origin/gh/XuehaiPan/72/head 2025-03-14T05:32:00.6759246Z + fabe748e933...ba98b3e681e gh/XuehaiPan/72/orig -> origin/gh/XuehaiPan/72/orig (forced update) 2025-03-14T05:32:00.6773347Z afdde8178df..59096115f82 gh/XuehaiPan/97/base -> origin/gh/XuehaiPan/97/base 2025-03-14T05:32:00.6779703Z cb62513bade..0e5501ab3c8 gh/XuehaiPan/97/head -> origin/gh/XuehaiPan/97/head 2025-03-14T05:32:00.6787386Z + f7c34825abf...047714a3f6b gh/XuehaiPan/97/orig -> origin/gh/XuehaiPan/97/orig (forced update) 2025-03-14T05:32:00.6795175Z 5a43050ab9b..016d7beb544 gh/XuehaiPan/98/base -> origin/gh/XuehaiPan/98/base 2025-03-14T05:32:00.6802690Z 71e5dee8c99..f59a954ce40 gh/XuehaiPan/98/head -> origin/gh/XuehaiPan/98/head 2025-03-14T05:32:00.6810184Z + adb3fd6e0b1...c2504709c1d gh/XuehaiPan/98/orig -> origin/gh/XuehaiPan/98/orig (forced update) 2025-03-14T05:32:00.6846097Z 8636b394e08..9db33f3cda6 gh/malfet/227/base -> origin/gh/malfet/227/base 2025-03-14T05:32:00.6852086Z 259919aeab3..4b1f10d8c28 gh/malfet/227/head -> origin/gh/malfet/227/head 2025-03-14T05:32:00.6861170Z + 4c129ff8a05...13ad80715a8 gh/malfet/227/orig -> origin/gh/malfet/227/orig (forced update) 2025-03-14T05:32:00.6880715Z 75c1feab54b..6c5546475f6 gh/xmfan/197/head -> origin/gh/xmfan/197/head 2025-03-14T05:32:00.6889082Z + d30cde840bf...be447b425d6 gh/xmfan/197/orig -> origin/gh/xmfan/197/orig (forced update) 2025-03-14T05:32:00.6896844Z + 44f565b7a1b...217eb23e4eb gh/xmfan/199/orig -> origin/gh/xmfan/199/orig (forced update) 2025-03-14T05:32:00.6909403Z e1c0f487ca2..bd72c961cb2 jnair/mi300_docker_caching_workflow -> origin/jnair/mi300_docker_caching_workflow 2025-03-14T05:32:00.6912685Z e66ad221e97..d4496346b90 main -> origin/main 2025-03-14T05:32:00.6924837Z + 277cb23fd22...c4b8be3d09f update_submodule_FBGEMM -> origin/update_submodule_FBGEMM (forced update) 2025-03-14T05:32:00.6926595Z 4098a229a04..15cd6921a56 viable/strict -> origin/viable/strict 2025-03-14T05:32:00.6934711Z 1579a02c823..3ffd7552df1 wdvr/iss145259_alt -> origin/wdvr/iss145259_alt 2025-03-14T05:32:00.6938665Z t [tag update] ciflow/inductor/113257 -> ciflow/inductor/113257 2025-03-14T05:32:00.6940608Z t [tag update] ciflow/inductor/129420 -> ciflow/inductor/129420 2025-03-14T05:32:00.6942643Z t [tag update] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-03-14T05:32:00.6944491Z t [tag update] ciflow/inductor/138214 -> ciflow/inductor/138214 2025-03-14T05:32:00.6949811Z t [tag update] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-03-14T05:32:00.6951772Z t [tag update] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-03-14T05:32:00.6953605Z t [tag update] ciflow/inductor/148569 -> ciflow/inductor/148569 2025-03-14T05:32:00.6955941Z t [tag update] ciflow/inductor/149014 -> ciflow/inductor/149014 2025-03-14T05:32:00.6958109Z t [tag update] ciflow/inductor/149027 -> ciflow/inductor/149027 2025-03-14T05:32:00.6959942Z * [new tag] ciflow/inductor/149176 -> ciflow/inductor/149176 2025-03-14T05:32:00.6961942Z * [new tag] ciflow/inductor/149178 -> ciflow/inductor/149178 2025-03-14T05:32:00.6964962Z t [tag update] ciflow/rocm-mi300/148394 -> ciflow/rocm-mi300/148394 2025-03-14T05:32:00.6966978Z * [new tag] ciflow/rocm/147527 -> ciflow/rocm/147527 2025-03-14T05:32:00.6968722Z t [tag update] ciflow/rocm/148394 -> ciflow/rocm/148394 2025-03-14T05:32:00.6971049Z t [tag update] ciflow/trunk/113257 -> ciflow/trunk/113257 2025-03-14T05:32:00.6972399Z t [tag update] ciflow/trunk/113258 -> ciflow/trunk/113258 2025-03-14T05:32:00.6974592Z t [tag update] ciflow/trunk/129420 -> ciflow/trunk/129420 2025-03-14T05:32:00.6976526Z t [tag update] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-03-14T05:32:00.6979615Z * [new tag] ciflow/trunk/146289 -> ciflow/trunk/146289 2025-03-14T05:32:00.6981632Z * [new tag] ciflow/trunk/147527 -> ciflow/trunk/147527 2025-03-14T05:32:00.6983748Z t [tag update] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-03-14T05:32:00.6985807Z * [new tag] ciflow/trunk/149018 -> ciflow/trunk/149018 2025-03-14T05:32:00.6987264Z * [new tag] ciflow/trunk/149064 -> ciflow/trunk/149064 2025-03-14T05:32:00.7780391Z [command]/usr/bin/git rev-parse --verify --quiet aed0b7a742a2d7b7901790622829cbd2135049a4^{object} 2025-03-14T05:32:00.7810095Z aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:32:00.7816224Z ##[endgroup] 2025-03-14T05:32:00.7816777Z ##[group]Determining the checkout info 2025-03-14T05:32:00.7817458Z ##[endgroup] 2025-03-14T05:32:00.7822838Z [command]/usr/bin/git sparse-checkout disable 2025-03-14T05:32:00.9520881Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-03-14T05:32:00.9554413Z ##[group]Checking out the ref 2025-03-14T05:32:00.9558253Z [command]/usr/bin/git checkout --progress --force aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:32:01.1168266Z Previous HEAD position was 268fb634bb4 Update on "[dynamo] fix bug where non-recursive disable modifies the original function" 2025-03-14T05:32:01.1184073Z HEAD is now at aed0b7a742a [c10d] Add param recording for uniqueID broadcasting and allgather (#149166) 2025-03-14T05:32:01.1240892Z ##[endgroup] 2025-03-14T05:32:01.1241357Z ##[group]Setting up auth for fetching submodules 2025-03-14T05:32:01.1247632Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-03-14T05:32:01.1297956Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-03-14T05:32:01.1331578Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-03-14T05:32:01.1363710Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-03-14T05:32:01.1392789Z ##[endgroup] 2025-03-14T05:32:01.1393337Z ##[group]Fetching submodules 2025-03-14T05:32:01.1396100Z [command]/usr/bin/git submodule sync --recursive 2025-03-14T05:32:01.1764216Z Synchronizing submodule url for 'android/libs/fbjni' 2025-03-14T05:32:01.1791471Z Synchronizing submodule url for 'third_party/FP16' 2025-03-14T05:32:01.1819195Z Synchronizing submodule url for 'third_party/FXdiv' 2025-03-14T05:32:01.1846038Z Synchronizing submodule url for 'third_party/NNPACK' 2025-03-14T05:32:01.1872389Z Synchronizing submodule url for 'third_party/NVTX' 2025-03-14T05:32:01.1898855Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:01.1925149Z Synchronizing submodule url for 'third_party/XNNPACK' 2025-03-14T05:32:01.1966921Z Synchronizing submodule url for 'third_party/benchmark' 2025-03-14T05:32:01.1993499Z Synchronizing submodule url for 'third_party/composable_kernel' 2025-03-14T05:32:01.2026590Z Synchronizing submodule url for 'third_party/cpp-httplib' 2025-03-14T05:32:01.2052898Z Synchronizing submodule url for 'third_party/cpuinfo' 2025-03-14T05:32:01.2079842Z Synchronizing submodule url for 'third_party/cudnn_frontend' 2025-03-14T05:32:01.2106060Z Synchronizing submodule url for 'third_party/cutlass' 2025-03-14T05:32:01.2141156Z Synchronizing submodule url for 'third_party/eigen' 2025-03-14T05:32:01.2170802Z Synchronizing submodule url for 'third_party/fbgemm' 2025-03-14T05:32:01.2196437Z Synchronizing submodule url for 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:01.2222243Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:01.2248360Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:01.2281059Z Synchronizing submodule url for 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:01.2307003Z Synchronizing submodule url for 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:01.2337190Z Synchronizing submodule url for 'third_party/flash-attention' 2025-03-14T05:32:01.2362663Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:01.2394896Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:01.2436294Z Synchronizing submodule url for 'third_party/flatbuffers' 2025-03-14T05:32:01.2464821Z Synchronizing submodule url for 'third_party/fmt' 2025-03-14T05:32:01.2491584Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:01.2517628Z Synchronizing submodule url for 'third_party/gloo' 2025-03-14T05:32:01.2544056Z Synchronizing submodule url for 'third_party/googletest' 2025-03-14T05:32:01.2570862Z Synchronizing submodule url for 'third_party/ideep' 2025-03-14T05:32:01.2594175Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:01.2629452Z Synchronizing submodule url for 'third_party/ittapi' 2025-03-14T05:32:01.2656161Z Synchronizing submodule url for 'third_party/kineto' 2025-03-14T05:32:01.2681803Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:01.2707196Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:01.2737765Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:01.2765467Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:01.2793685Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:01.2818729Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:01.2850375Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:01.2877527Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:01.2904786Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:01.2934082Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:01.2963367Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:01.2990079Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:01.3018277Z Synchronizing submodule url for 'third_party/kleidiai' 2025-03-14T05:32:01.3045858Z Synchronizing submodule url for 'third_party/mimalloc' 2025-03-14T05:32:01.3072433Z Synchronizing submodule url for 'third_party/nlohmann' 2025-03-14T05:32:01.3100406Z Synchronizing submodule url for 'third_party/onnx' 2025-03-14T05:32:01.3142232Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:01.3174338Z Synchronizing submodule url for 'third_party/opentelemetry-cpp' 2025-03-14T05:32:01.3200576Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:01.3227794Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:01.3254506Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:01.3279992Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:01.3307493Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:01.3333625Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:01.3359722Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:01.3383302Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:01.3412523Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:01.3446606Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:01.3496004Z Synchronizing submodule url for 'third_party/pocketfft' 2025-03-14T05:32:01.3524835Z Synchronizing submodule url for 'third_party/protobuf' 2025-03-14T05:32:01.3556239Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:01.3583109Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:01.3617093Z Synchronizing submodule url for 'third_party/psimd' 2025-03-14T05:32:01.3647250Z Synchronizing submodule url for 'third_party/pthreadpool' 2025-03-14T05:32:01.3673269Z Synchronizing submodule url for 'third_party/pybind11' 2025-03-14T05:32:01.3699177Z Synchronizing submodule url for 'third_party/python-peachpy' 2025-03-14T05:32:01.3725059Z Synchronizing submodule url for 'third_party/sleef' 2025-03-14T05:32:01.3752319Z Synchronizing submodule url for 'third_party/tensorpipe' 2025-03-14T05:32:01.3775977Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:01.3801883Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:01.3826925Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:01.3853028Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:01.3876840Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:01.3922770Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-03-14T05:32:01.4523453Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-03-14T05:32:01.4691327Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-03-14T05:32:01.4816490Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-03-14T05:32:01.5201979Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-03-14T05:32:01.5787111Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2025-03-14T05:32:01.6330503Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2025-03-14T05:32:02.9278560Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-03-14T05:32:02.9623194Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2025-03-14T05:32:03.3974012Z Submodule path 'third_party/composable_kernel': checked out '8086bbe3a78d931eb96fe12fdc014082e18d18d3' 2025-03-14T05:32:03.4543447Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2025-03-14T05:32:03.5891261Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2025-03-14T05:32:03.6388752Z Submodule path 'third_party/cudnn_frontend': checked out '91b7532f3386768bba4f444ee7672b497f34da8a' 2025-03-14T05:32:04.6137163Z Submodule path 'third_party/cutlass': checked out 'afa1772203677c5118fcd82537a9c8fefbcc7008' 2025-03-14T05:32:04.9661854Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2025-03-14T05:32:05.0597685Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2025-03-14T05:32:05.1168232Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2025-03-14T05:32:05.2440009Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2025-03-14T05:32:05.9503393Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2025-03-14T05:32:06.0130147Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2025-03-14T05:32:06.0284819Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2025-03-14T05:32:06.1489472Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-03-14T05:32:06.5811550Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-03-14T05:32:07.5117908Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-03-14T05:32:07.7360537Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2025-03-14T05:32:07.7792557Z Submodule path 'third_party/fmt': checked out '123913715afeb8a437e6388b4473fcc4753e1c9a' 2025-03-14T05:32:07.8299497Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-03-14T05:32:07.8699291Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2025-03-14T05:32:07.9297378Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2025-03-14T05:32:07.9476197Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-03-14T05:32:08.7857419Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-03-14T05:32:08.8087973Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2025-03-14T05:32:08.9206909Z Submodule path 'third_party/kineto': checked out '2859721fd9e73d3ca1c56f827dbc64e6d68f78a2' 2025-03-14T05:32:09.0332705Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-03-14T05:32:09.2778988Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-03-14T05:32:09.3060198Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-03-14T05:32:09.3570497Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-03-14T05:32:09.3772188Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-03-14T05:32:09.3896883Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-03-14T05:32:09.4157528Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-03-14T05:32:09.4729604Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-03-14T05:32:09.6193724Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-03-14T05:32:09.6442011Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-03-14T05:32:09.6880826Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-03-14T05:32:09.7504777Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-03-14T05:32:09.8040684Z Submodule path 'third_party/kleidiai': checked out 'ef685a13cfbe8d418aa2ed34350e21e4938358b6' 2025-03-14T05:32:09.8643441Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2025-03-14T05:32:10.0163119Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2025-03-14T05:32:10.6953599Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2025-03-14T05:32:10.7487729Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2025-03-14T05:32:10.8778472Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-03-14T05:32:10.9098458Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-03-14T05:32:10.9648426Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-03-14T05:32:10.9828971Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-03-14T05:32:11.1469916Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-03-14T05:32:11.1646883Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-03-14T05:32:11.1868344Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-03-14T05:32:11.2147443Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-03-14T05:32:11.5528948Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-03-14T05:32:11.6161047Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-03-14T05:32:12.5724257Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-03-14T05:32:12.5875244Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2025-03-14T05:32:12.9900923Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-03-14T05:32:13.0122288Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-03-14T05:32:13.0823062Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-03-14T05:32:13.0950743Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-03-14T05:32:13.1118555Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-03-14T05:32:13.1658401Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-03-14T05:32:13.2056570Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-03-14T05:32:13.2636972Z Submodule path 'third_party/sleef': checked out '56e1f79cb140fb9326d612d0be06b5250565cade' 2025-03-14T05:32:13.3091497Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2025-03-14T05:32:13.3696207Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-03-14T05:32:13.3931195Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-03-14T05:32:13.4801510Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2025-03-14T05:32:13.5220252Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-03-14T05:32:13.5339169Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-03-14T05:32:13.5391094Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-03-14T05:32:13.5751342Z Entering 'android/libs/fbjni' 2025-03-14T05:32:13.5804126Z Entering 'third_party/FP16' 2025-03-14T05:32:13.5856493Z Entering 'third_party/FXdiv' 2025-03-14T05:32:13.5908338Z Entering 'third_party/NNPACK' 2025-03-14T05:32:13.5964830Z Entering 'third_party/NVTX' 2025-03-14T05:32:13.6021104Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:13.6075015Z Entering 'third_party/XNNPACK' 2025-03-14T05:32:13.6144891Z Entering 'third_party/benchmark' 2025-03-14T05:32:13.6199297Z Entering 'third_party/composable_kernel' 2025-03-14T05:32:13.6262626Z Entering 'third_party/cpp-httplib' 2025-03-14T05:32:13.6315086Z Entering 'third_party/cpuinfo' 2025-03-14T05:32:13.6369496Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:32:13.6421313Z Entering 'third_party/cutlass' 2025-03-14T05:32:13.6483734Z Entering 'third_party/eigen' 2025-03-14T05:32:13.6542742Z Entering 'third_party/fbgemm' 2025-03-14T05:32:13.6598448Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:13.6649638Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:13.6699689Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:13.6757375Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:13.6806697Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:13.6861462Z Entering 'third_party/flash-attention' 2025-03-14T05:32:13.6912883Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:13.6970232Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:13.7031623Z Entering 'third_party/flatbuffers' 2025-03-14T05:32:13.7090592Z Entering 'third_party/fmt' 2025-03-14T05:32:13.7143623Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:13.7202027Z Entering 'third_party/gloo' 2025-03-14T05:32:13.7257744Z Entering 'third_party/googletest' 2025-03-14T05:32:13.7310344Z Entering 'third_party/ideep' 2025-03-14T05:32:13.7362556Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:13.7422655Z Entering 'third_party/ittapi' 2025-03-14T05:32:13.7478565Z Entering 'third_party/kineto' 2025-03-14T05:32:13.7530736Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:13.7581466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:13.7639919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:13.7691251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:13.7743417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:13.7793919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:13.7849418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:13.7900901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:13.7955550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:13.8008726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:13.8064995Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:13.8115848Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:13.8170570Z Entering 'third_party/kleidiai' 2025-03-14T05:32:13.8225752Z Entering 'third_party/mimalloc' 2025-03-14T05:32:13.8283350Z Entering 'third_party/nlohmann' 2025-03-14T05:32:13.8342141Z Entering 'third_party/onnx' 2025-03-14T05:32:13.8413927Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:13.8471993Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:32:13.8525763Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:13.8578168Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:13.8628187Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:13.8678242Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:13.8730859Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:13.8781067Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:13.8831593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:13.8881307Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:13.8936337Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:13.8991420Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:13.9066072Z Entering 'third_party/pocketfft' 2025-03-14T05:32:13.9119015Z Entering 'third_party/protobuf' 2025-03-14T05:32:13.9175691Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:13.9226004Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:13.9280802Z Entering 'third_party/psimd' 2025-03-14T05:32:13.9333509Z Entering 'third_party/pthreadpool' 2025-03-14T05:32:13.9389006Z Entering 'third_party/pybind11' 2025-03-14T05:32:13.9442248Z Entering 'third_party/python-peachpy' 2025-03-14T05:32:13.9493723Z Entering 'third_party/sleef' 2025-03-14T05:32:13.9552596Z Entering 'third_party/tensorpipe' 2025-03-14T05:32:13.9605062Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:13.9656385Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:13.9705508Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:13.9755885Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:13.9805146Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:13.9882884Z ##[endgroup] 2025-03-14T05:32:13.9883345Z ##[group]Persisting credentials for submodules 2025-03-14T05:32:13.9889990Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-03-14T05:32:14.0251163Z Entering 'android/libs/fbjni' 2025-03-14T05:32:14.0300765Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0301268Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0345584Z Entering 'third_party/FP16' 2025-03-14T05:32:14.0393622Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0394029Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0440627Z Entering 'third_party/FXdiv' 2025-03-14T05:32:14.0484506Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0484877Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0531088Z Entering 'third_party/NNPACK' 2025-03-14T05:32:14.0575065Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0575542Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0618627Z Entering 'third_party/NVTX' 2025-03-14T05:32:14.0662401Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0662904Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0708377Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:14.0752225Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0752712Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0797604Z Entering 'third_party/XNNPACK' 2025-03-14T05:32:14.0841640Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0842057Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0906401Z Entering 'third_party/benchmark' 2025-03-14T05:32:14.0952907Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0953277Z url.https://github.com/.insteadof 2025-03-14T05:32:14.0998568Z Entering 'third_party/composable_kernel' 2025-03-14T05:32:14.1044622Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1045331Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1099020Z Entering 'third_party/cpp-httplib' 2025-03-14T05:32:14.1142801Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1143172Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1187346Z Entering 'third_party/cpuinfo' 2025-03-14T05:32:14.1232664Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1233019Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1278181Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:32:14.1321963Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1322476Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1367195Z Entering 'third_party/cutlass' 2025-03-14T05:32:14.1411240Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1411600Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1465955Z Entering 'third_party/eigen' 2025-03-14T05:32:14.1509648Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1510029Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1556547Z Entering 'third_party/fbgemm' 2025-03-14T05:32:14.1600890Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1601246Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1644303Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:14.1687442Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1687821Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1731569Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:14.1774899Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1775255Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1819164Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:14.1862042Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1862398Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1912412Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:14.1955883Z url.https://github.com/.insteadof 2025-03-14T05:32:14.1956235Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2000096Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:14.2043216Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2043567Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2089323Z Entering 'third_party/flash-attention' 2025-03-14T05:32:14.2133554Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2134031Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2178448Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:14.2223358Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2223809Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2277212Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:14.2320027Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2320423Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2375740Z Entering 'third_party/flatbuffers' 2025-03-14T05:32:14.2419470Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2419864Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2468795Z Entering 'third_party/fmt' 2025-03-14T05:32:14.2512125Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2512520Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2556585Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:14.2600220Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2600620Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2648321Z Entering 'third_party/gloo' 2025-03-14T05:32:14.2691619Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2692043Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2735988Z Entering 'third_party/googletest' 2025-03-14T05:32:14.2784353Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2784746Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2830017Z Entering 'third_party/ideep' 2025-03-14T05:32:14.2873422Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2873825Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2916729Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:14.2962561Z url.https://github.com/.insteadof 2025-03-14T05:32:14.2962976Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3014905Z Entering 'third_party/ittapi' 2025-03-14T05:32:14.3058959Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3059373Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3103090Z Entering 'third_party/kineto' 2025-03-14T05:32:14.3147873Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3148455Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3191516Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:14.3234971Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3235342Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3280051Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:14.3323989Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3324498Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3373082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:14.3416576Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3417085Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3462097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:14.3508036Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3508414Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3553495Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:14.3596595Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3597097Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3640139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:14.3684186Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3684541Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3732380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:14.3775122Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3775630Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3819947Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:14.3863074Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3863588Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3908137Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:14.3954943Z url.https://github.com/.insteadof 2025-03-14T05:32:14.3955302Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4001387Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:14.4044031Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4044384Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4091610Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:14.4135797Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4136311Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4180053Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:14.4222986Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4223342Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4274142Z Entering 'third_party/kleidiai' 2025-03-14T05:32:14.4318206Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4318550Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4362851Z Entering 'third_party/mimalloc' 2025-03-14T05:32:14.4406048Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4406397Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4450479Z Entering 'third_party/nlohmann' 2025-03-14T05:32:14.4494141Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4494837Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4542546Z Entering 'third_party/onnx' 2025-03-14T05:32:14.4587660Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4588155Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4649191Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:14.4692886Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4693251Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4742240Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:32:14.4786929Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4787286Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4833462Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:14.4876299Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4876651Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4919172Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:14.4964152Z url.https://github.com/.insteadof 2025-03-14T05:32:14.4964498Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5008138Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:14.5050896Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5051437Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5093908Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:14.5141838Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5142220Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5186795Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:14.5230915Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5231426Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5274846Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:14.5317931Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5318293Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5361796Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:14.5405165Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5405656Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5447662Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:14.5491123Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5491584Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5537443Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:14.5580131Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5580586Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5627196Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:14.5673964Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5674450Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5738774Z Entering 'third_party/pocketfft' 2025-03-14T05:32:14.5783099Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5783582Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5827118Z Entering 'third_party/protobuf' 2025-03-14T05:32:14.5872600Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5873082Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5918787Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:14.5963329Z url.https://github.com/.insteadof 2025-03-14T05:32:14.5963697Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6007715Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:14.6053635Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6053998Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6100478Z Entering 'third_party/psimd' 2025-03-14T05:32:14.6144946Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6145521Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6189827Z Entering 'third_party/pthreadpool' 2025-03-14T05:32:14.6234602Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6234954Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6278449Z Entering 'third_party/pybind11' 2025-03-14T05:32:14.6322552Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6322911Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6367234Z Entering 'third_party/python-peachpy' 2025-03-14T05:32:14.6411188Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6411602Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6456116Z Entering 'third_party/sleef' 2025-03-14T05:32:14.6498988Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6510134Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6545207Z Entering 'third_party/tensorpipe' 2025-03-14T05:32:14.6589591Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6590036Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6635519Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:14.6679334Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6679695Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6722170Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:14.6764808Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6765178Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6808824Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:14.6852214Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6852573Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6899932Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:14.6942451Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6942829Z url.https://github.com/.insteadof 2025-03-14T05:32:14.6987905Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:14.7031408Z url.https://github.com/.insteadof 2025-03-14T05:32:14.7031867Z url.https://github.com/.insteadof 2025-03-14T05:32:14.7103620Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-03-14T05:32:14.7464266Z Entering 'android/libs/fbjni' 2025-03-14T05:32:14.7529234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-03-14T05:32:14.7551334Z Entering 'third_party/FP16' 2025-03-14T05:32:14.7613955Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-03-14T05:32:14.7637265Z Entering 'third_party/FXdiv' 2025-03-14T05:32:14.7703303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-03-14T05:32:14.7725261Z Entering 'third_party/NNPACK' 2025-03-14T05:32:14.7805487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-03-14T05:32:14.7827954Z Entering 'third_party/NVTX' 2025-03-14T05:32:14.7889052Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-03-14T05:32:14.7910520Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:14.7972542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-03-14T05:32:14.7994973Z Entering 'third_party/XNNPACK' 2025-03-14T05:32:14.8056468Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-03-14T05:32:14.8094539Z Entering 'third_party/benchmark' 2025-03-14T05:32:14.8155449Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-03-14T05:32:14.8176568Z Entering 'third_party/composable_kernel' 2025-03-14T05:32:14.8248376Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-03-14T05:32:14.8276724Z Entering 'third_party/cpp-httplib' 2025-03-14T05:32:14.8338826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-03-14T05:32:14.8362053Z Entering 'third_party/cpuinfo' 2025-03-14T05:32:14.8422815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-03-14T05:32:14.8445078Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:32:14.8504576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-03-14T05:32:14.8525749Z Entering 'third_party/cutlass' 2025-03-14T05:32:14.8587159Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-03-14T05:32:14.8616707Z Entering 'third_party/eigen' 2025-03-14T05:32:14.8677606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2025-03-14T05:32:14.8700633Z Entering 'third_party/fbgemm' 2025-03-14T05:32:14.8765138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-03-14T05:32:14.8783970Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:14.8844940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2025-03-14T05:32:14.8865130Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:14.8930234Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2025-03-14T05:32:14.8950487Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:14.9009866Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2025-03-14T05:32:14.9037852Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:14.9111226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:14.9132892Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:14.9196483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2025-03-14T05:32:14.9219334Z Entering 'third_party/flash-attention' 2025-03-14T05:32:14.9287790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-03-14T05:32:14.9310273Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:14.9373079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-03-14T05:32:14.9401473Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:14.9469988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-03-14T05:32:14.9503705Z Entering 'third_party/flatbuffers' 2025-03-14T05:32:14.9566239Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-03-14T05:32:14.9592609Z Entering 'third_party/fmt' 2025-03-14T05:32:14.9667936Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-03-14T05:32:14.9689908Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:14.9764197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-03-14T05:32:14.9786971Z Entering 'third_party/gloo' 2025-03-14T05:32:14.9850711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-03-14T05:32:14.9873515Z Entering 'third_party/googletest' 2025-03-14T05:32:14.9936213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:14.9959040Z Entering 'third_party/ideep' 2025-03-14T05:32:15.0023594Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-03-14T05:32:15.0043285Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:15.0106837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-03-14T05:32:15.0139370Z Entering 'third_party/ittapi' 2025-03-14T05:32:15.0202447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-03-14T05:32:15.0229945Z Entering 'third_party/kineto' 2025-03-14T05:32:15.0295596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-03-14T05:32:15.0315374Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:15.0389862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-03-14T05:32:15.0409837Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:15.0474822Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-03-14T05:32:15.0497647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:15.0560612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-03-14T05:32:15.0582456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:15.0647337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-03-14T05:32:15.0667927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:15.0731125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-03-14T05:32:15.0749697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:15.0812494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-03-14T05:32:15.0842890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:15.0905686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-03-14T05:32:15.0927572Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:15.0989697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:15.1011112Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:15.1073719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-03-14T05:32:15.1097262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:15.1164297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-03-14T05:32:15.1187430Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:15.1249779Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-03-14T05:32:15.1271208Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:15.1334757Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-03-14T05:32:15.1360153Z Entering 'third_party/kleidiai' 2025-03-14T05:32:15.1423059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-03-14T05:32:15.1446709Z Entering 'third_party/mimalloc' 2025-03-14T05:32:15.1510169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-03-14T05:32:15.1533972Z Entering 'third_party/nlohmann' 2025-03-14T05:32:15.1596465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-03-14T05:32:15.1620164Z Entering 'third_party/onnx' 2025-03-14T05:32:15.1683988Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-03-14T05:32:15.1719759Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:15.1782595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-03-14T05:32:15.1810035Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:32:15.1873783Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-03-14T05:32:15.1895591Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:15.1957641Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-03-14T05:32:15.1979177Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:15.2041477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:15.2063256Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:15.2129756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-03-14T05:32:15.2150884Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:15.2212530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-03-14T05:32:15.2235750Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:15.2297262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-03-14T05:32:15.2318815Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:15.2381067Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-03-14T05:32:15.2402752Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:15.2465686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-03-14T05:32:15.2486003Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:15.2548800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-03-14T05:32:15.2571153Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:15.2633814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-03-14T05:32:15.2657885Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:15.2719345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-03-14T05:32:15.2763109Z Entering 'third_party/pocketfft' 2025-03-14T05:32:15.2827412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-03-14T05:32:15.2848416Z Entering 'third_party/protobuf' 2025-03-14T05:32:15.2913268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-03-14T05:32:15.2937097Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:15.2998895Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-03-14T05:32:15.3019905Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:15.3084416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:15.3113677Z Entering 'third_party/psimd' 2025-03-14T05:32:15.3178236Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-03-14T05:32:15.3199865Z Entering 'third_party/pthreadpool' 2025-03-14T05:32:15.3263274Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-03-14T05:32:15.3285933Z Entering 'third_party/pybind11' 2025-03-14T05:32:15.3350417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-03-14T05:32:15.3372599Z Entering 'third_party/python-peachpy' 2025-03-14T05:32:15.3435308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-03-14T05:32:15.3457554Z Entering 'third_party/sleef' 2025-03-14T05:32:15.3519681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-03-14T05:32:15.3542274Z Entering 'third_party/tensorpipe' 2025-03-14T05:32:15.3605826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-03-14T05:32:15.3627259Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:15.3691585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-03-14T05:32:15.3717688Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:15.3779797Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-03-14T05:32:15.3801479Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:15.3863419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-03-14T05:32:15.3884830Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:15.3947986Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-03-14T05:32:15.3967175Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:15.4031709Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-03-14T05:32:15.5113360Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-03-14T05:32:15.5476264Z Entering 'android/libs/fbjni' 2025-03-14T05:32:15.5530255Z Entering 'third_party/FP16' 2025-03-14T05:32:15.5584041Z Entering 'third_party/FXdiv' 2025-03-14T05:32:15.5637164Z Entering 'third_party/NNPACK' 2025-03-14T05:32:15.5689547Z Entering 'third_party/NVTX' 2025-03-14T05:32:15.5742620Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:15.5795808Z Entering 'third_party/XNNPACK' 2025-03-14T05:32:15.5863990Z Entering 'third_party/benchmark' 2025-03-14T05:32:15.5919522Z Entering 'third_party/composable_kernel' 2025-03-14T05:32:15.5979896Z Entering 'third_party/cpp-httplib' 2025-03-14T05:32:15.6038023Z Entering 'third_party/cpuinfo' 2025-03-14T05:32:15.6090755Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:32:15.6143818Z Entering 'third_party/cutlass' 2025-03-14T05:32:15.6204869Z Entering 'third_party/eigen' 2025-03-14T05:32:15.6260282Z Entering 'third_party/fbgemm' 2025-03-14T05:32:15.6312359Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:15.6362715Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:15.6413536Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:15.6474454Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:15.6525040Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:15.6580355Z Entering 'third_party/flash-attention' 2025-03-14T05:32:15.6638182Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:15.6695419Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:15.6756137Z Entering 'third_party/flatbuffers' 2025-03-14T05:32:15.6810786Z Entering 'third_party/fmt' 2025-03-14T05:32:15.6865160Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:15.6920115Z Entering 'third_party/gloo' 2025-03-14T05:32:15.6973479Z Entering 'third_party/googletest' 2025-03-14T05:32:15.7030041Z Entering 'third_party/ideep' 2025-03-14T05:32:15.7081307Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:15.7142273Z Entering 'third_party/ittapi' 2025-03-14T05:32:15.7198204Z Entering 'third_party/kineto' 2025-03-14T05:32:15.7251836Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:15.7302693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:15.7362307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:15.7413750Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:15.7467924Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:15.7515845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:15.7572081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:15.7622303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:15.7675724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:15.7729691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:15.7786116Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:15.7838452Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:15.7894025Z Entering 'third_party/kleidiai' 2025-03-14T05:32:15.7950357Z Entering 'third_party/mimalloc' 2025-03-14T05:32:15.8003265Z Entering 'third_party/nlohmann' 2025-03-14T05:32:15.8057693Z Entering 'third_party/onnx' 2025-03-14T05:32:15.8124512Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:15.8186819Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:32:15.8241083Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:15.8291568Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:15.8348006Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:15.8398580Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:15.8451547Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:15.8506108Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:15.8556869Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:15.8605361Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:15.8658118Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:15.8710785Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:15.8784860Z Entering 'third_party/pocketfft' 2025-03-14T05:32:15.8838324Z Entering 'third_party/protobuf' 2025-03-14T05:32:15.8895548Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:15.8951517Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:15.9005002Z Entering 'third_party/psimd' 2025-03-14T05:32:15.9057786Z Entering 'third_party/pthreadpool' 2025-03-14T05:32:15.9109986Z Entering 'third_party/pybind11' 2025-03-14T05:32:15.9163298Z Entering 'third_party/python-peachpy' 2025-03-14T05:32:15.9216358Z Entering 'third_party/sleef' 2025-03-14T05:32:15.9270092Z Entering 'third_party/tensorpipe' 2025-03-14T05:32:15.9323822Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:15.9376291Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:15.9431269Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:15.9482222Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:15.9537393Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:15.9614374Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-03-14T05:32:15.9976133Z Entering 'android/libs/fbjni' 2025-03-14T05:32:16.0027890Z Entering 'third_party/FP16' 2025-03-14T05:32:16.0080095Z Entering 'third_party/FXdiv' 2025-03-14T05:32:16.0132999Z Entering 'third_party/NNPACK' 2025-03-14T05:32:16.0185509Z Entering 'third_party/NVTX' 2025-03-14T05:32:16.0238571Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T05:32:16.0290539Z Entering 'third_party/XNNPACK' 2025-03-14T05:32:16.0358491Z Entering 'third_party/benchmark' 2025-03-14T05:32:16.0410512Z Entering 'third_party/composable_kernel' 2025-03-14T05:32:16.0470005Z Entering 'third_party/cpp-httplib' 2025-03-14T05:32:16.0521257Z Entering 'third_party/cpuinfo' 2025-03-14T05:32:16.0577824Z Entering 'third_party/cudnn_frontend' 2025-03-14T05:32:16.0629815Z Entering 'third_party/cutlass' 2025-03-14T05:32:16.0690903Z Entering 'third_party/eigen' 2025-03-14T05:32:16.0747953Z Entering 'third_party/fbgemm' 2025-03-14T05:32:16.0799815Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T05:32:16.0850921Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T05:32:16.0902788Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T05:32:16.0966276Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T05:32:16.1016680Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T05:32:16.1070254Z Entering 'third_party/flash-attention' 2025-03-14T05:32:16.1124608Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T05:32:16.1182606Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T05:32:16.1250042Z Entering 'third_party/flatbuffers' 2025-03-14T05:32:16.1305081Z Entering 'third_party/fmt' 2025-03-14T05:32:16.1359478Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T05:32:16.1411767Z Entering 'third_party/gloo' 2025-03-14T05:32:16.1464562Z Entering 'third_party/googletest' 2025-03-14T05:32:16.1516284Z Entering 'third_party/ideep' 2025-03-14T05:32:16.1567358Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T05:32:16.1627569Z Entering 'third_party/ittapi' 2025-03-14T05:32:16.1681717Z Entering 'third_party/kineto' 2025-03-14T05:32:16.1734018Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T05:32:16.1785270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T05:32:16.1838765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T05:32:16.1889390Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T05:32:16.1940782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T05:32:16.1990262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T05:32:16.2047305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T05:32:16.2099863Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T05:32:16.2150552Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T05:32:16.2202698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T05:32:16.2259083Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T05:32:16.2311145Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T05:32:16.2367672Z Entering 'third_party/kleidiai' 2025-03-14T05:32:16.2422311Z Entering 'third_party/mimalloc' 2025-03-14T05:32:16.2476646Z Entering 'third_party/nlohmann' 2025-03-14T05:32:16.2531205Z Entering 'third_party/onnx' 2025-03-14T05:32:16.2598849Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T05:32:16.2655868Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T05:32:16.2707795Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T05:32:16.2759067Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T05:32:16.2810421Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T05:32:16.2861113Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T05:32:16.2917586Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T05:32:16.2968635Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T05:32:16.3019357Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T05:32:16.3068914Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T05:32:16.3121779Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T05:32:16.3176298Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T05:32:16.3252996Z Entering 'third_party/pocketfft' 2025-03-14T05:32:16.3305757Z Entering 'third_party/protobuf' 2025-03-14T05:32:16.3360527Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T05:32:16.3410714Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T05:32:16.3467026Z Entering 'third_party/psimd' 2025-03-14T05:32:16.3520038Z Entering 'third_party/pthreadpool' 2025-03-14T05:32:16.3573810Z Entering 'third_party/pybind11' 2025-03-14T05:32:16.3627497Z Entering 'third_party/python-peachpy' 2025-03-14T05:32:16.3678096Z Entering 'third_party/sleef' 2025-03-14T05:32:16.3730219Z Entering 'third_party/tensorpipe' 2025-03-14T05:32:16.3781671Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T05:32:16.3834899Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T05:32:16.3886628Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T05:32:16.3936997Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T05:32:16.3986321Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T05:32:16.4060922Z ##[endgroup] 2025-03-14T05:32:16.4108865Z [command]/usr/bin/git log -1 --format=%H 2025-03-14T05:32:16.4138937Z aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:32:16.4328739Z Prepare all required actions 2025-03-14T05:32:16.4329445Z Getting action download info 2025-03-14T05:32:16.5513240Z ##[group]Run ./.github/actions/setup-linux 2025-03-14T05:32:16.5513571Z env: 2025-03-14T05:32:16.5513803Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:16.5514077Z ##[endgroup] 2025-03-14T05:32:16.5560393Z ##[group]Run set -euo pipefail 2025-03-14T05:32:16.5560746Z set -euo pipefail 2025-03-14T05:32:16.5561054Z function get_ec2_metadata() { 2025-03-14T05:32:16.5561446Z  # Pulled from instance metadata endpoint for EC2 2025-03-14T05:32:16.5562077Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-03-14T05:32:16.5562651Z  category=$1 2025-03-14T05:32:16.5563027Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-03-14T05:32:16.5563474Z  runner_name_str=i-0166a710cfefd3e7e 2025-03-14T05:32:16.5563876Z  if [[ -f /.inarc ]]; then 2025-03-14T05:32:16.5564268Z  echo "ARC Runner, no info on ec2 metadata" 2025-03-14T05:32:16.5564674Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-03-14T05:32:16.5565146Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-03-14T05:32:16.5565577Z  else 2025-03-14T05:32:16.5566392Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-03-14T05:32:16.5567249Z  fi 2025-03-14T05:32:16.5567483Z } 2025-03-14T05:32:16.5567770Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-03-14T05:32:16.5568201Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-03-14T05:32:16.5568675Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-03-14T05:32:16.5569096Z echo "system info $(uname -a)" 2025-03-14T05:32:16.5578062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:16.5578439Z env: 2025-03-14T05:32:16.5578681Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:16.5578945Z ##[endgroup] 2025-03-14T05:32:16.5747989Z ami-id: ami-08b5b3a93ed654d19 2025-03-14T05:32:16.5862142Z instance-id: i-0166a710cfefd3e7e 2025-03-14T05:32:16.5975252Z instance-type: g5.4xlarge 2025-03-14T05:32:16.5988726Z system info Linux ip-10-0-17-58.ec2.internal 6.1.129-138.220.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Feb 25 22:18:43 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-03-14T05:32:16.6021666Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:32:16.6022751Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:32:16.6032610Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:16.6033009Z env: 2025-03-14T05:32:16.6033244Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:16.6033525Z ##[endgroup] 2025-03-14T05:32:16.6099942Z ##[group]Run if systemctl is-active --quiet docker; then 2025-03-14T05:32:16.6100399Z if systemctl is-active --quiet docker; then 2025-03-14T05:32:16.6100803Z  echo "Docker daemon is running..."; 2025-03-14T05:32:16.6101149Z else 2025-03-14T05:32:16.6101523Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2025-03-14T05:32:16.6101942Z fi 2025-03-14T05:32:16.6109907Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:16.6110297Z env: 2025-03-14T05:32:16.6110532Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:16.6110803Z ##[endgroup] 2025-03-14T05:32:16.6194488Z Docker daemon is running... 2025-03-14T05:32:16.6242930Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T05:32:16.6243237Z with: 2025-03-14T05:32:16.6243461Z shell: bash 2025-03-14T05:32:16.6243915Z timeout_minutes: 5 2025-03-14T05:32:16.6244184Z max_attempts: 3 2025-03-14T05:32:16.6244438Z retry_wait_seconds: 30 2025-03-14T05:32:16.6246546Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-03-14T05:32:16.6248685Z polling_interval_seconds: 1 2025-03-14T05:32:16.6248986Z warning_on_retry: true 2025-03-14T05:32:16.6249273Z continue_on_error: false 2025-03-14T05:32:16.6249548Z env: 2025-03-14T05:32:16.6249787Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:16.6250074Z AWS_RETRY_MODE: standard 2025-03-14T05:32:16.6250340Z AWS_MAX_ATTEMPTS: 5 2025-03-14T05:32:16.6250618Z AWS_DEFAULT_REGION: us-east-1 2025-03-14T05:32:16.6250902Z ##[endgroup] 2025-03-14T05:32:18.5021647Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T05:32:18.5022262Z Configure a credential helper to remove this warning. See 2025-03-14T05:32:18.5022844Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T05:32:18.5023218Z 2025-03-14T05:32:18.5023332Z Login Succeeded 2025-03-14T05:32:18.7083689Z Command completed after 1 attempt(s). 2025-03-14T05:32:18.7167228Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T05:32:18.7167762Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T05:32:18.7168220Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T05:32:18.7177690Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:18.7178065Z env: 2025-03-14T05:32:18.7178293Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:18.7178564Z ##[endgroup] 2025-03-14T05:32:18.7289174Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T05:32:18.7289734Z # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T05:32:18.7290161Z # shellcheck disable=SC2046 2025-03-14T05:32:18.7290511Z docker stop $(docker ps -q) || true 2025-03-14T05:32:18.7290870Z # Prune all of the docker images 2025-03-14T05:32:18.7291207Z docker system prune -af 2025-03-14T05:32:18.7299390Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:18.7299937Z env: 2025-03-14T05:32:18.7300163Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:18.7300438Z ##[endgroup] 2025-03-14T05:32:18.7580495Z "docker stop" requires at least 1 argument. 2025-03-14T05:32:18.7581230Z See 'docker stop --help'. 2025-03-14T05:32:18.7581601Z 2025-03-14T05:32:18.7581975Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-03-14T05:32:18.7582513Z 2025-03-14T05:32:18.7582757Z Stop one or more running containers 2025-03-14T05:32:23.7850798Z Deleted Images: 2025-03-14T05:32:23.7851176Z untagged: pytorch/manylinux2_28-builder:cuda11.8 2025-03-14T05:32:23.7851836Z untagged: pytorch/manylinux2_28-builder@sha256:c4a5bf09005ddcb1ba6397d8970a9e9919a6c70541d0d37a9699d3850dc3fb07 2025-03-14T05:32:23.7852587Z deleted: sha256:3d13f614b0ac7717e378c2c892985965d07c4e6066596e7b70ad5fd97967d1dc 2025-03-14T05:32:23.7853194Z deleted: sha256:6221f1d510438df2e371109ab977dbf4d444c6370a2290560eb4493135ecea98 2025-03-14T05:32:23.7853834Z deleted: sha256:27f4fc2b367255e2762e9d1bf32b47f3bb07d63923e98e9783730311c5cf93a4 2025-03-14T05:32:23.7854448Z deleted: sha256:4d3dc903a18bf674e1b93a26393ddb2110f3f42c0614f83c619a3c7346b1a11a 2025-03-14T05:32:23.7855079Z deleted: sha256:9d8efc8e0dda586bc0a2bcdf7fd3f6ac562a5963c3a1453a9effcc37e774bc18 2025-03-14T05:32:23.7856055Z deleted: sha256:c0d399e4f4f1bf449080b40a93e39826b44779f50d617ea70e43e767ab6a0b79 2025-03-14T05:32:23.7856671Z deleted: sha256:6d402d89497cd28c7b16abe2939d5372c1fc5097d742b2e83d77a3c15f400a04 2025-03-14T05:32:23.7857293Z deleted: sha256:a61c4e0d90bbcb4389473aaadb2c7ac7fdd46326783af5f4bf7bc2c41d7702e6 2025-03-14T05:32:23.7857916Z deleted: sha256:eaa91c17e712169ae19a05886f539ad39bd7ad2831d3321a951cf51212912493 2025-03-14T05:32:23.7858527Z deleted: sha256:37873311a6f27b655a9bdc647daf59d7c049c792fb77ab565102a140f4b53101 2025-03-14T05:32:23.7859137Z deleted: sha256:c945e869bc0059b28d48d93b2b1e9c8d3c962433a70cd422032f2445a2728a7f 2025-03-14T05:32:23.7859754Z deleted: sha256:17646da35e3e41cc406b6a730eee1d41145b638ebab31aae7d1d44ba14a346a1 2025-03-14T05:32:23.7860380Z deleted: sha256:2cf6b6910121173fd27470edabd87b88fa72c0e7169ab34642091b3e6f906ef1 2025-03-14T05:32:23.7861009Z deleted: sha256:871a68fdcb7ee9fbadf1fe46434a799bcb0de3582d3049bbbcc3ae88c6443ffb 2025-03-14T05:32:23.7861658Z deleted: sha256:d8242ba769f966822344df4caed06d0d657868d79a585fc8a6688afdde0fb718 2025-03-14T05:32:23.7862268Z deleted: sha256:26a16b3671c9607e1e89ea553bf89087b694492762d41296d2e05a97e4e7c03b 2025-03-14T05:32:23.7862892Z deleted: sha256:483ad8c0d1ecb23dec6652efa7d56b36e6ecf52b3ff199b9e7fdcd46086ef7d3 2025-03-14T05:32:23.7863516Z deleted: sha256:699c56d9c486ead4b7ec910c277f07d6a190f0761207f973c0eac670b03b7f4e 2025-03-14T05:32:23.7864141Z deleted: sha256:bdd3df1709029e7459ae583cb415ec5da0d5c37fe4ecd459ede1ccd80699775f 2025-03-14T05:32:23.7864768Z deleted: sha256:c8027c6bfa9cbbd929d71d9b9035f19096b76aec589e4e3ab39c02ed2cb9f32f 2025-03-14T05:32:23.7865400Z deleted: sha256:b30359030df1a4672ce01f801f387e38ab2e71b754b9dfb012370543f4c0daad 2025-03-14T05:32:23.7866042Z deleted: sha256:65cdb98404640cd017616669a5d3433b9f2d4474664584723aabaf55c78b9772 2025-03-14T05:32:23.7866759Z deleted: sha256:708ec990f3b527a12d3c6e780f38c202716bf1e405b49d0c6ff8aee05c81f8fb 2025-03-14T05:32:23.7867388Z deleted: sha256:e66b4c03f5bccb7aeb89814bcef211b4a14abd9d8703ed2236a3c068a5039e03 2025-03-14T05:32:23.7868009Z deleted: sha256:d6453d2d244b6f473de2fc7978d8ec923841a9295d5397a9b9029faed9684803 2025-03-14T05:32:23.7868619Z deleted: sha256:a7c877f7831dbd12863b7ed2c852c24411a8c7f2202404238eaec9cb812728b1 2025-03-14T05:32:23.7869233Z deleted: sha256:3a91e1ab9724fc472ffee9f472766c417e667635f3acac45559a7fe19ee13774 2025-03-14T05:32:23.7869839Z deleted: sha256:759856a145e5380e0fc76e62c6f9a904c88895a5a250f3408128a0966e6d8fbb 2025-03-14T05:32:23.7870445Z deleted: sha256:75d2f730541644ed380f2f7ec9afb377c1f4863003fafce00ebf93b9c7342210 2025-03-14T05:32:23.7870805Z 2025-03-14T05:32:23.7876346Z Total reclaimed space: 11.82GB 2025-03-14T05:32:23.7924404Z ##[group]Run set +e 2025-03-14T05:32:23.7924899Z set +e 2025-03-14T05:32:23.7925146Z set -x 2025-03-14T05:32:23.7925388Z  2025-03-14T05:32:23.7925658Z PT_DOMAIN=download.pytorch.org 2025-03-14T05:32:23.7926508Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-03-14T05:32:23.7927313Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-03-14T05:32:23.7927833Z # one is returned at random 2025-03-14T05:32:23.7928233Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-03-14T05:32:23.7928611Z  2025-03-14T05:32:23.7928859Z if [ -z "${RESOLVED_IP}" ]; then 2025-03-14T05:32:23.7929289Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-03-14T05:32:23.7929804Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-03-14T05:32:23.7930192Z  2025-03-14T05:32:23.7930456Z  if [ -z "${RESOLVED_IP}" ]; then 2025-03-14T05:32:23.7930849Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-03-14T05:32:23.7931215Z  exit 1 2025-03-14T05:32:23.7931465Z  fi 2025-03-14T05:32:23.7931688Z fi 2025-03-14T05:32:23.7932100Z  2025-03-14T05:32:23.7932376Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-03-14T05:32:23.7932750Z  # Clean up any old records first 2025-03-14T05:32:23.7933119Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-03-14T05:32:23.7933450Z fi 2025-03-14T05:32:23.7933674Z  2025-03-14T05:32:23.7933994Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-03-14T05:32:23.7934388Z cat /etc/hosts 2025-03-14T05:32:23.7943080Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:23.7943450Z env: 2025-03-14T05:32:23.7943677Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:23.7943960Z ##[endgroup] 2025-03-14T05:32:23.7973989Z + PT_DOMAIN=download.pytorch.org 2025-03-14T05:32:23.7980919Z ++ dig -4 +short download.pytorch.org 2025-03-14T05:32:23.7981933Z ++ tail -n1 2025-03-14T05:32:23.8347953Z + RESOLVED_IP=18.160.10.22 2025-03-14T05:32:23.8348255Z + '[' -z 18.160.10.22 ']' 2025-03-14T05:32:23.8348586Z + grep -r download.pytorch.org /etc/hosts 2025-03-14T05:32:23.8361973Z 18.160.10.28 download.pytorch.org 2025-03-14T05:32:23.8364202Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2025-03-14T05:32:23.9739100Z + echo '18.160.10.22 download.pytorch.org' 2025-03-14T05:32:23.9739552Z + sudo tee -a /etc/hosts 2025-03-14T05:32:24.1488164Z 18.160.10.22 download.pytorch.org 2025-03-14T05:32:24.1510378Z + cat /etc/hosts 2025-03-14T05:32:24.1520434Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-03-14T05:32:24.1525336Z ::1 localhost6 localhost6.localdomain6 2025-03-14T05:32:24.1525720Z 18.160.10.22 download.pytorch.org 2025-03-14T05:32:24.1673134Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-03-14T05:32:24.1673600Z with: 2025-03-14T05:32:24.1674413Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1675305Z docker-build-dir: .ci/docker 2025-03-14T05:32:24.1675611Z working-directory: . 2025-03-14T05:32:24.1675976Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.1676380Z force-push: false 2025-03-14T05:32:24.1676628Z env: 2025-03-14T05:32:24.1676859Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:24.1677133Z ##[endgroup] 2025-03-14T05:32:24.1717767Z ##[group]Run set -ex 2025-03-14T05:32:24.1718070Z set -ex 2025-03-14T05:32:24.1718315Z  2025-03-14T05:32:24.1718711Z # If the docker build directory or the build script doesn't exist, the action will 2025-03-14T05:32:24.1719399Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-03-14T05:32:24.1720148Z # job could then download the pre-built image as usual 2025-03-14T05:32:24.1720664Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2025-03-14T05:32:24.1721138Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1721597Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1722006Z  2025-03-14T05:32:24.1722379Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2025-03-14T05:32:24.1722819Z  exit 0 2025-03-14T05:32:24.1723061Z else 2025-03-14T05:32:24.1723343Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1723678Z fi 2025-03-14T05:32:24.1723895Z  2025-03-14T05:32:24.1724248Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-03-14T05:32:24.1724838Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-03-14T05:32:24.1725364Z  # use it as it is, but first let's extract the tag 2025-03-14T05:32:24.1725852Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-03-14T05:32:24.1726697Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1727190Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1727600Z else 2025-03-14T05:32:24.1727922Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-03-14T05:32:24.1728379Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1729005Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1729537Z fi 2025-03-14T05:32:24.1738472Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:24.1738847Z env: 2025-03-14T05:32:24.1739076Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:24.1739349Z REPO_NAME: pytorch 2025-03-14T05:32:24.1740185Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1741062Z DOCKER_BUILD_DIR: .ci/docker 2025-03-14T05:32:24.1741438Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.1741822Z ##[endgroup] 2025-03-14T05:32:24.1770761Z + [[ ! -d .ci/docker ]] 2025-03-14T05:32:24.1771119Z + [[ ! -f .ci/docker/build.sh ]] 2025-03-14T05:32:24.1771437Z + echo skip=false 2025-03-14T05:32:24.1772759Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-03-14T05:32:24.1778917Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1780028Z ++ awk -F '[:,]' '{print $2}' 2025-03-14T05:32:24.1806028Z + DOCKER_TAG=aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1806635Z + echo docker-tag=aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1807921Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1844037Z ##[group]Run set +e 2025-03-14T05:32:24.1844342Z set +e 2025-03-14T05:32:24.1844594Z set -x 2025-03-14T05:32:24.1844835Z  2025-03-14T05:32:24.1845064Z login() { 2025-03-14T05:32:24.1845554Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-03-14T05:32:24.1846222Z } 2025-03-14T05:32:24.1846452Z  2025-03-14T05:32:24.1846684Z retry () { 2025-03-14T05:32:24.1846970Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-03-14T05:32:24.1847293Z } 2025-03-14T05:32:24.1847524Z  2025-03-14T05:32:24.1847775Z retry login "${DOCKER_REGISTRY}" 2025-03-14T05:32:24.1848094Z  2025-03-14T05:32:24.1848332Z START_TIME=$(date +%s) 2025-03-14T05:32:24.1848637Z # Wait up to 120 minutes 2025-03-14T05:32:24.1849010Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-03-14T05:32:24.1849498Z  # Check if image already exists, if it does then skip building it 2025-03-14T05:32:24.1849981Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-03-14T05:32:24.1850344Z  exit 0 2025-03-14T05:32:24.1850594Z  fi 2025-03-14T05:32:24.1850823Z  2025-03-14T05:32:24.1851213Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-03-14T05:32:24.1851861Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-03-14T05:32:24.1852504Z  # latter, it will wait for the Docker images to become available before continuing 2025-03-14T05:32:24.1853013Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-03-14T05:32:24.1853422Z  # It's a Docker build job, let's build the image 2025-03-14T05:32:24.1853776Z  break 2025-03-14T05:32:24.1854038Z  else 2025-03-14T05:32:24.1854396Z  # It's a regular build job, wait for the image to become available 2025-03-14T05:32:24.1854814Z  sleep 300 2025-03-14T05:32:24.1855078Z  fi 2025-03-14T05:32:24.1855310Z done 2025-03-14T05:32:24.1855570Z  2025-03-14T05:32:24.1855973Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-03-14T05:32:24.1856549Z # be empty. The default action would be to continue rebuild the image 2025-03-14T05:32:24.1857070Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-03-14T05:32:24.1857532Z  # if we're on the base branch then use the parent commit 2025-03-14T05:32:24.1857946Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-03-14T05:32:24.1858281Z else 2025-03-14T05:32:24.1858638Z  # otherwise we're on a PR, so use the most recent base commit 2025-03-14T05:32:24.1859128Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-03-14T05:32:24.1859505Z fi 2025-03-14T05:32:24.1859741Z  2025-03-14T05:32:24.1859995Z if [[ -z "${MERGE_BASE}" ]]; then 2025-03-14T05:32:24.1860367Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1860708Z  2025-03-14T05:32:24.1861332Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-03-14T05:32:24.1861875Z  exit 0 2025-03-14T05:32:24.1862118Z fi 2025-03-14T05:32:24.1862346Z  2025-03-14T05:32:24.1862661Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-03-14T05:32:24.1863323Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-03-14T05:32:24.1863882Z  exit 1 2025-03-14T05:32:24.1864125Z fi 2025-03-14T05:32:24.1864343Z  2025-03-14T05:32:24.1864713Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-03-14T05:32:24.1865350Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-03-14T05:32:24.1865923Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-03-14T05:32:24.1866705Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-03-14T05:32:24.1867526Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-03-14T05:32:24.1867974Z fi 2025-03-14T05:32:24.1868213Z  2025-03-14T05:32:24.1868488Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-03-14T05:32:24.1876834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:24.1877206Z env: 2025-03-14T05:32:24.1877442Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:24.1877728Z DOCKER_BUILD_DIR: .ci/docker 2025-03-14T05:32:24.1878079Z BASE_REVISION: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:32:24.1878994Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1879920Z DOCKER_TAG: aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.1880367Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.1880765Z DOCKER_PUSH: 2025-03-14T05:32:24.1881013Z ##[endgroup] 2025-03-14T05:32:24.1907962Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.1908413Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.1911264Z + aws ecr get-login-password --region us-east-1 2025-03-14T05:32:24.1912189Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:24.7024080Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T05:32:24.7024698Z Configure a credential helper to remove this warning. See 2025-03-14T05:32:24.7025274Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T05:32:24.7025665Z 2025-03-14T05:32:24.7025954Z Login Succeeded 2025-03-14T05:32:24.7052196Z ++ date +%s 2025-03-14T05:32:24.7065358Z + START_TIME=1741930344 2025-03-14T05:32:24.7069240Z ++ date +%s 2025-03-14T05:32:24.7080766Z + [[ 1741923144 -lt 1741930344 ]] 2025-03-14T05:32:24.7082243Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:24.9182100Z { 2025-03-14T05:32:24.9182410Z "schemaVersion": 2, 2025-03-14T05:32:24.9182937Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-03-14T05:32:24.9183394Z "config": { 2025-03-14T05:32:24.9183743Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-03-14T05:32:24.9184149Z "size": 52935, 2025-03-14T05:32:24.9184572Z "digest": "sha256:9f77b6c3483857c0bff989bce733b5bd5d6fc70a10591ed0f8d1de80d0e77bfd" 2025-03-14T05:32:24.9185157Z }, 2025-03-14T05:32:24.9185448Z "layers": [ 2025-03-14T05:32:24.9185749Z { 2025-03-14T05:32:24.9186210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9186965Z "size": 28583948, 2025-03-14T05:32:24.9187832Z "digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1" 2025-03-14T05:32:24.9188327Z }, 2025-03-14T05:32:24.9188573Z { 2025-03-14T05:32:24.9188969Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9189385Z "size": 7964619, 2025-03-14T05:32:24.9189801Z "digest": "sha256:49e139a3d6c2f1801aa0cea1eb34e57c5314065b679325df205026fb175383b8" 2025-03-14T05:32:24.9190261Z }, 2025-03-14T05:32:24.9190463Z { 2025-03-14T05:32:24.9190867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9191280Z "size": 57379226, 2025-03-14T05:32:24.9191706Z "digest": "sha256:a14844c8c51f98ebee2b5eda8bff8742dd804385e1dbbeb56928946113000293" 2025-03-14T05:32:24.9192166Z }, 2025-03-14T05:32:24.9192372Z { 2025-03-14T05:32:24.9192706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9193118Z "size": 187, 2025-03-14T05:32:24.9193541Z "digest": "sha256:18fb524087fbed3307d23afde734ef1df452f55e7a22e28b56ce1e6eb9c6b3d9" 2025-03-14T05:32:24.9194150Z }, 2025-03-14T05:32:24.9194346Z { 2025-03-14T05:32:24.9194687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9195103Z "size": 6885, 2025-03-14T05:32:24.9195525Z "digest": "sha256:efd686a7b2c8bb052a3bb919973c4bee33c2bbcff08fd431362b90c9469fbc4d" 2025-03-14T05:32:24.9195992Z }, 2025-03-14T05:32:24.9196195Z { 2025-03-14T05:32:24.9196528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9196941Z "size": 1438282733, 2025-03-14T05:32:24.9197362Z "digest": "sha256:52c648e213347a29001a45b2c4f834cc60621a3d6ab68fa06ebd2075f6487ec5" 2025-03-14T05:32:24.9197813Z }, 2025-03-14T05:32:24.9198015Z { 2025-03-14T05:32:24.9198346Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9198756Z "size": 62694, 2025-03-14T05:32:24.9199175Z "digest": "sha256:56e384e4e5aa7ebddf31686fc6ae62ef19fdc5311bbea8186563fc2aeca2fb52" 2025-03-14T05:32:24.9199641Z }, 2025-03-14T05:32:24.9199847Z { 2025-03-14T05:32:24.9200173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9200581Z "size": 1684, 2025-03-14T05:32:24.9200997Z "digest": "sha256:fb71b792ec6c458c338cace5ce42f653e4ae4b24442a49b7c47af3a38342b5bc" 2025-03-14T05:32:24.9201458Z }, 2025-03-14T05:32:24.9201658Z { 2025-03-14T05:32:24.9201996Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9202404Z "size": 1523, 2025-03-14T05:32:24.9202809Z "digest": "sha256:5509576f2693abf94a50a70a59bbf8b519f20705f147d495ddde85981c8189fa" 2025-03-14T05:32:24.9203258Z }, 2025-03-14T05:32:24.9203459Z { 2025-03-14T05:32:24.9203793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9204208Z "size": 2515216700, 2025-03-14T05:32:24.9204629Z "digest": "sha256:1e6c6f2d245956001435a5df27fd4defea08affa7f6c22252464f758447db401" 2025-03-14T05:32:24.9205084Z }, 2025-03-14T05:32:24.9205288Z { 2025-03-14T05:32:24.9205636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9206085Z "size": 86619, 2025-03-14T05:32:24.9206482Z "digest": "sha256:a1fe8922734d371e0c0fd5b04183758331e08f45553720ecff4b062824e0c8e1" 2025-03-14T05:32:24.9206932Z }, 2025-03-14T05:32:24.9216535Z { 2025-03-14T05:32:24.9216966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9217610Z "size": 1895, 2025-03-14T05:32:24.9218183Z "digest": "sha256:b6dbf6a349bb138f3c043a63744a5726a8cabbdb33a101bcb2974bd2443576b5" 2025-03-14T05:32:24.9218759Z }, 2025-03-14T05:32:24.9218968Z { 2025-03-14T05:32:24.9219308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9219741Z "size": 245809802, 2025-03-14T05:32:24.9220345Z "digest": "sha256:73d0341633364e49fcc64aa471fe942fbe8d160db68f9a62d8efdf396ba61586" 2025-03-14T05:32:24.9221023Z }, 2025-03-14T05:32:24.9221322Z { 2025-03-14T05:32:24.9221970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9222626Z "size": 703, 2025-03-14T05:32:24.9223255Z "digest": "sha256:f2a6fb332e8e2f85d989521b6e0ebf5cfb8fb18d0c3f53686ea7a989f3e2db28" 2025-03-14T05:32:24.9223984Z }, 2025-03-14T05:32:24.9224297Z { 2025-03-14T05:32:24.9224816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9225462Z "size": 1255, 2025-03-14T05:32:24.9226079Z "digest": "sha256:991f90b489b149166bbc0e7ac9d8bef13697c5d4934652e4898847e9e3399277" 2025-03-14T05:32:24.9227232Z }, 2025-03-14T05:32:24.9227547Z { 2025-03-14T05:32:24.9228054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9228704Z "size": 485, 2025-03-14T05:32:24.9229349Z "digest": "sha256:68199204396f4d8cd760d149b2bec7f6a58069074b1ea88ca6fc9bb55a6309bd" 2025-03-14T05:32:24.9230080Z }, 2025-03-14T05:32:24.9230397Z { 2025-03-14T05:32:24.9230919Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9231713Z "size": 91727383, 2025-03-14T05:32:24.9232215Z "digest": "sha256:8d533b261ba2a83360001b0d5a4fa23e9fb5ef1d45b5f570841c7e88f42fa0c4" 2025-03-14T05:32:24.9232682Z }, 2025-03-14T05:32:24.9232886Z { 2025-03-14T05:32:24.9233226Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9233641Z "size": 3720, 2025-03-14T05:32:24.9234057Z "digest": "sha256:9552b75545b1f7ea035b2c6a9bc5c9110aac689a5a82402714a1e908ab7b87fb" 2025-03-14T05:32:24.9234521Z }, 2025-03-14T05:32:24.9234727Z { 2025-03-14T05:32:24.9235072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9235491Z "size": 1860, 2025-03-14T05:32:24.9235960Z "digest": "sha256:330a44046cfdafc7f173b0af2a21c36240c683608cba1fadae7ea4a893c50f13" 2025-03-14T05:32:24.9236426Z }, 2025-03-14T05:32:24.9236634Z { 2025-03-14T05:32:24.9236973Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9237398Z "size": 701, 2025-03-14T05:32:24.9237829Z "digest": "sha256:fb9c93639b4043da5bee2c9144ca9cfdc4d78f4ea398226d0543eb383872ac82" 2025-03-14T05:32:24.9238301Z }, 2025-03-14T05:32:24.9238513Z { 2025-03-14T05:32:24.9238857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9239279Z "size": 477, 2025-03-14T05:32:24.9239698Z "digest": "sha256:a5d1ad88496e5092d20986159ded209dc66ca7ecf26904a3be763c50eb5f3b37" 2025-03-14T05:32:24.9240175Z }, 2025-03-14T05:32:24.9240374Z { 2025-03-14T05:32:24.9240715Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9241138Z "size": 2923135652, 2025-03-14T05:32:24.9241587Z "digest": "sha256:4e2ec52a1b4f3b459a27ea68f5fe337f0d74845bc9553c5be5ce181dbf84498f" 2025-03-14T05:32:24.9242070Z }, 2025-03-14T05:32:24.9242281Z { 2025-03-14T05:32:24.9242624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9243041Z "size": 380, 2025-03-14T05:32:24.9243452Z "digest": "sha256:1576583c5ff83f09a227a083342d256062f047fe6fc466f2553f08aba32e028a" 2025-03-14T05:32:24.9243910Z }, 2025-03-14T05:32:24.9244117Z { 2025-03-14T05:32:24.9244456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9244867Z "size": 12922, 2025-03-14T05:32:24.9245290Z "digest": "sha256:ae76494b3f0faf177f07b531d05493b8f6e1ba6a9b24b84acce1289bb4251693" 2025-03-14T05:32:24.9245761Z }, 2025-03-14T05:32:24.9245975Z { 2025-03-14T05:32:24.9246320Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9246738Z "size": 863, 2025-03-14T05:32:24.9247154Z "digest": "sha256:96b47e94ca2f2cb3a745f0807c540c56a831420099db6b9f1f132b6999f8f41e" 2025-03-14T05:32:24.9247620Z }, 2025-03-14T05:32:24.9247834Z { 2025-03-14T05:32:24.9248173Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9248597Z "size": 106, 2025-03-14T05:32:24.9249207Z "digest": "sha256:e26a64ee2eebd4c0bf3f79062035d4158a72d0c7875e0e26366ef1c6b92d8f53" 2025-03-14T05:32:24.9249682Z }, 2025-03-14T05:32:24.9249891Z { 2025-03-14T05:32:24.9250227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9250657Z "size": 503, 2025-03-14T05:32:24.9251070Z "digest": "sha256:97f84a5ed9bbc76fc24d305ea898bc6fdb0681d4bb55f51e6c0c9015f93d0374" 2025-03-14T05:32:24.9251553Z }, 2025-03-14T05:32:24.9251768Z { 2025-03-14T05:32:24.9252112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9252534Z "size": 121477401, 2025-03-14T05:32:24.9252962Z "digest": "sha256:1cc35cfdfb65b588b5ef88828f4b4ec601c610c9624b791578d1419558757700" 2025-03-14T05:32:24.9253422Z }, 2025-03-14T05:32:24.9253625Z { 2025-03-14T05:32:24.9253961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9254378Z "size": 109, 2025-03-14T05:32:24.9254809Z "digest": "sha256:cfe527e10a79566dea8ccbb08dea12bd3a63e6d00eaabec0bfaf591b411e621e" 2025-03-14T05:32:24.9255369Z }, 2025-03-14T05:32:24.9255576Z { 2025-03-14T05:32:24.9255926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9256351Z "size": 490, 2025-03-14T05:32:24.9256798Z "digest": "sha256:528c12012cc2e036e2fc7b09e0dd5a1a5662efbaaff9ecf923e58df95061310c" 2025-03-14T05:32:24.9257278Z }, 2025-03-14T05:32:24.9257480Z { 2025-03-14T05:32:24.9257831Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9258263Z "size": 295, 2025-03-14T05:32:24.9258684Z "digest": "sha256:3a0e7232565e118d798825dae4723c348544900139cc3b12c462bfd6c772c5e3" 2025-03-14T05:32:24.9259146Z }, 2025-03-14T05:32:24.9259365Z { 2025-03-14T05:32:24.9259721Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9260153Z "size": 103, 2025-03-14T05:32:24.9260578Z "digest": "sha256:ac1e95ec684d244600badaef01b48b1aec2fc11855bc018195f75b941706cf01" 2025-03-14T05:32:24.9261061Z }, 2025-03-14T05:32:24.9261280Z { 2025-03-14T05:32:24.9261629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9262060Z "size": 1473, 2025-03-14T05:32:24.9262498Z "digest": "sha256:f46926cb3922c5be4b7a255be3cfaff257cb564f9c18145f7e085bd94d3a0b88" 2025-03-14T05:32:24.9262975Z }, 2025-03-14T05:32:24.9263186Z { 2025-03-14T05:32:24.9263521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9263946Z "size": 427757051, 2025-03-14T05:32:24.9264379Z "digest": "sha256:dd5d197efeb1ca410b03078ebb81189d2f54158d061f744b06567c35bb3682b6" 2025-03-14T05:32:24.9264849Z }, 2025-03-14T05:32:24.9265057Z { 2025-03-14T05:32:24.9265398Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9265840Z "size": 164, 2025-03-14T05:32:24.9266279Z "digest": "sha256:6ec4020a8fae159a0e15b174c108a3581e7723a2eeeb5ec2459cb0ca071de284" 2025-03-14T05:32:24.9266850Z }, 2025-03-14T05:32:24.9267066Z { 2025-03-14T05:32:24.9267409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9267827Z "size": 802, 2025-03-14T05:32:24.9268235Z "digest": "sha256:39e1750b6399b91d360c3e1fe08ae1084d9f42e99d531e79e32903ff6a9b9dd3" 2025-03-14T05:32:24.9268695Z }, 2025-03-14T05:32:24.9268897Z { 2025-03-14T05:32:24.9269231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9269634Z "size": 35861010, 2025-03-14T05:32:24.9270063Z "digest": "sha256:d17cf48b138eea7b80eac09fc6eb839a1ca5c79708e6494438018dd7ee05f234" 2025-03-14T05:32:24.9270523Z }, 2025-03-14T05:32:24.9270728Z { 2025-03-14T05:32:24.9271065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9271495Z "size": 104, 2025-03-14T05:32:24.9271909Z "digest": "sha256:86c6b7977b8bd142d8392a15232d816649002d76609a2671ce99c1483f9bed98" 2025-03-14T05:32:24.9272378Z }, 2025-03-14T05:32:24.9272588Z { 2025-03-14T05:32:24.9273033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9273463Z "size": 425, 2025-03-14T05:32:24.9273890Z "digest": "sha256:b1829fd86eff785e1cd2a1bc5b43a14eea96b280ab21959d8a655a96551e46f2" 2025-03-14T05:32:24.9274379Z }, 2025-03-14T05:32:24.9274597Z { 2025-03-14T05:32:24.9274953Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9275391Z "size": 20262151, 2025-03-14T05:32:24.9275835Z "digest": "sha256:4c8da014a48e84abbbb0c25d4ceeb509d0d15afa37fd69cf5da6491ee935933d" 2025-03-14T05:32:24.9276323Z }, 2025-03-14T05:32:24.9276536Z { 2025-03-14T05:32:24.9276886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9277316Z "size": 641, 2025-03-14T05:32:24.9277741Z "digest": "sha256:3e933db6894bc779ff9fe5413678715e3ce6f673c7501c6fc46811e68db0e4b1" 2025-03-14T05:32:24.9278202Z }, 2025-03-14T05:32:24.9278409Z { 2025-03-14T05:32:24.9278761Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9279268Z "size": 701, 2025-03-14T05:32:24.9279698Z "digest": "sha256:fb9c93639b4043da5bee2c9144ca9cfdc4d78f4ea398226d0543eb383872ac82" 2025-03-14T05:32:24.9280184Z }, 2025-03-14T05:32:24.9280406Z { 2025-03-14T05:32:24.9280758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9281185Z "size": 143, 2025-03-14T05:32:24.9281636Z "digest": "sha256:1bef7a4bcd0a0fc516099cc94ffc09fe3464bc5dafbd7cecf1e4e128e7e7d620" 2025-03-14T05:32:24.9282127Z }, 2025-03-14T05:32:24.9282329Z { 2025-03-14T05:32:24.9282683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9283111Z "size": 135, 2025-03-14T05:32:24.9283546Z "digest": "sha256:771ad9f2789ae02d673ac942ce9f28a1dc1d5caad67d96c6d328bf71faa65f43" 2025-03-14T05:32:24.9284029Z }, 2025-03-14T05:32:24.9284243Z { 2025-03-14T05:32:24.9284597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9285039Z "size": 5220115140, 2025-03-14T05:32:24.9285487Z "digest": "sha256:907000cb43f1dc9f5a146723107b2a624def679ac73b1cab2aa5725ab46cf6ab" 2025-03-14T05:32:24.9285967Z }, 2025-03-14T05:32:24.9286185Z { 2025-03-14T05:32:24.9286533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9286970Z "size": 195, 2025-03-14T05:32:24.9287398Z "digest": "sha256:540fab0ce5376c101d27d4fb240509b69d13325a4064c74d2ca83c982d9ce2d2" 2025-03-14T05:32:24.9287875Z }, 2025-03-14T05:32:24.9288093Z { 2025-03-14T05:32:24.9288425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9288854Z "size": 565, 2025-03-14T05:32:24.9289299Z "digest": "sha256:ca5aa8a8bb00feae0eddc40332de18eeb052387042fdeed9f92a9bf94de7b492" 2025-03-14T05:32:24.9289788Z }, 2025-03-14T05:32:24.9290011Z { 2025-03-14T05:32:24.9290348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9290769Z "size": 43163409, 2025-03-14T05:32:24.9291225Z "digest": "sha256:19197f3f8a266e9f14f9ad0bd506be2e3f72eadcb52e5acfc23d594b3ad6e7e7" 2025-03-14T05:32:24.9291718Z }, 2025-03-14T05:32:24.9291933Z { 2025-03-14T05:32:24.9292277Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9292708Z "size": 106, 2025-03-14T05:32:24.9293134Z "digest": "sha256:50b63f41fa174e3b4faec14dc8090ac0a368fd9800231525eff41448616d21ce" 2025-03-14T05:32:24.9293613Z }, 2025-03-14T05:32:24.9293825Z { 2025-03-14T05:32:24.9294175Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9294596Z "size": 1401, 2025-03-14T05:32:24.9295052Z "digest": "sha256:26c4a7b9d650a4992ce4065cf18f3ca13a0a8a5a1fc86c7cce58d1bad230a6bf" 2025-03-14T05:32:24.9295530Z }, 2025-03-14T05:32:24.9295749Z { 2025-03-14T05:32:24.9296099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9296560Z "size": 701, 2025-03-14T05:32:24.9297091Z "digest": "sha256:fb9c93639b4043da5bee2c9144ca9cfdc4d78f4ea398226d0543eb383872ac82" 2025-03-14T05:32:24.9297577Z }, 2025-03-14T05:32:24.9297783Z { 2025-03-14T05:32:24.9298125Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9298545Z "size": 140, 2025-03-14T05:32:24.9298963Z "digest": "sha256:712e4e3f44e2c62bb49c2de099d220f2482285fbf9b8ff9887375ce35d21dd59" 2025-03-14T05:32:24.9299434Z }, 2025-03-14T05:32:24.9299643Z { 2025-03-14T05:32:24.9299985Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9300401Z "size": 120, 2025-03-14T05:32:24.9300820Z "digest": "sha256:a03d53d11afa8e326d5f7d12154c7e6896b667318b30e0974f98d40fc1126f7a" 2025-03-14T05:32:24.9301281Z }, 2025-03-14T05:32:24.9301490Z { 2025-03-14T05:32:24.9301835Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9302259Z "size": 6570494848, 2025-03-14T05:32:24.9302686Z "digest": "sha256:298cf77443f2d1e60b5692100904240767762489ae61dc57f65645a460b8e445" 2025-03-14T05:32:24.9303221Z }, 2025-03-14T05:32:24.9303426Z { 2025-03-14T05:32:24.9303762Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9304177Z "size": 174, 2025-03-14T05:32:24.9304615Z "digest": "sha256:6debb4801aa588df1f4980615765bbdd4ed2fae729eb7b43fc08c0f01a0c4373" 2025-03-14T05:32:24.9305096Z }, 2025-03-14T05:32:24.9305308Z { 2025-03-14T05:32:24.9305654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9306091Z "size": 908, 2025-03-14T05:32:24.9306514Z "digest": "sha256:c127e62d25664923c9ee243cc44eb7b8a2c0c7b38f35b4686597e22dea7aa463" 2025-03-14T05:32:24.9307064Z }, 2025-03-14T05:32:24.9307267Z { 2025-03-14T05:32:24.9307607Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9308021Z "size": 701, 2025-03-14T05:32:24.9308439Z "digest": "sha256:fb9c93639b4043da5bee2c9144ca9cfdc4d78f4ea398226d0543eb383872ac82" 2025-03-14T05:32:24.9308926Z }, 2025-03-14T05:32:24.9309146Z { 2025-03-14T05:32:24.9309497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9309916Z "size": 135, 2025-03-14T05:32:24.9310338Z "digest": "sha256:7d113dac572033116182c6d382508a8d2c4c618b000861a12be543a170b6b971" 2025-03-14T05:32:24.9310805Z }, 2025-03-14T05:32:24.9311027Z { 2025-03-14T05:32:24.9311375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9311805Z "size": 32, 2025-03-14T05:32:24.9312231Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T05:32:24.9312713Z }, 2025-03-14T05:32:24.9312919Z { 2025-03-14T05:32:24.9313253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9313665Z "size": 156, 2025-03-14T05:32:24.9314079Z "digest": "sha256:41636a0cf6cfcfc036f6c93f2dba4935deaa8c58ed41775265efd8845de05349" 2025-03-14T05:32:24.9314550Z }, 2025-03-14T05:32:24.9314754Z { 2025-03-14T05:32:24.9315111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9315527Z "size": 1899, 2025-03-14T05:32:24.9315948Z "digest": "sha256:6b28eaf423002704e1555e6880afbdaef8bf2d205b1e88d4ae19a1076c3e695b" 2025-03-14T05:32:24.9316412Z }, 2025-03-14T05:32:24.9316622Z { 2025-03-14T05:32:24.9316956Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9317376Z "size": 196365711, 2025-03-14T05:32:24.9317799Z "digest": "sha256:8836075b6cd4d8b767419a7c40c32d3951e1179982c397b52498e2befb99fb3d" 2025-03-14T05:32:24.9318246Z }, 2025-03-14T05:32:24.9318452Z { 2025-03-14T05:32:24.9318785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9319207Z "size": 163, 2025-03-14T05:32:24.9319620Z "digest": "sha256:794e80013542eeb7620e5177ada4324d00948f1bf001a3bcbd5c9a332e167747" 2025-03-14T05:32:24.9320075Z }, 2025-03-14T05:32:24.9320283Z { 2025-03-14T05:32:24.9320726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9321148Z "size": 7943, 2025-03-14T05:32:24.9321560Z "digest": "sha256:50f65d9ffc32655e9576df0c73d072777457306525900e61715903c98210f8d7" 2025-03-14T05:32:24.9322029Z }, 2025-03-14T05:32:24.9322248Z { 2025-03-14T05:32:24.9322596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9323017Z "size": 8071, 2025-03-14T05:32:24.9323455Z "digest": "sha256:eebbd71eb5f766bc8ba4b9d1a76afbab56e97aca1b474a27e7763bca7b5e8e40" 2025-03-14T05:32:24.9323946Z }, 2025-03-14T05:32:24.9324157Z { 2025-03-14T05:32:24.9324504Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9324922Z "size": 302, 2025-03-14T05:32:24.9325339Z "digest": "sha256:6646f0ede462a8d23e5b8f2b84778c2ec51e3ead5d90b9370741823fccbf4192" 2025-03-14T05:32:24.9325802Z }, 2025-03-14T05:32:24.9325998Z { 2025-03-14T05:32:24.9326685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9327288Z "size": 7630989, 2025-03-14T05:32:24.9327756Z "digest": "sha256:8552cd082ff68ef2fae8cd8af143e3df9e48d2f7a1b343bc18f035006073d6e0" 2025-03-14T05:32:24.9328297Z }, 2025-03-14T05:32:24.9328514Z { 2025-03-14T05:32:24.9328892Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9329357Z "size": 108, 2025-03-14T05:32:24.9329832Z "digest": "sha256:cfc32c95496a06d94e6b67393056faf9a957ed50b4debe47d6a91d4aae102e2a" 2025-03-14T05:32:24.9330365Z }, 2025-03-14T05:32:24.9330580Z { 2025-03-14T05:32:24.9330950Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9331420Z "size": 54145659, 2025-03-14T05:32:24.9331901Z "digest": "sha256:14d81158e9c4ef27e0dec6dc05b8cd3be7101f4be3212ddbec3c2bf397e5c1b3" 2025-03-14T05:32:24.9332377Z }, 2025-03-14T05:32:24.9332585Z { 2025-03-14T05:32:24.9332933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9333356Z "size": 495, 2025-03-14T05:32:24.9333769Z "digest": "sha256:1f7ab46ead9545fb8e2b5cd879a5e4a177415a84139749e9af2dfa907adb42c0" 2025-03-14T05:32:24.9334237Z }, 2025-03-14T05:32:24.9334442Z { 2025-03-14T05:32:24.9334779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9335196Z "size": 1179582145, 2025-03-14T05:32:24.9335626Z "digest": "sha256:49ef4891eff9d3fb31752716a248dae0d7e39d9fb76afe324557bcc49a0a68d8" 2025-03-14T05:32:24.9336093Z }, 2025-03-14T05:32:24.9336300Z { 2025-03-14T05:32:24.9336646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9337062Z "size": 106, 2025-03-14T05:32:24.9337473Z "digest": "sha256:b4c737e437c344480302d7355bc9aa5ba77e267b92ee294bac9e4a28e83a9926" 2025-03-14T05:32:24.9337931Z }, 2025-03-14T05:32:24.9338127Z { 2025-03-14T05:32:24.9338463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9338876Z "size": 613, 2025-03-14T05:32:24.9339288Z "digest": "sha256:c6d1e46b29684c2e7b8983020377de47a4cb34e3c338c2811a882c5471992f75" 2025-03-14T05:32:24.9339750Z }, 2025-03-14T05:32:24.9339952Z { 2025-03-14T05:32:24.9340289Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9340704Z "size": 317359197, 2025-03-14T05:32:24.9341120Z "digest": "sha256:c754fb44357294111b5d35978291ccb879ba71a9c2f00b19b829fe5e5c203b8a" 2025-03-14T05:32:24.9341570Z }, 2025-03-14T05:32:24.9341769Z { 2025-03-14T05:32:24.9342105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9342515Z "size": 111, 2025-03-14T05:32:24.9342932Z "digest": "sha256:cfb5071726ef9c3c889d1aaea6a043eaff5227ef2e6c24bf2fd1961828c0cbef" 2025-03-14T05:32:24.9343396Z }, 2025-03-14T05:32:24.9343601Z { 2025-03-14T05:32:24.9343926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9344335Z "size": 529, 2025-03-14T05:32:24.9344867Z "digest": "sha256:8be47e434613a3148316410ad09dfc7f78b8c38817b3430a011322c61bdf4b91" 2025-03-14T05:32:24.9345323Z }, 2025-03-14T05:32:24.9345524Z { 2025-03-14T05:32:24.9345862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9346280Z "size": 26099, 2025-03-14T05:32:24.9346766Z "digest": "sha256:a68c1967b44da96765aa1dde9b11dc1586ccf4b4282347a52739d85bdefce7b1" 2025-03-14T05:32:24.9347229Z }, 2025-03-14T05:32:24.9347438Z { 2025-03-14T05:32:24.9347772Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9348184Z "size": 106, 2025-03-14T05:32:24.9348605Z "digest": "sha256:985724505cd9e9e2e0cf8c2b38deb74cf3c395aad4e1f4c4c7e0ba8a8a072620" 2025-03-14T05:32:24.9349076Z }, 2025-03-14T05:32:24.9349284Z { 2025-03-14T05:32:24.9349629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9350051Z "size": 32, 2025-03-14T05:32:24.9350470Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T05:32:24.9351033Z }, 2025-03-14T05:32:24.9351244Z { 2025-03-14T05:32:24.9351591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9352010Z "size": 32, 2025-03-14T05:32:24.9352497Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T05:32:24.9353046Z }, 2025-03-14T05:32:24.9360024Z { 2025-03-14T05:32:24.9360376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9360791Z "size": 32, 2025-03-14T05:32:24.9361200Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T05:32:24.9361665Z }, 2025-03-14T05:32:24.9361867Z { 2025-03-14T05:32:24.9362198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-03-14T05:32:24.9362609Z "size": 32, 2025-03-14T05:32:24.9363014Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-03-14T05:32:24.9363470Z } 2025-03-14T05:32:24.9363687Z ] 2025-03-14T05:32:24.9363885Z } 2025-03-14T05:32:24.9364097Z + exit 0 2025-03-14T05:32:24.9401017Z ##[group]Run set -eux 2025-03-14T05:32:24.9401305Z set -eux 2025-03-14T05:32:24.9402124Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-03-14T05:32:24.9412012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:24.9412381Z env: 2025-03-14T05:32:24.9412605Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:24.9412871Z ##[endgroup] 2025-03-14T05:32:24.9445087Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-03-14T05:32:24.9445576Z + jq --raw-output .SecretString 2025-03-14T05:32:24.9447130Z + jq -r .docker_hub_readonly_token 2025-03-14T05:32:24.9448707Z + docker login --username pytorchbot --password-stdin 2025-03-14T05:32:25.5302017Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T05:32:25.5303473Z Configure a credential helper to remove this warning. See 2025-03-14T05:32:25.5304771Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T05:32:25.5305771Z 2025-03-14T05:32:25.5305942Z Login Succeeded 2025-03-14T05:32:25.5404624Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2025-03-14T05:32:25.5405016Z tag=${ECR_DOCKER_IMAGE##*/} 2025-03-14T05:32:25.5405429Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-03-14T05:32:25.5414429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:25.5414817Z env: 2025-03-14T05:32:25.5415057Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:25.5415901Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:25.5416768Z ##[endgroup] 2025-03-14T05:32:25.5447764Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks-aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:25.5500617Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-03-14T05:32:25.5501050Z with: 2025-03-14T05:32:25.5501842Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:25.5502789Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:25.5503173Z env: 2025-03-14T05:32:25.5503401Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:25.5503674Z ##[endgroup] 2025-03-14T05:32:25.5540868Z ##[group]Run set -x 2025-03-14T05:32:25.5541139Z set -x 2025-03-14T05:32:25.5541376Z set +e 2025-03-14T05:32:25.5541609Z  2025-03-14T05:32:25.5541829Z login() { 2025-03-14T05:32:25.5542301Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-03-14T05:32:25.5542986Z } 2025-03-14T05:32:25.5543211Z  2025-03-14T05:32:25.5543459Z retry () { 2025-03-14T05:32:25.5543754Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-03-14T05:32:25.5544076Z } 2025-03-14T05:32:25.5544301Z  2025-03-14T05:32:25.5544562Z retry login "${DOCKER_REGISTRY}" 2025-03-14T05:32:25.5544873Z  2025-03-14T05:32:25.5545097Z set -e 2025-03-14T05:32:25.5545454Z # ignore output since only exit code is used for conditional 2025-03-14T05:32:25.5546003Z # only pull docker image if it's not available locally 2025-03-14T05:32:25.5546618Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-03-14T05:32:25.5547116Z  retry docker pull "${DOCKER_IMAGE}" 2025-03-14T05:32:25.5547452Z fi 2025-03-14T05:32:25.5555573Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:32:25.5555955Z env: 2025-03-14T05:32:25.5556192Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:32:25.5557079Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:25.5558030Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:25.5558432Z ##[endgroup] 2025-03-14T05:32:25.5584942Z + set +e 2025-03-14T05:32:25.5585657Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:25.5586250Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:25.5588382Z + aws ecr get-login-password --region us-east-1 2025-03-14T05:32:25.5590409Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-03-14T05:32:26.0650685Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-03-14T05:32:26.0652276Z Configure a credential helper to remove this warning. See 2025-03-14T05:32:26.0653670Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-03-14T05:32:26.0654439Z 2025-03-14T05:32:26.0654642Z Login Succeeded 2025-03-14T05:32:26.0673611Z + set -e 2025-03-14T05:32:26.0674744Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:26.0824959Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:26.0826807Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:32:26.3756563Z aa89d6e739080d90fa18625d57297c6734465849: Pulling from pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks 2025-03-14T05:32:26.3763165Z 86e5016c2693: Pulling fs layer 2025-03-14T05:32:26.3763612Z 49e139a3d6c2: Pulling fs layer 2025-03-14T05:32:26.3764729Z a14844c8c51f: Pulling fs layer 2025-03-14T05:32:26.3765209Z 18fb524087fb: Pulling fs layer 2025-03-14T05:32:26.3765645Z efd686a7b2c8: Pulling fs layer 2025-03-14T05:32:26.3766064Z 52c648e21334: Pulling fs layer 2025-03-14T05:32:26.3766496Z 56e384e4e5aa: Pulling fs layer 2025-03-14T05:32:26.3766934Z fb71b792ec6c: Pulling fs layer 2025-03-14T05:32:26.3767455Z 5509576f2693: Pulling fs layer 2025-03-14T05:32:26.3767962Z 1e6c6f2d2459: Pulling fs layer 2025-03-14T05:32:26.3768520Z a1fe8922734d: Pulling fs layer 2025-03-14T05:32:26.3769034Z b6dbf6a349bb: Pulling fs layer 2025-03-14T05:32:26.3769458Z 73d034163336: Pulling fs layer 2025-03-14T05:32:26.3769824Z f2a6fb332e8e: Pulling fs layer 2025-03-14T05:32:26.3770117Z 991f90b489b1: Pulling fs layer 2025-03-14T05:32:26.3770398Z 68199204396f: Pulling fs layer 2025-03-14T05:32:26.3770691Z 8d533b261ba2: Pulling fs layer 2025-03-14T05:32:26.3770979Z 9552b75545b1: Pulling fs layer 2025-03-14T05:32:26.3771469Z fb71b792ec6c: Waiting 2025-03-14T05:32:26.3771746Z 330a44046cfd: Pulling fs layer 2025-03-14T05:32:26.3772099Z fb9c93639b40: Pulling fs layer 2025-03-14T05:32:26.3772380Z a1fe8922734d: Waiting 2025-03-14T05:32:26.3772646Z a5d1ad88496e: Pulling fs layer 2025-03-14T05:32:26.3772940Z 4e2ec52a1b4f: Pulling fs layer 2025-03-14T05:32:26.3773224Z 5509576f2693: Waiting 2025-03-14T05:32:26.3773576Z b6dbf6a349bb: Waiting 2025-03-14T05:32:26.3773955Z 1e6c6f2d2459: Waiting 2025-03-14T05:32:26.3774390Z 73d034163336: Waiting 2025-03-14T05:32:26.3774840Z 1576583c5ff8: Pulling fs layer 2025-03-14T05:32:26.3775296Z ae76494b3f0f: Pulling fs layer 2025-03-14T05:32:26.3775656Z 96b47e94ca2f: Pulling fs layer 2025-03-14T05:32:26.3775963Z e26a64ee2eeb: Pulling fs layer 2025-03-14T05:32:26.3776266Z 97f84a5ed9bb: Pulling fs layer 2025-03-14T05:32:26.3776604Z f2a6fb332e8e: Waiting 2025-03-14T05:32:26.3776880Z 1cc35cfdfb65: Pulling fs layer 2025-03-14T05:32:26.3777165Z 991f90b489b1: Waiting 2025-03-14T05:32:26.3777445Z cfe527e10a79: Pulling fs layer 2025-03-14T05:32:26.3777757Z 528c12012cc2: Pulling fs layer 2025-03-14T05:32:26.3778036Z 68199204396f: Waiting 2025-03-14T05:32:26.3778302Z 3a0e7232565e: Pulling fs layer 2025-03-14T05:32:26.3778586Z 8d533b261ba2: Waiting 2025-03-14T05:32:26.3778892Z 9552b75545b1: Waiting 2025-03-14T05:32:26.3779276Z ac1e95ec684d: Pulling fs layer 2025-03-14T05:32:26.3779699Z 330a44046cfd: Waiting 2025-03-14T05:32:26.3780008Z f46926cb3922: Pulling fs layer 2025-03-14T05:32:26.3780281Z fb9c93639b40: Waiting 2025-03-14T05:32:26.3780548Z dd5d197efeb1: Pulling fs layer 2025-03-14T05:32:26.3780841Z 6ec4020a8fae: Pulling fs layer 2025-03-14T05:32:26.3781124Z a5d1ad88496e: Waiting 2025-03-14T05:32:26.3781443Z 39e1750b6399: Pulling fs layer 2025-03-14T05:32:26.3781733Z d17cf48b138e: Pulling fs layer 2025-03-14T05:32:26.3782023Z 86c6b7977b8b: Pulling fs layer 2025-03-14T05:32:26.3782304Z ae76494b3f0f: Waiting 2025-03-14T05:32:26.3782561Z 4e2ec52a1b4f: Waiting 2025-03-14T05:32:26.3782841Z b1829fd86eff: Pulling fs layer 2025-03-14T05:32:26.3783122Z 96b47e94ca2f: Waiting 2025-03-14T05:32:26.3783373Z 1576583c5ff8: Waiting 2025-03-14T05:32:26.3783634Z 4c8da014a48e: Pulling fs layer 2025-03-14T05:32:26.3783915Z e26a64ee2eeb: Waiting 2025-03-14T05:32:26.3784167Z 97f84a5ed9bb: Waiting 2025-03-14T05:32:26.3784416Z cfe527e10a79: Waiting 2025-03-14T05:32:26.3784665Z 1cc35cfdfb65: Waiting 2025-03-14T05:32:26.3784913Z 528c12012cc2: Waiting 2025-03-14T05:32:26.3785160Z ac1e95ec684d: Waiting 2025-03-14T05:32:26.3785415Z 3e933db6894b: Pulling fs layer 2025-03-14T05:32:26.3785693Z 3a0e7232565e: Waiting 2025-03-14T05:32:26.3785960Z 39e1750b6399: Waiting 2025-03-14T05:32:26.3786352Z dd5d197efeb1: Waiting 2025-03-14T05:32:26.3786883Z 1bef7a4bcd0a: Pulling fs layer 2025-03-14T05:32:26.3787318Z d17cf48b138e: Waiting 2025-03-14T05:32:26.3787684Z 86c6b7977b8b: Waiting 2025-03-14T05:32:26.3788063Z 771ad9f2789a: Pulling fs layer 2025-03-14T05:32:26.3788528Z 6ec4020a8fae: Waiting 2025-03-14T05:32:26.3788921Z 18fb524087fb: Waiting 2025-03-14T05:32:26.3789320Z b1829fd86eff: Waiting 2025-03-14T05:32:26.3789956Z 907000cb43f1: Pulling fs layer 2025-03-14T05:32:26.3790424Z 52c648e21334: Waiting 2025-03-14T05:32:26.3790816Z 4c8da014a48e: Waiting 2025-03-14T05:32:26.3791218Z efd686a7b2c8: Waiting 2025-03-14T05:32:26.3791610Z 540fab0ce537: Pulling fs layer 2025-03-14T05:32:26.3791929Z 3e933db6894b: Waiting 2025-03-14T05:32:26.3792179Z 56e384e4e5aa: Waiting 2025-03-14T05:32:26.3792436Z 1bef7a4bcd0a: Waiting 2025-03-14T05:32:26.3792706Z ca5aa8a8bb00: Pulling fs layer 2025-03-14T05:32:26.3793051Z 771ad9f2789a: Waiting 2025-03-14T05:32:26.3793297Z 907000cb43f1: Waiting 2025-03-14T05:32:26.3793559Z 19197f3f8a26: Pulling fs layer 2025-03-14T05:32:26.3793922Z ca5aa8a8bb00: Waiting 2025-03-14T05:32:26.3794245Z 50b63f41fa17: Pulling fs layer 2025-03-14T05:32:26.3794540Z 19197f3f8a26: Waiting 2025-03-14T05:32:26.3794812Z 26c4a7b9d650: Pulling fs layer 2025-03-14T05:32:26.3795111Z 50b63f41fa17: Waiting 2025-03-14T05:32:26.3795488Z 712e4e3f44e2: Pulling fs layer 2025-03-14T05:32:26.3795761Z 26c4a7b9d650: Waiting 2025-03-14T05:32:26.3796036Z a03d53d11afa: Pulling fs layer 2025-03-14T05:32:26.3796316Z 712e4e3f44e2: Waiting 2025-03-14T05:32:26.3796580Z 298cf77443f2: Pulling fs layer 2025-03-14T05:32:26.3796876Z 6debb4801aa5: Pulling fs layer 2025-03-14T05:32:26.3797166Z c127e62d2566: Pulling fs layer 2025-03-14T05:32:26.3797444Z a03d53d11afa: Waiting 2025-03-14T05:32:26.3797696Z 298cf77443f2: Waiting 2025-03-14T05:32:26.3797946Z 6debb4801aa5: Waiting 2025-03-14T05:32:26.3798209Z 7d113dac5720: Pulling fs layer 2025-03-14T05:32:26.3798501Z 4f4fb700ef54: Pulling fs layer 2025-03-14T05:32:26.3798791Z 41636a0cf6cf: Pulling fs layer 2025-03-14T05:32:26.3799084Z 6b28eaf42300: Pulling fs layer 2025-03-14T05:32:26.3799370Z 8836075b6cd4: Pulling fs layer 2025-03-14T05:32:26.3799646Z 4f4fb700ef54: Waiting 2025-03-14T05:32:26.3799898Z 794e80013542: Pulling fs layer 2025-03-14T05:32:26.3800175Z 41636a0cf6cf: Waiting 2025-03-14T05:32:26.3800434Z 6b28eaf42300: Waiting 2025-03-14T05:32:26.3800680Z 8836075b6cd4: Waiting 2025-03-14T05:32:26.3800944Z 50f65d9ffc32: Pulling fs layer 2025-03-14T05:32:26.3801243Z eebbd71eb5f7: Pulling fs layer 2025-03-14T05:32:26.3801539Z 6646f0ede462: Pulling fs layer 2025-03-14T05:32:26.3801828Z 8552cd082ff6: Pulling fs layer 2025-03-14T05:32:26.3802104Z 50f65d9ffc32: Waiting 2025-03-14T05:32:26.3802358Z eebbd71eb5f7: Waiting 2025-03-14T05:32:26.3802625Z cfc32c95496a: Pulling fs layer 2025-03-14T05:32:26.3802901Z 794e80013542: Waiting 2025-03-14T05:32:26.3803147Z 8552cd082ff6: Waiting 2025-03-14T05:32:26.3803395Z 6646f0ede462: Waiting 2025-03-14T05:32:26.3803655Z 14d81158e9c4: Pulling fs layer 2025-03-14T05:32:26.3803946Z 1f7ab46ead95: Pulling fs layer 2025-03-14T05:32:26.3804240Z 49ef4891eff9: Pulling fs layer 2025-03-14T05:32:26.3804571Z 14d81158e9c4: Waiting 2025-03-14T05:32:26.3804836Z b4c737e437c3: Pulling fs layer 2025-03-14T05:32:26.3805117Z 1f7ab46ead95: Waiting 2025-03-14T05:32:26.3805371Z 49ef4891eff9: Waiting 2025-03-14T05:32:26.3805648Z c6d1e46b2968: Pulling fs layer 2025-03-14T05:32:26.3805939Z b4c737e437c3: Waiting 2025-03-14T05:32:26.3806201Z c754fb443572: Pulling fs layer 2025-03-14T05:32:26.3806493Z cfb5071726ef: Pulling fs layer 2025-03-14T05:32:26.3806788Z 8be47e434613: Pulling fs layer 2025-03-14T05:32:26.3807080Z a68c1967b44d: Pulling fs layer 2025-03-14T05:32:26.3807366Z 985724505cd9: Pulling fs layer 2025-03-14T05:32:26.3807643Z cfb5071726ef: Waiting 2025-03-14T05:32:26.3807935Z c6d1e46b2968: Waiting 2025-03-14T05:32:26.3808177Z c754fb443572: Waiting 2025-03-14T05:32:26.3808425Z 8be47e434613: Waiting 2025-03-14T05:32:26.3808672Z 985724505cd9: Waiting 2025-03-14T05:32:26.5136090Z 49e139a3d6c2: Verifying Checksum 2025-03-14T05:32:26.5136447Z 49e139a3d6c2: Download complete 2025-03-14T05:32:26.6727649Z efd686a7b2c8: Download complete 2025-03-14T05:32:26.7278372Z 86e5016c2693: Verifying Checksum 2025-03-14T05:32:26.7278706Z 86e5016c2693: Download complete 2025-03-14T05:32:26.8094909Z 56e384e4e5aa: Verifying Checksum 2025-03-14T05:32:26.8095627Z 56e384e4e5aa: Download complete 2025-03-14T05:32:26.8824593Z fb71b792ec6c: Verifying Checksum 2025-03-14T05:32:26.8824940Z fb71b792ec6c: Download complete 2025-03-14T05:32:26.9654862Z 5509576f2693: Download complete 2025-03-14T05:32:27.0145461Z a14844c8c51f: Verifying Checksum 2025-03-14T05:32:27.0146330Z a14844c8c51f: Download complete 2025-03-14T05:32:27.1113034Z a1fe8922734d: Verifying Checksum 2025-03-14T05:32:27.1113510Z a1fe8922734d: Download complete 2025-03-14T05:32:27.1793317Z b6dbf6a349bb: Verifying Checksum 2025-03-14T05:32:27.1793828Z b6dbf6a349bb: Download complete 2025-03-14T05:32:27.8276217Z 86e5016c2693: Pull complete 2025-03-14T05:32:28.1289713Z 49e139a3d6c2: Pull complete 2025-03-14T05:32:28.8978897Z a14844c8c51f: Pull complete 2025-03-14T05:32:28.9264429Z 18fb524087fb: Pull complete 2025-03-14T05:32:28.9552447Z efd686a7b2c8: Pull complete 2025-03-14T05:32:29.6944707Z 73d034163336: Verifying Checksum 2025-03-14T05:32:29.6945070Z 73d034163336: Download complete 2025-03-14T05:32:29.7958503Z f2a6fb332e8e: Verifying Checksum 2025-03-14T05:32:29.7958956Z f2a6fb332e8e: Download complete 2025-03-14T05:32:29.8867033Z 991f90b489b1: Download complete 2025-03-14T05:32:29.9799155Z 68199204396f: Verifying Checksum 2025-03-14T05:32:29.9799493Z 68199204396f: Download complete 2025-03-14T05:32:30.9592165Z 8d533b261ba2: Verifying Checksum 2025-03-14T05:32:30.9592547Z 8d533b261ba2: Download complete 2025-03-14T05:32:31.0523445Z 9552b75545b1: Verifying Checksum 2025-03-14T05:32:31.0523769Z 9552b75545b1: Download complete 2025-03-14T05:32:31.1183085Z 330a44046cfd: Verifying Checksum 2025-03-14T05:32:31.1183488Z 330a44046cfd: Download complete 2025-03-14T05:32:31.2006204Z fb9c93639b40: Verifying Checksum 2025-03-14T05:32:31.2006547Z fb9c93639b40: Download complete 2025-03-14T05:32:31.2789628Z a5d1ad88496e: Verifying Checksum 2025-03-14T05:32:31.2790013Z a5d1ad88496e: Download complete 2025-03-14T05:32:41.2081390Z 52c648e21334: Verifying Checksum 2025-03-14T05:32:41.2081783Z 52c648e21334: Download complete 2025-03-14T05:32:41.2777214Z 1576583c5ff8: Download complete 2025-03-14T05:32:41.3507080Z ae76494b3f0f: Verifying Checksum 2025-03-14T05:32:41.3507435Z ae76494b3f0f: Download complete 2025-03-14T05:32:41.4297129Z 96b47e94ca2f: Verifying Checksum 2025-03-14T05:32:41.4297498Z 96b47e94ca2f: Download complete 2025-03-14T05:32:41.5221317Z e26a64ee2eeb: Verifying Checksum 2025-03-14T05:32:41.5221658Z e26a64ee2eeb: Download complete 2025-03-14T05:32:41.5960833Z 97f84a5ed9bb: Verifying Checksum 2025-03-14T05:32:41.5961225Z 97f84a5ed9bb: Download complete 2025-03-14T05:32:42.8723519Z 1cc35cfdfb65: Verifying Checksum 2025-03-14T05:32:42.8723903Z 1cc35cfdfb65: Download complete 2025-03-14T05:32:42.9526041Z cfe527e10a79: Download complete 2025-03-14T05:32:43.0405839Z 528c12012cc2: Verifying Checksum 2025-03-14T05:32:43.0406272Z 528c12012cc2: Download complete 2025-03-14T05:32:43.1303895Z 3a0e7232565e: Verifying Checksum 2025-03-14T05:32:43.1304260Z 3a0e7232565e: Download complete 2025-03-14T05:32:43.2239458Z ac1e95ec684d: Download complete 2025-03-14T05:32:43.2939551Z f46926cb3922: Verifying Checksum 2025-03-14T05:32:43.2950714Z f46926cb3922: Download complete 2025-03-14T05:32:47.6453617Z dd5d197efeb1: Verifying Checksum 2025-03-14T05:32:47.6454116Z dd5d197efeb1: Download complete 2025-03-14T05:32:47.7353843Z 6ec4020a8fae: Verifying Checksum 2025-03-14T05:32:47.7354186Z 6ec4020a8fae: Download complete 2025-03-14T05:32:47.8290301Z 39e1750b6399: Verifying Checksum 2025-03-14T05:32:47.8290643Z 39e1750b6399: Download complete 2025-03-14T05:32:48.2620717Z d17cf48b138e: Verifying Checksum 2025-03-14T05:32:48.2621092Z d17cf48b138e: Download complete 2025-03-14T05:32:48.3406982Z 86c6b7977b8b: Download complete 2025-03-14T05:32:48.4429882Z b1829fd86eff: Verifying Checksum 2025-03-14T05:32:48.4430370Z b1829fd86eff: Download complete 2025-03-14T05:32:48.7098585Z 4c8da014a48e: Verifying Checksum 2025-03-14T05:32:48.7099688Z 4c8da014a48e: Download complete 2025-03-14T05:32:48.8141425Z 3e933db6894b: Download complete 2025-03-14T05:32:48.8832202Z 1bef7a4bcd0a: Verifying Checksum 2025-03-14T05:32:48.8833001Z 1bef7a4bcd0a: Download complete 2025-03-14T05:32:48.9540578Z 771ad9f2789a: Download complete 2025-03-14T05:32:52.1874222Z 1e6c6f2d2459: Verifying Checksum 2025-03-14T05:32:52.1874779Z 1e6c6f2d2459: Download complete 2025-03-14T05:32:52.2761887Z 540fab0ce537: Verifying Checksum 2025-03-14T05:32:52.2762446Z 540fab0ce537: Download complete 2025-03-14T05:32:52.3523264Z ca5aa8a8bb00: Verifying Checksum 2025-03-14T05:32:52.3523749Z ca5aa8a8bb00: Download complete 2025-03-14T05:32:52.8442620Z 19197f3f8a26: Verifying Checksum 2025-03-14T05:32:52.8443116Z 19197f3f8a26: Download complete 2025-03-14T05:32:52.9370360Z 50b63f41fa17: Verifying Checksum 2025-03-14T05:32:52.9371548Z 50b63f41fa17: Download complete 2025-03-14T05:32:53.0382996Z 26c4a7b9d650: Verifying Checksum 2025-03-14T05:32:53.0383489Z 26c4a7b9d650: Download complete 2025-03-14T05:32:53.1075185Z 712e4e3f44e2: Verifying Checksum 2025-03-14T05:32:53.1076465Z 712e4e3f44e2: Download complete 2025-03-14T05:32:53.2545003Z a03d53d11afa: Download complete 2025-03-14T05:32:54.0604786Z 52c648e21334: Pull complete 2025-03-14T05:32:54.2085545Z 56e384e4e5aa: Pull complete 2025-03-14T05:32:54.3645056Z fb71b792ec6c: Pull complete 2025-03-14T05:32:54.4578949Z 5509576f2693: Pull complete 2025-03-14T05:33:00.5852064Z 4e2ec52a1b4f: Verifying Checksum 2025-03-14T05:33:00.5852507Z 4e2ec52a1b4f: Download complete 2025-03-14T05:33:00.6823725Z 6debb4801aa5: Verifying Checksum 2025-03-14T05:33:00.6824089Z 6debb4801aa5: Download complete 2025-03-14T05:33:00.7786370Z c127e62d2566: Verifying Checksum 2025-03-14T05:33:00.7786844Z c127e62d2566: Download complete 2025-03-14T05:33:00.8492589Z 7d113dac5720: Verifying Checksum 2025-03-14T05:33:00.8492987Z 7d113dac5720: Download complete 2025-03-14T05:33:00.8552130Z 4f4fb700ef54: Verifying Checksum 2025-03-14T05:33:00.8552504Z 4f4fb700ef54: Download complete 2025-03-14T05:33:00.9421131Z 41636a0cf6cf: Download complete 2025-03-14T05:33:01.0214326Z 6b28eaf42300: Download complete 2025-03-14T05:33:03.0486893Z 8836075b6cd4: Verifying Checksum 2025-03-14T05:33:03.0487415Z 8836075b6cd4: Download complete 2025-03-14T05:33:03.1454943Z 794e80013542: Download complete 2025-03-14T05:33:03.2293929Z 50f65d9ffc32: Download complete 2025-03-14T05:33:03.3270470Z eebbd71eb5f7: Verifying Checksum 2025-03-14T05:33:03.3270904Z eebbd71eb5f7: Download complete 2025-03-14T05:33:03.4120175Z 6646f0ede462: Verifying Checksum 2025-03-14T05:33:03.4120636Z 6646f0ede462: Download complete 2025-03-14T05:33:03.5710760Z 8552cd082ff6: Verifying Checksum 2025-03-14T05:33:03.5711255Z 8552cd082ff6: Download complete 2025-03-14T05:33:03.6402057Z cfc32c95496a: Verifying Checksum 2025-03-14T05:33:03.6402551Z cfc32c95496a: Download complete 2025-03-14T05:33:04.2404097Z 14d81158e9c4: Verifying Checksum 2025-03-14T05:33:04.2404462Z 14d81158e9c4: Download complete 2025-03-14T05:33:04.3131781Z 1f7ab46ead95: Verifying Checksum 2025-03-14T05:33:04.3132132Z 1f7ab46ead95: Download complete 2025-03-14T05:33:33.0459565Z 49ef4891eff9: Verifying Checksum 2025-03-14T05:33:33.0459971Z 49ef4891eff9: Download complete 2025-03-14T05:33:33.1169977Z b4c737e437c3: Verifying Checksum 2025-03-14T05:33:33.1170466Z b4c737e437c3: Download complete 2025-03-14T05:33:33.1945326Z c6d1e46b2968: Download complete 2025-03-14T05:33:41.3543246Z c754fb443572: Verifying Checksum 2025-03-14T05:33:41.3543613Z c754fb443572: Download complete 2025-03-14T05:33:41.4282899Z cfb5071726ef: Verifying Checksum 2025-03-14T05:33:41.4283353Z cfb5071726ef: Download complete 2025-03-14T05:33:41.5164416Z 8be47e434613: Download complete 2025-03-14T05:33:41.5906894Z a68c1967b44d: Verifying Checksum 2025-03-14T05:33:41.5907259Z a68c1967b44d: Download complete 2025-03-14T05:33:41.6602776Z 985724505cd9: Download complete 2025-03-14T05:34:22.5255066Z 907000cb43f1: Verifying Checksum 2025-03-14T05:34:22.5255428Z 907000cb43f1: Download complete 2025-03-14T05:34:37.7537130Z 298cf77443f2: Verifying Checksum 2025-03-14T05:34:37.7537489Z 298cf77443f2: Download complete 2025-03-14T05:34:53.7806032Z 1e6c6f2d2459: Pull complete 2025-03-14T05:34:53.9822766Z a1fe8922734d: Pull complete 2025-03-14T05:34:54.2156664Z b6dbf6a349bb: Pull complete 2025-03-14T05:35:01.8468449Z 73d034163336: Pull complete 2025-03-14T05:35:01.9264409Z f2a6fb332e8e: Pull complete 2025-03-14T05:35:02.0051006Z 991f90b489b1: Pull complete 2025-03-14T05:35:02.0298984Z 68199204396f: Pull complete 2025-03-14T05:35:04.3372400Z 8d533b261ba2: Pull complete 2025-03-14T05:35:04.5334581Z 9552b75545b1: Pull complete 2025-03-14T05:35:04.7550507Z 330a44046cfd: Pull complete 2025-03-14T05:35:04.9457073Z fb9c93639b40: Pull complete 2025-03-14T05:35:05.1336031Z a5d1ad88496e: Pull complete 2025-03-14T05:35:57.7140884Z 4e2ec52a1b4f: Pull complete 2025-03-14T05:35:57.8641654Z 1576583c5ff8: Pull complete 2025-03-14T05:35:58.0866658Z ae76494b3f0f: Pull complete 2025-03-14T05:35:58.2775544Z 96b47e94ca2f: Pull complete 2025-03-14T05:35:58.5073708Z e26a64ee2eeb: Pull complete 2025-03-14T05:35:58.7160499Z 97f84a5ed9bb: Pull complete 2025-03-14T05:36:01.2327477Z 1cc35cfdfb65: Pull complete 2025-03-14T05:36:01.4117413Z cfe527e10a79: Pull complete 2025-03-14T05:36:01.5957505Z 528c12012cc2: Pull complete 2025-03-14T05:36:01.8391381Z 3a0e7232565e: Pull complete 2025-03-14T05:36:02.0284086Z ac1e95ec684d: Pull complete 2025-03-14T05:36:02.2708175Z f46926cb3922: Pull complete 2025-03-14T05:36:10.7542989Z dd5d197efeb1: Pull complete 2025-03-14T05:36:10.9263467Z 6ec4020a8fae: Pull complete 2025-03-14T05:36:11.1663408Z 39e1750b6399: Pull complete 2025-03-14T05:36:11.9810033Z d17cf48b138e: Pull complete 2025-03-14T05:36:12.1668981Z 86c6b7977b8b: Pull complete 2025-03-14T05:36:12.4046644Z b1829fd86eff: Pull complete 2025-03-14T05:36:12.8468922Z 4c8da014a48e: Pull complete 2025-03-14T05:36:13.0445579Z 3e933db6894b: Pull complete 2025-03-14T05:36:13.4517966Z 1bef7a4bcd0a: Pull complete 2025-03-14T05:36:13.6637962Z 771ad9f2789a: Pull complete 2025-03-14T05:36:51.9285920Z 907000cb43f1: Pull complete 2025-03-14T05:36:52.1219875Z 540fab0ce537: Pull complete 2025-03-14T05:36:52.3137782Z ca5aa8a8bb00: Pull complete 2025-03-14T05:36:54.9421694Z 19197f3f8a26: Pull complete 2025-03-14T05:36:55.1454740Z 50b63f41fa17: Pull complete 2025-03-14T05:36:55.3651563Z 26c4a7b9d650: Pull complete 2025-03-14T05:36:55.7072445Z 712e4e3f44e2: Pull complete 2025-03-14T05:36:55.8268362Z a03d53d11afa: Pull complete 2025-03-14T05:39:20.4462279Z 298cf77443f2: Pull complete 2025-03-14T05:39:20.6620578Z 6debb4801aa5: Pull complete 2025-03-14T05:39:20.8581081Z c127e62d2566: Pull complete 2025-03-14T05:39:21.2081044Z 7d113dac5720: Pull complete 2025-03-14T05:39:21.3067951Z 4f4fb700ef54: Pull complete 2025-03-14T05:39:21.4473407Z 41636a0cf6cf: Pull complete 2025-03-14T05:39:21.5588893Z 6b28eaf42300: Pull complete 2025-03-14T05:39:32.6739099Z 8836075b6cd4: Pull complete 2025-03-14T05:39:32.8536025Z 794e80013542: Pull complete 2025-03-14T05:39:33.0595596Z 50f65d9ffc32: Pull complete 2025-03-14T05:39:33.2650778Z eebbd71eb5f7: Pull complete 2025-03-14T05:39:33.4663186Z 6646f0ede462: Pull complete 2025-03-14T05:39:34.8095358Z 8552cd082ff6: Pull complete 2025-03-14T05:39:34.8634986Z cfc32c95496a: Pull complete 2025-03-14T05:39:37.2863497Z 14d81158e9c4: Pull complete 2025-03-14T05:39:37.3697942Z 1f7ab46ead95: Pull complete 2025-03-14T05:40:18.4075098Z 49ef4891eff9: Pull complete 2025-03-14T05:40:18.6093063Z b4c737e437c3: Pull complete 2025-03-14T05:40:18.7914311Z c6d1e46b2968: Pull complete 2025-03-14T05:40:30.0928555Z c754fb443572: Pull complete 2025-03-14T05:40:30.3001249Z cfb5071726ef: Pull complete 2025-03-14T05:40:30.5030081Z 8be47e434613: Pull complete 2025-03-14T05:40:30.6966468Z a68c1967b44d: Pull complete 2025-03-14T05:40:30.8894627Z 985724505cd9: Pull complete 2025-03-14T05:40:31.7477892Z Digest: sha256:20eb41577713f879cca4c6a57bd64d737c482cad39fe2b18409f513444e2b522 2025-03-14T05:40:31.7937543Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:40:31.8223546Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:40:31.8295584Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:40:31.8296482Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-03-14T05:40:31.8307139Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:40:31.8307515Z env: 2025-03-14T05:40:31.8307745Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:40:31.8308013Z ##[endgroup] 2025-03-14T05:40:31.8633055Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-03-14T05:40:31.8633707Z with: 2025-03-14T05:40:31.8634053Z driver-version: 550.54.15 2025-03-14T05:40:31.8634459Z env: 2025-03-14T05:40:31.8634797Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:40:31.8635556Z ##[endgroup] 2025-03-14T05:40:31.8836550Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-03-14T05:40:31.8837181Z with: 2025-03-14T05:40:31.8837515Z timeout_minutes: 10 2025-03-14T05:40:31.8837907Z max_attempts: 3 2025-03-14T05:40:31.8875963Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2025-03-14T05:40:31.8914282Z retry_wait_seconds: 10 2025-03-14T05:40:31.8914722Z polling_interval_seconds: 1 2025-03-14T05:40:31.8915190Z warning_on_retry: true 2025-03-14T05:40:31.8915614Z continue_on_error: false 2025-03-14T05:40:31.8916031Z env: 2025-03-14T05:40:31.8916372Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:40:31.8916804Z DRIVER_VERSION: 550.54.15 2025-03-14T05:40:31.8917225Z ##[endgroup] 2025-03-14T05:40:32.7453310Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2025-03-14T05:40:32.7454051Z + pre_install_nvidia_driver_amzn2 2025-03-14T05:40:32.7454758Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-03-14T05:40:35.4842772Z No match for argument: nvidia-driver-latest-dkms 2025-03-14T05:40:35.4843195Z No packages marked for removal. 2025-03-14T05:40:35.6784260Z Dependencies resolved. 2025-03-14T05:40:35.6793621Z Nothing to do. 2025-03-14T05:40:35.6793983Z Complete! 2025-03-14T05:40:35.7802222Z + install_nvidia_driver_common 2025-03-14T05:40:35.7809509Z + echo 'Before installing NVIDIA driver' 2025-03-14T05:40:35.7810028Z + lspci 2025-03-14T05:40:35.7821398Z Before installing NVIDIA driver 2025-03-14T05:40:35.7994829Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-03-14T05:40:35.7995585Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-03-14T05:40:35.7996309Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-03-14T05:40:35.7997011Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-03-14T05:40:35.7997714Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-03-14T05:40:35.7998436Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-03-14T05:40:35.7998956Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2025-03-14T05:40:35.7999469Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-03-14T05:40:35.8000218Z + lsmod 2025-03-14T05:40:35.8042159Z Module Size Used by 2025-03-14T05:40:35.8042559Z xt_nat 16384 0 2025-03-14T05:40:35.8042961Z ib_core 454656 0 2025-03-14T05:40:35.8043342Z veth 36864 0 2025-03-14T05:40:35.8043868Z nvidia_modeset 1351680 0 2025-03-14T05:40:35.8044173Z video 65536 1 nvidia_modeset 2025-03-14T05:40:35.8044505Z wmi 36864 1 video 2025-03-14T05:40:35.8044821Z nvidia_uvm 4706304 0 2025-03-14T05:40:35.8045228Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2025-03-14T05:40:35.8045717Z drm 602112 1 nvidia 2025-03-14T05:40:35.8046168Z drm_panel_orientation_quirks 32768 1 drm 2025-03-14T05:40:35.8046665Z backlight 24576 3 video,drm,nvidia_modeset 2025-03-14T05:40:35.8047040Z i2c_core 110592 2 nvidia,drm 2025-03-14T05:40:35.8047361Z xt_conntrack 16384 1 2025-03-14T05:40:35.8047663Z nft_chain_nat 16384 3 2025-03-14T05:40:35.8047953Z xt_MASQUERADE 20480 1 2025-03-14T05:40:35.8048299Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2025-03-14T05:40:35.8048675Z nf_conntrack_netlink 57344 0 2025-03-14T05:40:35.8049146Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-03-14T05:40:35.8049634Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-03-14T05:40:35.8049970Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-03-14T05:40:35.8050290Z xfrm_user 57344 1 2025-03-14T05:40:35.8050583Z xfrm_algo 16384 1 xfrm_user 2025-03-14T05:40:35.8050902Z xt_addrtype 16384 2 2025-03-14T05:40:35.8051189Z nft_compat 20480 4 2025-03-14T05:40:35.8051516Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-03-14T05:40:35.8051965Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-03-14T05:40:35.8052370Z br_netfilter 36864 0 2025-03-14T05:40:35.8052680Z bridge 323584 1 br_netfilter 2025-03-14T05:40:35.8053003Z stp 16384 1 bridge 2025-03-14T05:40:35.8053316Z llc 16384 2 bridge,stp 2025-03-14T05:40:35.8053629Z overlay 167936 0 2025-03-14T05:40:35.8053919Z tls 135168 0 2025-03-14T05:40:35.8054198Z nls_ascii 16384 1 2025-03-14T05:40:35.8054495Z nls_cp437 20480 1 2025-03-14T05:40:35.8054775Z vfat 24576 1 2025-03-14T05:40:35.8055057Z fat 86016 1 vfat 2025-03-14T05:40:35.8055354Z sunrpc 696320 1 2025-03-14T05:40:35.8055634Z ena 180224 0 2025-03-14T05:40:35.8055909Z i8042 45056 0 2025-03-14T05:40:35.8056191Z serio 28672 3 i8042 2025-03-14T05:40:35.8056498Z ghash_clmulni_intel 16384 0 2025-03-14T05:40:35.8056791Z button 24576 0 2025-03-14T05:40:35.8057069Z sch_fq_codel 20480 17 2025-03-14T05:40:35.8057366Z dm_mod 188416 0 2025-03-14T05:40:35.8057645Z fuse 163840 1 2025-03-14T05:40:35.8057925Z configfs 57344 1 2025-03-14T05:40:35.8058205Z loop 36864 0 2025-03-14T05:40:35.8058495Z dax 45056 1 dm_mod 2025-03-14T05:40:35.8058802Z dmi_sysfs 20480 0 2025-03-14T05:40:35.8059088Z crc32_pclmul 16384 0 2025-03-14T05:40:35.8059370Z crc32c_intel 24576 0 2025-03-14T05:40:35.8059645Z efivarfs 24576 1 2025-03-14T05:40:35.8059915Z + modinfo nvidia 2025-03-14T05:40:35.8061163Z filename: /lib/modules/6.1.129-138.220.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-03-14T05:40:35.8061833Z alias: char-major-195-* 2025-03-14T05:40:35.8062269Z version: 550.54.15 2025-03-14T05:40:35.8062636Z supported: external 2025-03-14T05:40:35.8062985Z license: NVIDIA 2025-03-14T05:40:35.8063284Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2025-03-14T05:40:35.8063883Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2025-03-14T05:40:35.8064277Z srcversion: 833721318DA517F0C2FEC97 2025-03-14T05:40:35.8064656Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-03-14T05:40:35.8065048Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-03-14T05:40:35.8065507Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-03-14T05:40:35.8065850Z depends: i2c-core,drm 2025-03-14T05:40:35.8066136Z retpoline: Y 2025-03-14T05:40:35.8066387Z name: nvidia 2025-03-14T05:40:35.8066870Z vermagic: 6.1.129-138.220.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-03-14T05:40:35.8067488Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-03-14T05:40:35.8067968Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-03-14T05:40:35.8068416Z parm: NVreg_ResmanDebugLevel:int 2025-03-14T05:40:35.8068757Z parm: NVreg_RmLogonRC:int 2025-03-14T05:40:35.8069094Z parm: NVreg_ModifyDeviceFiles:int 2025-03-14T05:40:35.8069441Z parm: NVreg_DeviceFileUID:int 2025-03-14T05:40:35.8069777Z parm: NVreg_DeviceFileGID:int 2025-03-14T05:40:35.8070116Z parm: NVreg_DeviceFileMode:int 2025-03-14T05:40:35.8070614Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-03-14T05:40:35.8071039Z parm: NVreg_UsePageAttributeTable:int 2025-03-14T05:40:35.8071408Z parm: NVreg_EnablePCIeGen3:int 2025-03-14T05:40:35.8071745Z parm: NVreg_EnableMSI:int 2025-03-14T05:40:35.8072071Z parm: NVreg_TCEBypassMode:int 2025-03-14T05:40:35.8072415Z parm: NVreg_EnableStreamMemOPs:int 2025-03-14T05:40:35.8072812Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-03-14T05:40:35.8073245Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-03-14T05:40:35.8073760Z parm: NVreg_EnableS0ixPowerManagement:int 2025-03-14T05:40:35.8074210Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-03-14T05:40:35.8074654Z parm: NVreg_DynamicPowerManagement:int 2025-03-14T05:40:35.8075104Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-03-14T05:40:35.8075547Z parm: NVreg_EnableGpuFirmware:int 2025-03-14T05:40:35.8075917Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-03-14T05:40:35.8076330Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-03-14T05:40:35.8076728Z parm: NVreg_EnableUserNUMAManagement:int 2025-03-14T05:40:35.8077095Z parm: NVreg_MemoryPoolSize:int 2025-03-14T05:40:35.8077447Z parm: NVreg_KMallocHeapMaxSize:int 2025-03-14T05:40:35.8077809Z parm: NVreg_VMallocHeapMaxSize:int 2025-03-14T05:40:35.8078162Z parm: NVreg_IgnoreMMIOCheck:int 2025-03-14T05:40:35.8078503Z parm: NVreg_NvLinkDisable:int 2025-03-14T05:40:35.8078874Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-03-14T05:40:35.8079272Z parm: NVreg_RegisterPCIDriver:int 2025-03-14T05:40:35.8079635Z parm: NVreg_EnableResizableBar:int 2025-03-14T05:40:35.8080003Z parm: NVreg_EnableDbgBreakpoint:int 2025-03-14T05:40:35.8080372Z parm: NVreg_EnableNonblockingOpen:int 2025-03-14T05:40:35.8080733Z parm: NVreg_RegistryDwords:charp 2025-03-14T05:40:35.8081114Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-03-14T05:40:35.8081477Z parm: NVreg_RmMsg:charp 2025-03-14T05:40:35.8081795Z parm: NVreg_GpuBlacklist:charp 2025-03-14T05:40:35.8082147Z parm: NVreg_TemporaryFilePath:charp 2025-03-14T05:40:35.8082501Z parm: NVreg_ExcludedGpus:charp 2025-03-14T05:40:35.8082845Z parm: NVreg_DmaRemapPeerMmio:int 2025-03-14T05:40:35.8083207Z parm: NVreg_RmNvlinkBandwidth:charp 2025-03-14T05:40:35.8083566Z parm: NVreg_ImexChannelCount:int 2025-03-14T05:40:35.8083910Z parm: rm_firmware_active:charp 2025-03-14T05:40:35.8084235Z + HAS_NVIDIA_DRIVER=0 2025-03-14T05:40:35.8084511Z ++ command -v nvidia-smi 2025-03-14T05:40:35.8084913Z + '[' -x /usr/bin/nvidia-smi ']' 2025-03-14T05:40:35.8085196Z + set +e 2025-03-14T05:40:35.8085543Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-03-14T05:40:35.8818624Z + INSTALLED_DRIVER_VERSION=550.54.15 2025-03-14T05:40:35.8819185Z + NVIDIA_SMI_STATUS=0 2025-03-14T05:40:35.8819446Z + '[' 0 -ne 0 ']' 2025-03-14T05:40:35.8819694Z + '[' 550.54.15 '!=' 550.54.15 ']' 2025-03-14T05:40:35.8819984Z + HAS_NVIDIA_DRIVER=1 2025-03-14T05:40:35.8820434Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2025-03-14T05:40:35.8820926Z + set -e 2025-03-14T05:40:35.8821152Z + '[' 1 -eq 0 ']' 2025-03-14T05:40:35.8821567Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2025-03-14T05:40:35.8822057Z + post_install_nvidia_driver_common 2025-03-14T05:40:35.8825818Z + sudo modprobe nvidia 2025-03-14T05:40:35.9824880Z + echo 'After installing NVIDIA driver' 2025-03-14T05:40:35.9825248Z + lspci 2025-03-14T05:40:35.9825530Z After installing NVIDIA driver 2025-03-14T05:40:35.9941179Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-03-14T05:40:35.9941821Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-03-14T05:40:35.9942424Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-03-14T05:40:35.9942978Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-03-14T05:40:35.9943492Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-03-14T05:40:35.9944055Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-03-14T05:40:35.9944571Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2025-03-14T05:40:35.9945083Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-03-14T05:40:35.9945518Z + lsmod 2025-03-14T05:40:35.9974170Z Module Size Used by 2025-03-14T05:40:35.9974490Z xt_nat 16384 0 2025-03-14T05:40:35.9974790Z ib_core 454656 0 2025-03-14T05:40:35.9975159Z veth 36864 0 2025-03-14T05:40:35.9975469Z nvidia_modeset 1351680 0 2025-03-14T05:40:35.9975782Z video 65536 1 nvidia_modeset 2025-03-14T05:40:35.9976120Z wmi 36864 1 video 2025-03-14T05:40:35.9976420Z nvidia_uvm 4706304 0 2025-03-14T05:40:35.9976749Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2025-03-14T05:40:35.9977104Z drm 602112 1 nvidia 2025-03-14T05:40:35.9977440Z drm_panel_orientation_quirks 32768 1 drm 2025-03-14T05:40:35.9977827Z backlight 24576 3 video,drm,nvidia_modeset 2025-03-14T05:40:35.9978205Z i2c_core 110592 2 nvidia,drm 2025-03-14T05:40:35.9978524Z xt_conntrack 16384 1 2025-03-14T05:40:35.9978813Z nft_chain_nat 16384 3 2025-03-14T05:40:35.9979104Z xt_MASQUERADE 20480 1 2025-03-14T05:40:35.9979471Z nf_nat 57344 3 xt_nat,nft_chain_nat,xt_MASQUERADE 2025-03-14T05:40:35.9979848Z nf_conntrack_netlink 57344 0 2025-03-14T05:40:35.9980304Z nf_conntrack 184320 5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-03-14T05:40:35.9980796Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-03-14T05:40:35.9981142Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-03-14T05:40:35.9981469Z xfrm_user 57344 1 2025-03-14T05:40:35.9981766Z xfrm_algo 16384 1 xfrm_user 2025-03-14T05:40:35.9982086Z xt_addrtype 16384 2 2025-03-14T05:40:35.9982378Z nft_compat 20480 4 2025-03-14T05:40:35.9982716Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-03-14T05:40:35.9983165Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-03-14T05:40:35.9983571Z br_netfilter 36864 0 2025-03-14T05:40:35.9983879Z bridge 323584 1 br_netfilter 2025-03-14T05:40:35.9984483Z stp 16384 1 bridge 2025-03-14T05:40:35.9984807Z llc 16384 2 bridge,stp 2025-03-14T05:40:35.9985123Z overlay 167936 0 2025-03-14T05:40:35.9985404Z tls 135168 0 2025-03-14T05:40:35.9985681Z nls_ascii 16384 1 2025-03-14T05:40:35.9986107Z nls_cp437 20480 1 2025-03-14T05:40:35.9986384Z vfat 24576 1 2025-03-14T05:40:35.9986782Z fat 86016 1 vfat 2025-03-14T05:40:35.9987076Z sunrpc 696320 1 2025-03-14T05:40:35.9987353Z ena 180224 0 2025-03-14T05:40:35.9987631Z i8042 45056 0 2025-03-14T05:40:35.9987914Z serio 28672 3 i8042 2025-03-14T05:40:35.9988217Z ghash_clmulni_intel 16384 0 2025-03-14T05:40:35.9988511Z button 24576 0 2025-03-14T05:40:35.9988786Z sch_fq_codel 20480 17 2025-03-14T05:40:35.9989073Z dm_mod 188416 0 2025-03-14T05:40:35.9989350Z fuse 163840 1 2025-03-14T05:40:35.9989640Z configfs 57344 1 2025-03-14T05:40:35.9989919Z loop 36864 0 2025-03-14T05:40:35.9990203Z dax 45056 1 dm_mod 2025-03-14T05:40:35.9990504Z dmi_sysfs 20480 0 2025-03-14T05:40:35.9990792Z crc32_pclmul 16384 0 2025-03-14T05:40:35.9991074Z crc32c_intel 24576 0 2025-03-14T05:40:35.9991353Z efivarfs 24576 1 2025-03-14T05:40:35.9991635Z + modinfo nvidia 2025-03-14T05:40:35.9996218Z filename: /lib/modules/6.1.129-138.220.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-03-14T05:40:35.9996734Z alias: char-major-195-* 2025-03-14T05:40:35.9997033Z version: 550.54.15 2025-03-14T05:40:35.9997312Z supported: external 2025-03-14T05:40:35.9997589Z license: NVIDIA 2025-03-14T05:40:35.9997885Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2025-03-14T05:40:35.9998253Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2025-03-14T05:40:35.9998609Z srcversion: 833721318DA517F0C2FEC97 2025-03-14T05:40:35.9998963Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-03-14T05:40:35.9999336Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-03-14T05:40:35.9999695Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-03-14T05:40:36.0000046Z depends: i2c-core,drm 2025-03-14T05:40:36.0000334Z retpoline: Y 2025-03-14T05:40:36.0000584Z name: nvidia 2025-03-14T05:40:36.0000973Z vermagic: 6.1.129-138.220.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-03-14T05:40:36.0001476Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-03-14T05:40:36.0001953Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-03-14T05:40:36.0002404Z parm: NVreg_ResmanDebugLevel:int 2025-03-14T05:40:36.0002749Z parm: NVreg_RmLogonRC:int 2025-03-14T05:40:36.0003089Z parm: NVreg_ModifyDeviceFiles:int 2025-03-14T05:40:36.0003437Z parm: NVreg_DeviceFileUID:int 2025-03-14T05:40:36.0003778Z parm: NVreg_DeviceFileGID:int 2025-03-14T05:40:36.0004112Z parm: NVreg_DeviceFileMode:int 2025-03-14T05:40:36.0004505Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-03-14T05:40:36.0005023Z parm: NVreg_UsePageAttributeTable:int 2025-03-14T05:40:36.0005439Z parm: NVreg_EnablePCIeGen3:int 2025-03-14T05:40:36.0005772Z parm: NVreg_EnableMSI:int 2025-03-14T05:40:36.0006096Z parm: NVreg_TCEBypassMode:int 2025-03-14T05:40:36.0006438Z parm: NVreg_EnableStreamMemOPs:int 2025-03-14T05:40:36.0006834Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-03-14T05:40:36.0007264Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-03-14T05:40:36.0007675Z parm: NVreg_EnableS0ixPowerManagement:int 2025-03-14T05:40:36.0008120Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-03-14T05:40:36.0008561Z parm: NVreg_DynamicPowerManagement:int 2025-03-14T05:40:36.0009137Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-03-14T05:40:36.0009582Z parm: NVreg_EnableGpuFirmware:int 2025-03-14T05:40:36.0009959Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-03-14T05:40:36.0010360Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-03-14T05:40:36.0010889Z parm: NVreg_EnableUserNUMAManagement:int 2025-03-14T05:40:36.0011258Z parm: NVreg_MemoryPoolSize:int 2025-03-14T05:40:36.0011614Z parm: NVreg_KMallocHeapMaxSize:int 2025-03-14T05:40:36.0011978Z parm: NVreg_VMallocHeapMaxSize:int 2025-03-14T05:40:36.0012332Z parm: NVreg_IgnoreMMIOCheck:int 2025-03-14T05:40:36.0012676Z parm: NVreg_NvLinkDisable:int 2025-03-14T05:40:36.0013055Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-03-14T05:40:36.0013445Z parm: NVreg_RegisterPCIDriver:int 2025-03-14T05:40:36.0013804Z parm: NVreg_EnableResizableBar:int 2025-03-14T05:40:36.0014166Z parm: NVreg_EnableDbgBreakpoint:int 2025-03-14T05:40:36.0014586Z parm: NVreg_EnableNonblockingOpen:int 2025-03-14T05:40:36.0014969Z parm: NVreg_RegistryDwords:charp 2025-03-14T05:40:36.0015348Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-03-14T05:40:36.0015785Z parm: NVreg_RmMsg:charp 2025-03-14T05:40:36.0016228Z parm: NVreg_GpuBlacklist:charp 2025-03-14T05:40:36.0016615Z parm: NVreg_TemporaryFilePath:charp 2025-03-14T05:40:36.0016974Z parm: NVreg_ExcludedGpus:charp 2025-03-14T05:40:36.0017321Z parm: NVreg_DmaRemapPeerMmio:int 2025-03-14T05:40:36.0017684Z parm: NVreg_RmNvlinkBandwidth:charp 2025-03-14T05:40:36.0018045Z parm: NVreg_ImexChannelCount:int 2025-03-14T05:40:36.0018390Z parm: rm_firmware_active:charp 2025-03-14T05:40:36.0018706Z + set +e 2025-03-14T05:40:36.0018927Z + nvidia-smi 2025-03-14T05:40:36.0203496Z Fri Mar 14 05:40:36 2025 2025-03-14T05:40:36.0203911Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:36.0204441Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2025-03-14T05:40:36.0204958Z |-----------------------------------------+------------------------+----------------------+ 2025-03-14T05:40:36.0205492Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-03-14T05:40:36.0206055Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-03-14T05:40:36.0206520Z | | | MIG M. | 2025-03-14T05:40:36.0206882Z |=========================================+========================+======================| 2025-03-14T05:40:36.0706362Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2025-03-14T05:40:36.0707041Z | 0% 19C P8 22W / 300W | 0MiB / 23028MiB | 0% Default | 2025-03-14T05:40:36.0707469Z | | | N/A | 2025-03-14T05:40:36.0707896Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T05:40:36.0711552Z 2025-03-14T05:40:36.0712079Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:36.0712542Z | Processes: | 2025-03-14T05:40:36.0713027Z | GPU GI CI PID Type Process name GPU Memory | 2025-03-14T05:40:36.0713483Z | ID ID Usage | 2025-03-14T05:40:36.0713855Z |=========================================================================================| 2025-03-14T05:40:36.0718585Z | No running processes found | 2025-03-14T05:40:36.0719099Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:36.3361274Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-03-14T05:40:36.3548751Z NVIDIA A10G 2025-03-14T05:40:36.3611518Z + NVIDIA_SMI_STATUS=0 2025-03-14T05:40:36.3612367Z + '[' 0 -eq 0 ']' 2025-03-14T05:40:36.3613126Z + echo 'INFO: Ignoring allowed status 0' 2025-03-14T05:40:36.3613809Z + set -e 2025-03-14T05:40:36.3614303Z INFO: Ignoring allowed status 0 2025-03-14T05:40:36.3621462Z == Installing nvidia container toolkit for amzn2023 == 2025-03-14T05:40:36.3625206Z + sudo yum install -y yum-utils 2025-03-14T05:40:37.0660976Z Last metadata expiration check: 3:26:39 ago on Fri Mar 14 02:13:58 2025. 2025-03-14T05:40:37.0902153Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-03-14T05:40:37.1296213Z Dependencies resolved. 2025-03-14T05:40:37.1477367Z Nothing to do. 2025-03-14T05:40:37.1477732Z Complete! 2025-03-14T05:40:37.2148339Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-03-14T05:40:37.2148975Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T05:40:37.2149865Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T05:40:37.4734418Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-03-14T05:40:37.5350918Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2025-03-14T05:40:38.0933689Z nvidia-container-toolkit 15 kB/s | 833 B 00:00 2025-03-14T05:40:38.1179624Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2025-03-14T05:40:38.1184886Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed. 2025-03-14T05:40:38.1574128Z Dependencies resolved. 2025-03-14T05:40:38.1752160Z Nothing to do. 2025-03-14T05:40:38.1752417Z Complete! 2025-03-14T05:40:38.2519139Z + sudo systemctl restart docker 2025-03-14T05:40:58.8197193Z nvidia-persistenced failed to initialize. Check syslog for more details. 2025-03-14T05:40:58.8437754Z Fri Mar 14 05:40:58 2025 2025-03-14T05:40:58.8439696Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:58.8440288Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2025-03-14T05:40:58.8440815Z |-----------------------------------------+------------------------+----------------------+ 2025-03-14T05:40:58.8441338Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-03-14T05:40:58.8441899Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-03-14T05:40:58.8442368Z | | | MIG M. | 2025-03-14T05:40:58.8442742Z |=========================================+========================+======================| 2025-03-14T05:40:58.8644274Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2025-03-14T05:40:58.8644753Z | 0% 19C P8 22W / 300W | 0MiB / 23028MiB | 0% Default | 2025-03-14T05:40:58.8655169Z | | | N/A | 2025-03-14T05:40:58.8655651Z +-----------------------------------------+------------------------+----------------------+ 2025-03-14T05:40:58.8656084Z 2025-03-14T05:40:58.8656504Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:58.8656962Z | Processes: | 2025-03-14T05:40:58.8657445Z | GPU GI CI PID Type Process name GPU Memory | 2025-03-14T05:40:58.8658169Z | ID ID Usage | 2025-03-14T05:40:58.8658551Z |=========================================================================================| 2025-03-14T05:40:58.8659139Z | No running processes found | 2025-03-14T05:40:58.8659633Z +-----------------------------------------------------------------------------------------+ 2025-03-14T05:40:59.9957299Z Command completed after 1 attempt(s). 2025-03-14T05:41:00.0054949Z Prepare all required actions 2025-03-14T05:41:00.0084394Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-03-14T05:41:00.0084742Z with: 2025-03-14T05:41:00.0085136Z github-token: *** 2025-03-14T05:41:00.0085381Z env: 2025-03-14T05:41:00.0085614Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:00.0085958Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:00.0086320Z ##[endgroup] 2025-03-14T05:41:00.0108224Z ##[group]Run set -eux 2025-03-14T05:41:00.0108515Z set -eux 2025-03-14T05:41:00.0108963Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-03-14T05:41:00.0121140Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:00.0121530Z env: 2025-03-14T05:41:00.0121760Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:00.0122104Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:00.0122626Z GITHUB_TOKEN: *** 2025-03-14T05:41:00.0122874Z ##[endgroup] 2025-03-14T05:41:00.0157414Z + python3 .github/scripts/get_workflow_job_id.py 13849515380 i-0166a710cfefd3e7e 2025-03-14T05:41:01.1203168Z setting job-id=38756916747 2025-03-14T05:41:01.1203745Z setting job-name=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:01.1317531Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-03-14T05:41:01.1318254Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-03-14T05:41:01.1318828Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2025-03-14T05:41:01.1319307Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-03-14T05:41:01.1328679Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:01.1329066Z env: 2025-03-14T05:41:01.1329304Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:01.1329654Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:01.1330019Z JOB_ID: 38756916747 2025-03-14T05:41:01.1330497Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:01.1331025Z WORKFLOW_NAME: inductor 2025-03-14T05:41:01.1331316Z WORKFLOW_RUN_ID: 13849515380 2025-03-14T05:41:01.1331631Z ##[endgroup] 2025-03-14T05:41:01.3696163Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T05:41:01.3885883Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2025-03-14T05:41:01.3889957Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2025-03-14T05:41:01.3893990Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-03-14T05:41:01.3997544Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-03-14T05:41:01.4001229Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-03-14T05:41:01.4087158Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (24.2) 2025-03-14T05:41:01.4114849Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.12.2) 2025-03-14T05:41:01.4118603Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.0.0) 2025-03-14T05:41:01.5348945Z Prepare all required actions 2025-03-14T05:41:01.5349578Z Getting action download info 2025-03-14T05:41:01.6590819Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-03-14T05:41:01.8759983Z Download action repository 'actions/download-artifact@v4' (SHA:cc203385981b70ca67e1cc392babf9cc229d5806) 2025-03-14T05:41:02.1387475Z ##[group]Run ./.github/actions/download-build-artifacts 2025-03-14T05:41:02.1388029Z with: 2025-03-14T05:41:02.1388432Z name: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:41:02.1388966Z s3-bucket: gha-artifacts 2025-03-14T05:41:02.1389391Z env: 2025-03-14T05:41:02.1389734Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:02.1390267Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:02.1390873Z ##[endgroup] 2025-03-14T05:41:02.1431255Z ##[group]Run seemethere/download-artifact-s3@v4 2025-03-14T05:41:02.1431602Z with: 2025-03-14T05:41:02.1431923Z name: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:41:02.1432266Z s3-bucket: gha-artifacts 2025-03-14T05:41:02.1432556Z region: us-east-1 2025-03-14T05:41:02.1432798Z env: 2025-03-14T05:41:02.1433040Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:02.1433389Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:02.1433761Z ##[endgroup] 2025-03-14T05:41:02.6037176Z (node:825457) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-03-14T05:41:02.6037656Z 2025-03-14T05:41:02.6037852Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-03-14T05:41:02.6038383Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-03-14T05:41:02.6038964Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-03-14T05:41:02.7147018Z Found 1 objects with prefix pytorch/pytorch/13849515380/linux-focal-cuda12.6-py3.10-gcc9-sm86/ 2025-03-14T05:41:02.7147781Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-03-14T05:41:10.9865385Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-03-14T05:41:10.9871876Z Artifact download has finished successfully 2025-03-14T05:41:11.0224811Z ##[group]Run unzip -o artifacts.zip 2025-03-14T05:41:11.0225162Z unzip -o artifacts.zip 2025-03-14T05:41:11.0235145Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:11.0235522Z env: 2025-03-14T05:41:11.0235756Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:11.0236100Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:11.0236465Z ##[endgroup] 2025-03-14T05:41:11.0285487Z Archive: artifacts.zip 2025-03-14T05:41:11.0286516Z creating: dist/ 2025-03-14T05:41:13.3543633Z inflating: dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl 2025-03-14T05:41:13.3678319Z inflating: dist/.ninja_log 2025-03-14T05:41:13.3678686Z creating: build/custom_test_artifacts/ 2025-03-14T05:41:13.3679140Z creating: build/custom_test_artifacts/custom-op-build/ 2025-03-14T05:41:13.3679641Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-03-14T05:41:13.3681367Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-03-14T05:41:13.3689211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T05:41:13.3689872Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/ 2025-03-14T05:41:13.3690781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T05:41:13.3691526Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T05:41:13.3692214Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T05:41:13.3695078Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T05:41:13.3696888Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T05:41:13.3697651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T05:41:13.3698372Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T05:41:13.3699067Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T05:41:13.3701714Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T05:41:13.3703256Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T05:41:13.3704538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T05:41:13.3706868Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T05:41:13.3708765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T05:41:13.3709547Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T05:41:13.3710253Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T05:41:13.3752245Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T05:41:13.3793709Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T05:41:13.3794721Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T05:41:13.3842203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T05:41:13.3843197Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T05:41:13.3845195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T05:41:13.3847284Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T05:41:13.3849250Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T05:41:13.3851161Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T05:41:13.3853124Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T05:41:13.3854243Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T05:41:13.3855201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T05:41:13.3856106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T05:41:13.3856987Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T05:41:13.3857845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T05:41:13.3858716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T05:41:13.3859561Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T05:41:13.3860420Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T05:41:13.3933142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T05:41:13.3934230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T05:41:13.4009465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T05:41:13.4010225Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-03-14T05:41:13.4010837Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-03-14T05:41:13.4011480Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-03-14T05:41:13.4012139Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-03-14T05:41:13.4012862Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-03-14T05:41:13.4013693Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-03-14T05:41:13.4014479Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-03-14T05:41:13.4015205Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-03-14T05:41:13.4015954Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-03-14T05:41:13.4016712Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-03-14T05:41:13.4017472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-03-14T05:41:13.4018230Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-03-14T05:41:13.4018996Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-03-14T05:41:13.4041024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-03-14T05:41:13.4240077Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-03-14T05:41:13.4240781Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-03-14T05:41:13.4241545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-03-14T05:41:13.4242403Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-03-14T05:41:13.4243221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-03-14T05:41:13.4244030Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-03-14T05:41:13.4244816Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-03-14T05:41:13.4245622Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-03-14T05:41:13.4247074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-03-14T05:41:13.4247867Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-03-14T05:41:13.4249004Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-03-14T05:41:13.4270894Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-03-14T05:41:13.4354818Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-03-14T05:41:13.4356481Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T05:41:13.4357980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-03-14T05:41:13.4359916Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-03-14T05:41:13.4361192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-03-14T05:41:13.4362434Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-03-14T05:41:13.4363414Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-03-14T05:41:13.4364007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-03-14T05:41:13.4364556Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-03-14T05:41:13.4365117Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-03-14T05:41:13.4529187Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-03-14T05:41:13.4592374Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-03-14T05:41:13.4592907Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-03-14T05:41:13.4593405Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-03-14T05:41:13.4593995Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-03-14T05:41:13.4601664Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T05:41:13.4602404Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/ 2025-03-14T05:41:13.4603132Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T05:41:13.4603915Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T05:41:13.4604674Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T05:41:13.4605958Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T05:41:13.4608107Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T05:41:13.4608955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T05:41:13.4609763Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T05:41:13.4610538Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T05:41:13.4612769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T05:41:13.4614153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T05:41:13.4615425Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T05:41:13.4617586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T05:41:13.4619872Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T05:41:13.4620655Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T05:41:13.4621343Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T05:41:13.4663987Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T05:41:13.4704490Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T05:41:13.4705484Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T05:41:13.4752958Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T05:41:13.4754094Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T05:41:13.4755196Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T05:41:13.4756205Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T05:41:13.4757187Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T05:41:13.4758140Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T05:41:13.4759107Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T05:41:13.4760074Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T05:41:13.4761279Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T05:41:13.4762184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T05:41:13.4763390Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T05:41:13.4764268Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T05:41:13.4765127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T05:41:13.4766097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T05:41:13.4769067Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T05:41:13.4843627Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T05:41:13.4844567Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T05:41:13.4919197Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T05:41:13.4919974Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-03-14T05:41:13.4920568Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-03-14T05:41:13.4921192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-03-14T05:41:13.4921847Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-03-14T05:41:13.4922592Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-03-14T05:41:13.4923496Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-03-14T05:41:13.4924306Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-03-14T05:41:13.4925049Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-03-14T05:41:13.4925822Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-03-14T05:41:13.4927127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-03-14T05:41:13.4928060Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-03-14T05:41:13.4928858Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-03-14T05:41:13.4929879Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-03-14T05:41:13.4951601Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-03-14T05:41:13.5016271Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-03-14T05:41:13.5017893Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T05:41:13.5019375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-03-14T05:41:13.5020715Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-03-14T05:41:13.5021945Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-03-14T05:41:13.5023174Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-03-14T05:41:13.5023846Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-03-14T05:41:13.5024428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-03-14T05:41:13.5024984Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-03-14T05:41:13.5025881Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-03-14T05:41:13.5075981Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-03-14T05:41:13.5076508Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-03-14T05:41:13.5077023Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-03-14T05:41:13.5077630Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-03-14T05:41:13.5085179Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-03-14T05:41:13.5085870Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/ 2025-03-14T05:41:13.5086548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-03-14T05:41:13.5087279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-03-14T05:41:13.5087993Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-03-14T05:41:13.5089989Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-03-14T05:41:13.5091809Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-03-14T05:41:13.5092593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-03-14T05:41:13.5093348Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-03-14T05:41:13.5094079Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-03-14T05:41:13.5096731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-03-14T05:41:13.5098312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-03-14T05:41:13.5099550Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-03-14T05:41:13.5101572Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-03-14T05:41:13.5103737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-03-14T05:41:13.5104555Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-03-14T05:41:13.5105290Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-03-14T05:41:13.5146681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-03-14T05:41:13.5188083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-03-14T05:41:13.5189256Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-03-14T05:41:13.5236565Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-03-14T05:41:13.5237594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-03-14T05:41:13.5238653Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-03-14T05:41:13.5239938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-03-14T05:41:13.5241003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-03-14T05:41:13.5242058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-03-14T05:41:13.5243094Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-03-14T05:41:13.5244120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-03-14T05:41:13.5245216Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-03-14T05:41:13.5246168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-03-14T05:41:13.5247078Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-03-14T05:41:13.5247998Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-03-14T05:41:13.5249091Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-03-14T05:41:13.5250017Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-03-14T05:41:13.5252817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-03-14T05:41:13.5327028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-03-14T05:41:13.5328105Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-03-14T05:41:13.5402463Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-03-14T05:41:13.5403261Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-03-14T05:41:13.5403934Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-03-14T05:41:13.5404609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-03-14T05:41:13.5405319Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-03-14T05:41:13.5406113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-03-14T05:41:13.5407013Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-03-14T05:41:13.5407887Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-03-14T05:41:13.5408706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-03-14T05:41:13.5409547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-03-14T05:41:13.5410708Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-03-14T05:41:13.5411559Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-03-14T05:41:13.5412410Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-03-14T05:41:13.5413249Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-03-14T05:41:13.5417843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-03-14T05:41:13.5540588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-03-14T05:41:13.5541416Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-03-14T05:41:13.5542258Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-03-14T05:41:13.5543197Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-03-14T05:41:13.5544093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-03-14T05:41:13.5544944Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-03-14T05:41:13.5545811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-03-14T05:41:13.5546698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-03-14T05:41:13.5547643Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-03-14T05:41:13.5548530Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-03-14T05:41:13.5549407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-03-14T05:41:13.5570819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-03-14T05:41:13.5626665Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-03-14T05:41:13.5628101Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-03-14T05:41:13.5628914Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-03-14T05:41:13.5629659Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-03-14T05:41:13.5630856Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-03-14T05:41:13.5632733Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-03-14T05:41:13.5633412Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-03-14T05:41:13.5636093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-03-14T05:41:13.5636882Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-03-14T05:41:13.5637889Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-03-14T05:41:13.5738526Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-03-14T05:41:13.5781778Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-03-14T05:41:13.5782272Z creating: build/lib/ 2025-03-14T05:41:13.5871103Z inflating: build/lib/libprotobuf-lite.a 2025-03-14T05:41:13.6325751Z inflating: build/lib/libprotobuf.a 2025-03-14T05:41:13.6828562Z inflating: build/lib/libprotoc.a 2025-03-14T05:41:13.6838573Z inflating: build/lib/libpthreadpool.a 2025-03-14T05:41:13.6846661Z inflating: build/lib/libcpuinfo.a 2025-03-14T05:41:13.6854596Z inflating: build/lib/libcpuinfo_internals.a 2025-03-14T05:41:13.6855379Z inflating: build/lib/libclog.a 2025-03-14T05:41:13.6875323Z inflating: build/lib/libpytorch_qnnpack.a 2025-03-14T05:41:13.6878152Z inflating: build/lib/libnnpack_reference_layers.a 2025-03-14T05:41:13.6896266Z inflating: build/lib/libnnpack.a 2025-03-14T05:41:13.7082349Z inflating: build/lib/libmicrokernels-prod.a 2025-03-14T05:41:13.7894406Z inflating: build/lib/libmicrokernels-all.a 2025-03-14T05:41:13.7975900Z inflating: build/lib/libXNNPACK.a 2025-03-14T05:41:13.8044859Z inflating: build/lib/libgtest.a 2025-03-14T05:41:13.8061846Z inflating: build/lib/libgmock.a 2025-03-14T05:41:13.8062419Z inflating: build/lib/libgmock_main.a 2025-03-14T05:41:13.8063688Z inflating: build/lib/libgtest_main.a 2025-03-14T05:41:13.8139130Z inflating: build/lib/libbenchmark.a 2025-03-14T05:41:13.8139582Z inflating: build/lib/libbenchmark_main.a 2025-03-14T05:41:13.8203728Z inflating: build/lib/libasmjit.a 2025-03-14T05:41:13.9463563Z inflating: build/lib/libfbgemm.a 2025-03-14T05:41:13.9471175Z inflating: build/lib/libittnotify.a 2025-03-14T05:41:13.9498999Z inflating: build/lib/libtensorpipe_uv.a 2025-03-14T05:41:14.0074118Z inflating: build/lib/libtensorpipe.a 2025-03-14T05:41:14.0329759Z inflating: build/lib/libtensorpipe_cuda.a 2025-03-14T05:41:14.0456959Z inflating: build/lib/libgloo.a 2025-03-14T05:41:14.0499574Z inflating: build/lib/libonnx_proto.a 2025-03-14T05:41:14.1229403Z inflating: build/lib/libonnx.a 2025-03-14T05:41:15.1790609Z inflating: build/lib/libdnnl.a 2025-03-14T05:41:15.1809983Z inflating: build/lib/libfmt.a 2025-03-14T05:41:15.2276111Z inflating: build/lib/libkineto.a 2025-03-14T05:41:15.2388856Z inflating: build/lib/libc10.so 2025-03-14T05:41:15.2448477Z inflating: build/lib/libc10_cuda.so 2025-03-14T05:41:15.2450482Z inflating: build/lib/libcaffe2_nvrtc.so 2025-03-14T05:41:15.2452033Z inflating: build/lib/libtorch_global_deps.so 2025-03-14T05:41:15.2840528Z inflating: build/lib/libgloo_cuda.a 2025-03-14T05:41:17.8369474Z inflating: build/lib/libtorch_cpu.so 2025-03-14T05:41:20.2858915Z inflating: build/lib/libtorch_cuda.so 2025-03-14T05:41:20.2863652Z inflating: build/lib/libunbox_lib.a 2025-03-14T05:41:20.2864619Z inflating: build/lib/libtorch.so 2025-03-14T05:41:21.1649297Z inflating: build/lib/libtorch_cuda_linalg.so 2025-03-14T05:41:21.1720165Z inflating: build/lib/libtorchbind_test.so 2025-03-14T05:41:21.1740328Z inflating: build/lib/libjitbackend_test.so 2025-03-14T05:41:21.1765319Z inflating: build/lib/libbackend_with_compiler.so 2025-03-14T05:41:21.1790271Z inflating: build/lib/libaoti_custom_ops.so 2025-03-14T05:41:21.1793033Z inflating: build/lib/libc10d_cuda_test.so 2025-03-14T05:41:21.1797443Z inflating: build/lib/libshm.so 2025-03-14T05:41:21.3858281Z inflating: build/lib/libtorch_python.so 2025-03-14T05:41:21.3893254Z inflating: build/lib/libnnapi_backend.so 2025-03-14T05:41:21.3893615Z creating: build/bin/ 2025-03-14T05:41:21.4344341Z inflating: build/bin/protoc-3.13.0.0 2025-03-14T05:41:21.4794296Z inflating: build/bin/protoc 2025-03-14T05:41:21.4848150Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-03-14T05:41:21.4903569Z inflating: build/bin/c10_DeviceGuard_test 2025-03-14T05:41:21.4958682Z inflating: build/bin/c10_Device_test 2025-03-14T05:41:21.5021143Z inflating: build/bin/c10_DispatchKeySet_test 2025-03-14T05:41:21.5077890Z inflating: build/bin/c10_Scalar_test 2025-03-14T05:41:21.5131057Z inflating: build/bin/c10_StreamGuard_test 2025-03-14T05:41:21.5185303Z inflating: build/bin/c10_SymInt_test 2025-03-14T05:41:21.5242689Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-03-14T05:41:21.5301931Z inflating: build/bin/c10_InlineStreamGuard_test 2025-03-14T05:41:21.5361607Z inflating: build/bin/c10_SizesAndStrides_test 2025-03-14T05:41:21.5435914Z inflating: build/bin/c10_cow_test 2025-03-14T05:41:21.5488951Z inflating: build/bin/c10_ArrayRef_test 2025-03-14T05:41:21.5545514Z inflating: build/bin/c10_Bitset_test 2025-03-14T05:41:21.5597789Z inflating: build/bin/c10_ConstexprCrc_test 2025-03-14T05:41:21.5651492Z inflating: build/bin/c10_DeadlockDetection_test 2025-03-14T05:41:21.5705240Z inflating: build/bin/c10_Half_test 2025-03-14T05:41:21.5764867Z inflating: build/bin/c10_LeftRight_test 2025-03-14T05:41:21.5823599Z inflating: build/bin/c10_Metaprogramming_test 2025-03-14T05:41:21.5880170Z inflating: build/bin/c10_NetworkFlow_test 2025-03-14T05:41:21.5933787Z inflating: build/bin/c10_Synchronized_test 2025-03-14T05:41:21.5992654Z inflating: build/bin/c10_ThreadLocal_test 2025-03-14T05:41:21.6048619Z inflating: build/bin/c10_TypeIndex_test 2025-03-14T05:41:21.6103773Z inflating: build/bin/c10_TypeList_test 2025-03-14T05:41:21.6156578Z inflating: build/bin/c10_TypeTraits_test 2025-03-14T05:41:21.6212118Z inflating: build/bin/c10_accumulate_test 2025-03-14T05:41:21.6271096Z inflating: build/bin/c10_bfloat16_test 2025-03-14T05:41:21.6325161Z inflating: build/bin/c10_bit_cast_test 2025-03-14T05:41:21.6385893Z inflating: build/bin/c10_complex_math_test 2025-03-14T05:41:21.6444078Z inflating: build/bin/c10_complex_test 2025-03-14T05:41:21.6497437Z inflating: build/bin/c10_error_test 2025-03-14T05:41:21.6553915Z inflating: build/bin/c10_exception_test 2025-03-14T05:41:21.6608373Z inflating: build/bin/c10_flags_test 2025-03-14T05:41:21.6662515Z inflating: build/bin/c10_generic_math_test 2025-03-14T05:41:21.6834011Z inflating: build/bin/c10_intrusive_ptr_test 2025-03-14T05:41:21.6888710Z inflating: build/bin/c10_irange_test 2025-03-14T05:41:21.6946031Z inflating: build/bin/c10_lazy_test 2025-03-14T05:41:21.7006604Z inflating: build/bin/c10_logging_test 2025-03-14T05:41:21.7086155Z inflating: build/bin/c10_optional_test 2025-03-14T05:41:21.7152062Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-03-14T05:41:21.7209045Z inflating: build/bin/c10_registry_test 2025-03-14T05:41:21.7368978Z inflating: build/bin/c10_small_vector_test 2025-03-14T05:41:21.7423956Z inflating: build/bin/c10_ssize_test 2025-03-14T05:41:21.7479373Z inflating: build/bin/c10_string_util_test 2025-03-14T05:41:21.7531619Z inflating: build/bin/c10_string_view_test 2025-03-14T05:41:21.7585619Z inflating: build/bin/c10_tempfile_test 2025-03-14T05:41:21.7645059Z inflating: build/bin/c10_typeid_test 2025-03-14T05:41:21.7694096Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-03-14T05:41:21.7749921Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-03-14T05:41:21.7806532Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-03-14T05:41:21.7862383Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-03-14T05:41:21.7917947Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-03-14T05:41:21.7974249Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-03-14T05:41:21.8032097Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-03-14T05:41:21.8087851Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-03-14T05:41:21.8140490Z inflating: build/bin/c10_cuda_CUDATest 2025-03-14T05:41:21.8551433Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-03-14T05:41:21.8971147Z inflating: build/bin/vec_test_all_types_AVX512 2025-03-14T05:41:21.9404575Z inflating: build/bin/vec_test_all_types_AVX2 2025-03-14T05:41:21.9462246Z inflating: build/bin/test_edge_op_registration 2025-03-14T05:41:21.9539438Z inflating: build/bin/Dict_test 2025-03-14T05:41:21.9595304Z inflating: build/bin/Dimname_test 2025-03-14T05:41:21.9663936Z inflating: build/bin/MaybeOwned_test 2025-03-14T05:41:21.9724958Z inflating: build/bin/NamedTensor_test 2025-03-14T05:41:21.9787613Z inflating: build/bin/apply_utils_test 2025-03-14T05:41:21.9850515Z inflating: build/bin/atest 2025-03-14T05:41:21.9917339Z inflating: build/bin/basic 2025-03-14T05:41:21.9975878Z inflating: build/bin/broadcast_test 2025-03-14T05:41:22.0031592Z inflating: build/bin/cpu_allocator_test 2025-03-14T05:41:22.0094136Z inflating: build/bin/cpu_generator_test 2025-03-14T05:41:22.0151235Z inflating: build/bin/cpu_profiling_allocator_test 2025-03-14T05:41:22.0247361Z inflating: build/bin/cpu_rng_test 2025-03-14T05:41:22.0300938Z inflating: build/bin/dispatch_key_set_test 2025-03-14T05:41:22.0355202Z inflating: build/bin/dlconvertor_test 2025-03-14T05:41:22.0417368Z inflating: build/bin/extension_backend_test 2025-03-14T05:41:22.0475930Z inflating: build/bin/half_test 2025-03-14T05:41:22.0578725Z inflating: build/bin/ivalue_test 2025-03-14T05:41:22.0632413Z inflating: build/bin/lazy_tensor_test 2025-03-14T05:41:22.0690010Z inflating: build/bin/math_kernel_test 2025-03-14T05:41:22.0747558Z inflating: build/bin/memory_format_test 2025-03-14T05:41:22.0804220Z inflating: build/bin/memory_overlapping_test 2025-03-14T05:41:22.0860687Z inflating: build/bin/mobile_memory_cleanup 2025-03-14T05:41:22.0920687Z inflating: build/bin/native_test 2025-03-14T05:41:22.0975161Z inflating: build/bin/operator_name_test 2025-03-14T05:41:22.1030433Z inflating: build/bin/operators_test 2025-03-14T05:41:22.1085209Z inflating: build/bin/packedtensoraccessor_test 2025-03-14T05:41:22.1156211Z inflating: build/bin/pow_test 2025-03-14T05:41:22.1217265Z inflating: build/bin/quantized_test 2025-03-14T05:41:22.1272184Z inflating: build/bin/reduce_ops_test 2025-03-14T05:41:22.1327393Z inflating: build/bin/reportMemoryUsage_test 2025-03-14T05:41:22.1388420Z inflating: build/bin/scalar_tensor_test 2025-03-14T05:41:22.1451254Z inflating: build/bin/scalar_test 2025-03-14T05:41:22.1506431Z inflating: build/bin/StorageUtils_test 2025-03-14T05:41:22.1562966Z inflating: build/bin/stride_properties_test 2025-03-14T05:41:22.1647277Z inflating: build/bin/tensor_iterator_test 2025-03-14T05:41:22.1705922Z inflating: build/bin/test_parallel 2025-03-14T05:41:22.1708739Z inflating: build/bin/thread_init_test 2025-03-14T05:41:22.1768264Z inflating: build/bin/type_ptr_test 2025-03-14T05:41:22.1831970Z inflating: build/bin/type_test 2025-03-14T05:41:22.1888343Z inflating: build/bin/undefined_tensor_test 2025-03-14T05:41:22.1889926Z inflating: build/bin/verify_api_visibility 2025-03-14T05:41:22.1964520Z inflating: build/bin/legacy_vmap_test 2025-03-14T05:41:22.2019865Z inflating: build/bin/weakref_test 2025-03-14T05:41:22.2075142Z inflating: build/bin/wrapdim_test 2025-03-14T05:41:22.2130506Z inflating: build/bin/xla_tensor_test 2025-03-14T05:41:22.2194410Z inflating: build/bin/IListRef_test 2025-03-14T05:41:22.2305461Z inflating: build/bin/List_test 2025-03-14T05:41:22.2376067Z inflating: build/bin/KernelFunction_test 2025-03-14T05:41:22.2502486Z inflating: build/bin/kernel_function_legacy_test 2025-03-14T05:41:22.2603122Z inflating: build/bin/kernel_function_test 2025-03-14T05:41:22.2736392Z inflating: build/bin/kernel_lambda_legacy_test 2025-03-14T05:41:22.2844100Z inflating: build/bin/kernel_lambda_test 2025-03-14T05:41:22.2909002Z inflating: build/bin/kernel_stackbased_test 2025-03-14T05:41:22.3008969Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-03-14T05:41:22.3064235Z inflating: build/bin/CppSignature_test 2025-03-14T05:41:22.3123339Z inflating: build/bin/backend_fallback_test 2025-03-14T05:41:22.3176188Z inflating: build/bin/op_allowlist_test 2025-03-14T05:41:22.3484659Z inflating: build/bin/op_registration_test 2025-03-14T05:41:22.3552401Z inflating: build/bin/inline_container_test 2025-03-14T05:41:22.3608549Z inflating: build/bin/cuda_allocator_test 2025-03-14T05:41:22.3665718Z inflating: build/bin/cuda_apply_test 2025-03-14T05:41:22.3728888Z inflating: build/bin/cuda_atomic_ops_test 2025-03-14T05:41:22.3789874Z inflating: build/bin/cuda_caching_host_allocator_test 2025-03-14T05:41:22.3864947Z inflating: build/bin/cuda_complex_math_test 2025-03-14T05:41:22.3928145Z inflating: build/bin/cuda_complex_test 2025-03-14T05:41:22.3993837Z inflating: build/bin/cuda_cub_test 2025-03-14T05:41:22.4047727Z inflating: build/bin/cuda_device_test 2025-03-14T05:41:22.4116404Z inflating: build/bin/cuda_distributions_test 2025-03-14T05:41:22.4172220Z inflating: build/bin/cuda_dlconvertor_test 2025-03-14T05:41:22.4232496Z inflating: build/bin/cuda_generator_test 2025-03-14T05:41:22.4286844Z inflating: build/bin/cuda_half_test 2025-03-14T05:41:22.4341717Z inflating: build/bin/cuda_integer_divider_test 2025-03-14T05:41:22.4395118Z inflating: build/bin/cuda_optional_test 2025-03-14T05:41:22.4451257Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-03-14T05:41:22.4507594Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-03-14T05:41:22.4561103Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-03-14T05:41:22.4626641Z inflating: build/bin/cuda_stream_test 2025-03-14T05:41:22.4682581Z inflating: build/bin/cuda_vectorized_test 2025-03-14T05:41:22.4736289Z inflating: build/bin/cuda_cudnn_test 2025-03-14T05:41:22.5320241Z inflating: build/bin/test_jit 2025-03-14T05:41:22.5335406Z inflating: build/bin/tutorial_tensorexpr 2025-03-14T05:41:22.5392093Z inflating: build/bin/BackoffTest 2025-03-14T05:41:22.5448743Z inflating: build/bin/FileStoreTest 2025-03-14T05:41:22.5507727Z inflating: build/bin/TCPStoreTest 2025-03-14T05:41:22.5564131Z inflating: build/bin/HashStoreTest 2025-03-14T05:41:22.5634526Z inflating: build/bin/ProcessGroupGlooTest 2025-03-14T05:41:22.6476607Z inflating: build/bin/test_tensorexpr 2025-03-14T05:41:22.6537765Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-03-14T05:41:22.6605126Z inflating: build/bin/ProcessGroupNCCLTest 2025-03-14T05:41:22.6619017Z inflating: build/bin/ProcessGroupMPITest 2025-03-14T05:41:22.6622170Z inflating: build/bin/example_allreduce 2025-03-14T05:41:22.6687922Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-03-14T05:41:22.6746544Z inflating: build/bin/test_dist_autograd 2025-03-14T05:41:22.6819052Z inflating: build/bin/test_cpp_rpc 2025-03-14T05:41:22.6821830Z inflating: build/bin/parallel_benchmark 2025-03-14T05:41:22.6892889Z inflating: build/bin/test_mobile_nnc 2025-03-14T05:41:22.6902116Z inflating: build/bin/aot_model_compiler_test 2025-03-14T05:41:22.8109744Z inflating: build/bin/test_api 2025-03-14T05:41:22.8468530Z inflating: build/bin/test_lazy 2025-03-14T05:41:22.8473023Z inflating: build/bin/torch_shm_manager 2025-03-14T05:41:22.8473391Z creating: .additional_ci_files/ 2025-03-14T05:41:22.8581664Z inflating: .additional_ci_files/test-times.json 2025-03-14T05:41:22.9001412Z inflating: .additional_ci_files/test-class-times.json 2025-03-14T05:41:22.9035575Z ##[group]Run rm artifacts.zip 2025-03-14T05:41:22.9035927Z rm artifacts.zip 2025-03-14T05:41:22.9044749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:22.9045119Z env: 2025-03-14T05:41:22.9045349Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:22.9045694Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:22.9046055Z ##[endgroup] 2025-03-14T05:41:23.0432565Z ##[group]Run df -H 2025-03-14T05:41:23.0432839Z df -H 2025-03-14T05:41:23.0442017Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:23.0442395Z env: 2025-03-14T05:41:23.0442619Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:23.0443162Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:23.0443687Z ##[endgroup] 2025-03-14T05:41:23.0532930Z Filesystem Size Used Avail Use% Mounted on 2025-03-14T05:41:23.0533325Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-03-14T05:41:23.0533675Z tmpfs 34G 82k 34G 1% /dev/shm 2025-03-14T05:41:23.0534031Z tmpfs 14G 553k 14G 1% /run 2025-03-14T05:41:23.0534372Z /dev/nvme0n1p1 161G 109G 53G 68% / 2025-03-14T05:41:23.0534708Z tmpfs 34G 1.6M 34G 1% /tmp 2025-03-14T05:41:23.0535062Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-03-14T05:41:23.0535428Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2025-03-14T05:41:23.0576545Z Prepare all required actions 2025-03-14T05:41:23.0577105Z Getting action download info 2025-03-14T05:41:23.1869787Z ##[group]Run ./.github/actions/download-td-artifacts 2025-03-14T05:41:23.1870156Z with: 2025-03-14T05:41:23.1870376Z env: 2025-03-14T05:41:23.1870612Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:23.1870982Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:23.1871352Z ##[endgroup] 2025-03-14T05:41:23.2009606Z ##[group]Run seemethere/download-artifact-s3@v4 2025-03-14T05:41:23.2009953Z with: 2025-03-14T05:41:23.2010179Z name: td_results 2025-03-14T05:41:23.2010442Z s3-bucket: gha-artifacts 2025-03-14T05:41:23.2010719Z region: us-east-1 2025-03-14T05:41:23.2010956Z env: 2025-03-14T05:41:23.2011181Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:23.2011519Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:23.2011887Z ##[endgroup] 2025-03-14T05:41:23.6511780Z (node:825477) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-03-14T05:41:23.6512335Z 2025-03-14T05:41:23.6512592Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-03-14T05:41:23.6513125Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-03-14T05:41:23.6513672Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-03-14T05:41:23.7527660Z Found 0 objects with prefix pytorch/pytorch/13849515380/td_results/ 2025-03-14T05:41:23.7533989Z Artifact download has finished successfully 2025-03-14T05:41:23.8058776Z ##[group]Run mkdir -p .additional_ci_files 2025-03-14T05:41:23.8059164Z mkdir -p .additional_ci_files 2025-03-14T05:41:23.8059610Z mv td_results.json .additional_ci_files/td_results.json || true 2025-03-14T05:41:23.8069185Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:23.8069564Z env: 2025-03-14T05:41:23.8069794Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:23.8070154Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:23.8070514Z ##[endgroup] 2025-03-14T05:41:23.8332178Z mv: cannot stat 'td_results.json': No such file or directory 2025-03-14T05:41:23.8520360Z ##[group]Run .github/scripts/parse_ref.py 2025-03-14T05:41:23.8520918Z .github/scripts/parse_ref.py 2025-03-14T05:41:23.8530906Z shell: /usr/bin/bash -e {0} 2025-03-14T05:41:23.8531190Z env: 2025-03-14T05:41:23.8531429Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:23.8531780Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:23.8532145Z ##[endgroup] 2025-03-14T05:41:23.8870936Z Prepare all required actions 2025-03-14T05:41:23.8871317Z Getting action download info 2025-03-14T05:41:24.0302141Z ##[group]Run ./.github/actions/filter-test-configs 2025-03-14T05:41:24.0302499Z with: 2025-03-14T05:41:24.0302911Z github-token: *** 2025-03-14T05:41:24.0304581Z test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2025-03-14T05:41:24.0306805Z job-name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:24.0307319Z env: 2025-03-14T05:41:24.0307725Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:24.0308077Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:24.0308442Z ##[endgroup] 2025-03-14T05:41:24.0496481Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T05:41:24.0496792Z with: 2025-03-14T05:41:24.0497011Z shell: bash 2025-03-14T05:41:24.0497246Z timeout_minutes: 10 2025-03-14T05:41:24.0497504Z max_attempts: 5 2025-03-14T05:41:24.0497763Z retry_wait_seconds: 30 2025-03-14T05:41:24.0498518Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-03-14T05:41:24.0499307Z polling_interval_seconds: 1 2025-03-14T05:41:24.0499613Z warning_on_retry: true 2025-03-14T05:41:24.0499895Z continue_on_error: false 2025-03-14T05:41:24.0500163Z env: 2025-03-14T05:41:24.0500392Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:24.0500735Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:24.0501418Z GITHUB_TOKEN: *** 2025-03-14T05:41:24.0501668Z ##[endgroup] 2025-03-14T05:41:24.9111917Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-03-14T05:41:25.2456080Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (2.27.1) 2025-03-14T05:41:25.2458430Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (6.0.1) 2025-03-14T05:41:25.2483027Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (1.26.20) 2025-03-14T05:41:25.2486181Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2025.1.31) 2025-03-14T05:41:25.2491920Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (2.0.12) 2025-03-14T05:41:25.2496823Z Requirement already satisfied: idna<4,>=2.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from requests==2.27.1) (3.7) 2025-03-14T05:41:26.1292626Z Command completed after 1 attempt(s). 2025-03-14T05:41:26.1486138Z ##[group]Run set -x 2025-03-14T05:41:26.1486542Z set -x 2025-03-14T05:41:26.1486786Z  2025-03-14T05:41:26.1487181Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-03-14T05:41:26.1487668Z # in runner workspace 2025-03-14T05:41:26.1488075Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-03-14T05:41:26.1498786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:26.1499170Z env: 2025-03-14T05:41:26.1499405Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:26.1499780Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:26.1500151Z ##[endgroup] 2025-03-14T05:41:26.1529992Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-03-14T05:41:26.1886261Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-03-14T05:41:26.1886683Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-03-14T05:41:26.1887036Z echo "Job name: ${JOB_NAME}" 2025-03-14T05:41:26.1887347Z  2025-03-14T05:41:26.1887740Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-03-14T05:41:26.1888223Z # in runner workspace 2025-03-14T05:41:26.1888656Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-03-14T05:41:26.1889135Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-03-14T05:41:26.1889485Z  --job-name "${JOB_NAME}" \ 2025-03-14T05:41:26.1891426Z  --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2025-03-14T05:41:26.1893187Z  --selected-test-configs "" \ 2025-03-14T05:41:26.1893543Z  --pr-number "${PR_NUMBER}" \ 2025-03-14T05:41:26.1893875Z  --tag "${TAG}" \ 2025-03-14T05:41:26.1894190Z  --event-name "${EVENT_NAME}" \ 2025-03-14T05:41:26.1894537Z  --schedule "${SCHEDULE}" \ 2025-03-14T05:41:26.1894869Z  --branch "${HEAD_BRANCH}" 2025-03-14T05:41:26.1903328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:26.1903719Z env: 2025-03-14T05:41:26.1903961Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:26.1904301Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:26.1904860Z GITHUB_TOKEN: *** 2025-03-14T05:41:26.1905339Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:26.1905853Z PR_NUMBER: 2025-03-14T05:41:26.1906089Z TAG: 2025-03-14T05:41:26.1906319Z EVENT_NAME: push 2025-03-14T05:41:26.1906675Z SCHEDULE: 2025-03-14T05:41:26.1906907Z HEAD_BRANCH: 2025-03-14T05:41:26.1907157Z ##[endgroup] 2025-03-14T05:41:26.1934923Z Workflow: inductor 2025-03-14T05:41:26.1935592Z Job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:26.4810153Z ##[group]Run echo "Filtered matrix:" 2025-03-14T05:41:26.4810503Z echo "Filtered matrix:" 2025-03-14T05:41:26.4812211Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2025-03-14T05:41:26.4813908Z  2025-03-14T05:41:26.4814130Z echo 2025-03-14T05:41:26.4814422Z echo "Is the current job unstable? False" 2025-03-14T05:41:26.4814767Z  2025-03-14T05:41:26.4814989Z echo 2025-03-14T05:41:26.4815262Z echo "Is keep-going label set? False" 2025-03-14T05:41:26.4815595Z  2025-03-14T05:41:26.4815817Z echo 2025-03-14T05:41:26.4816069Z echo "Renabled issues? " 2025-03-14T05:41:26.4825620Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:26.4826001Z env: 2025-03-14T05:41:26.4826666Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:26.4827117Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:26.4827820Z ##[endgroup] 2025-03-14T05:41:26.4855052Z Filtered matrix: 2025-03-14T05:41:26.4857064Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2025-03-14T05:41:26.4859220Z 2025-03-14T05:41:26.4859393Z Is the current job unstable? False 2025-03-14T05:41:26.4859685Z 2025-03-14T05:41:26.4859845Z Is keep-going label set? False 2025-03-14T05:41:26.4860297Z 2025-03-14T05:41:26.4860400Z Renabled issues? 2025-03-14T05:41:26.4970436Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-03-14T05:41:26.4970963Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-03-14T05:41:26.4979154Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T05:41:26.4979539Z env: 2025-03-14T05:41:26.4979773Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:26.4980126Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:26.4980492Z JOB_TIMEOUT: 240 2025-03-14T05:41:26.4980737Z ##[endgroup] 2025-03-14T05:41:26.5169866Z ##[group]Run set -x 2025-03-14T05:41:26.5170203Z set -x 2025-03-14T05:41:26.5170439Z  2025-03-14T05:41:26.5170702Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-03-14T05:41:26.5171095Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-03-14T05:41:26.5171490Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-03-14T05:41:26.5171868Z  TEST_COMMAND=.ci/onnx/test.sh 2025-03-14T05:41:26.5172170Z else 2025-03-14T05:41:26.5172430Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-03-14T05:41:26.5172737Z fi 2025-03-14T05:41:26.5172964Z  2025-03-14T05:41:26.5173237Z # Leaving 1GB for the runner and other things 2025-03-14T05:41:26.5173790Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-03-14T05:41:26.5174627Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-03-14T05:41:26.5175308Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-03-14T05:41:26.5175844Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-03-14T05:41:26.5176256Z  2025-03-14T05:41:26.5176590Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-03-14T05:41:26.5176950Z  SHM_OPTS= 2025-03-14T05:41:26.5177224Z  JENKINS_USER= 2025-03-14T05:41:26.5177588Z  # ensure that docker container cleanly exits in 12 hours 2025-03-14T05:41:26.5178063Z  # if for some reason cleanup action doesn't stop container 2025-03-14T05:41:26.5178466Z  # when job is cancelled 2025-03-14T05:41:26.5178784Z  DOCKER_SHELL_CMD="sleep 12h" 2025-03-14T05:41:26.5179090Z  2025-03-14T05:41:26.5179470Z  # since some steps are skipped on s390x, if they are necessary, run them here 2025-03-14T05:41:26.5180012Z  env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T05:41:26.5180464Z  env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-03-14T05:41:26.5180824Z else 2025-03-14T05:41:26.5181094Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-03-14T05:41:26.5181447Z  JENKINS_USER="--user jenkins" 2025-03-14T05:41:26.5181767Z  DOCKER_SHELL_CMD= 2025-03-14T05:41:26.5182049Z fi 2025-03-14T05:41:26.5182272Z  2025-03-14T05:41:26.5182619Z # detached container should get cleaned up by teardown_ec2_linux 2025-03-14T05:41:26.5183146Z # TODO: Stop building test binaries as part of the build phase 2025-03-14T05:41:26.5183748Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-03-14T05:41:26.5184274Z # shellcheck disable=SC2086,SC2090 2025-03-14T05:41:26.5184623Z container_name=$(docker run \ 2025-03-14T05:41:26.5184942Z  ${GPU_FLAG:-} \ 2025-03-14T05:41:26.5185263Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-03-14T05:41:26.5185619Z  -e BUILD_ENVIRONMENT \ 2025-03-14T05:41:26.5185930Z  -e PR_NUMBER \ 2025-03-14T05:41:26.5186217Z  -e GITHUB_ACTIONS \ 2025-03-14T05:41:26.5186662Z  -e GITHUB_REPOSITORY \ 2025-03-14T05:41:26.5186970Z  -e GITHUB_WORKFLOW \ 2025-03-14T05:41:26.5187443Z  -e GITHUB_JOB \ 2025-03-14T05:41:26.5187724Z  -e GITHUB_RUN_ID \ 2025-03-14T05:41:26.5188017Z  -e GITHUB_RUN_NUMBER \ 2025-03-14T05:41:26.5188320Z  -e GITHUB_RUN_ATTEMPT \ 2025-03-14T05:41:26.5188621Z  -e JOB_ID \ 2025-03-14T05:41:26.5188885Z  -e JOB_NAME \ 2025-03-14T05:41:26.5189155Z  -e BASE_SHA \ 2025-03-14T05:41:26.5189421Z  -e BRANCH \ 2025-03-14T05:41:26.5189682Z  -e SHA1 \ 2025-03-14T05:41:26.5189948Z  -e AWS_DEFAULT_REGION \ 2025-03-14T05:41:26.5190378Z  -e IN_WHEEL_TEST \ 2025-03-14T05:41:26.5190674Z  -e SHARD_NUMBER \ 2025-03-14T05:41:26.5190964Z  -e TEST_CONFIG \ 2025-03-14T05:41:26.5191260Z  -e NUM_TEST_SHARDS \ 2025-03-14T05:41:26.5191569Z  -e REENABLED_ISSUES \ 2025-03-14T05:41:26.5191886Z  -e CONTINUE_THROUGH_ERROR \ 2025-03-14T05:41:26.5192212Z  -e VERBOSE_TEST_LOGS \ 2025-03-14T05:41:26.5192530Z  -e TEST_SHOWLOCALS \ 2025-03-14T05:41:26.5192835Z  -e NO_TEST_TIMEOUT \ 2025-03-14T05:41:26.5193124Z  -e NO_TD \ 2025-03-14T05:41:26.5193398Z  -e TD_DISTRIBUTED \ 2025-03-14T05:41:26.5193697Z  -e PR_LABELS \ 2025-03-14T05:41:26.5194005Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-03-14T05:41:26.5194348Z  -e SCCACHE_BUCKET \ 2025-03-14T05:41:26.5194641Z  -e SCCACHE_REGION \ 2025-03-14T05:41:26.5194933Z  -e XLA_CUDA \ 2025-03-14T05:41:26.5195233Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-03-14T05:41:26.5195617Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-03-14T05:41:26.5195993Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-03-14T05:41:26.5196378Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-03-14T05:41:26.5196735Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-03-14T05:41:26.5197074Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-03-14T05:41:26.5197415Z  -e DASHBOARD_TAG \ 2025-03-14T05:41:26.5197711Z  -e IS_A100_RUNNER \ 2025-03-14T05:41:26.5198009Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-03-14T05:41:26.5198382Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-03-14T05:41:26.5198797Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-03-14T05:41:26.5199225Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-03-14T05:41:26.5199624Z  --security-opt seccomp=unconfined \ 2025-03-14T05:41:26.5199974Z  --cap-add=SYS_PTRACE \ 2025-03-14T05:41:26.5200283Z  --ipc=host \ 2025-03-14T05:41:26.5200556Z  ${SHM_OPTS} \ 2025-03-14T05:41:26.5200819Z  --tty \ 2025-03-14T05:41:26.5201074Z  --detach \ 2025-03-14T05:41:26.5201362Z  --name="${container_name}" \ 2025-03-14T05:41:26.5201692Z  ${JENKINS_USER} \ 2025-03-14T05:41:26.5202041Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-03-14T05:41:26.5202455Z  -w /var/lib/jenkins/workspace \ 2025-03-14T05:41:26.5202785Z  "${DOCKER_IMAGE}" \ 2025-03-14T05:41:26.5203087Z  ${DOCKER_SHELL_CMD} 2025-03-14T05:41:26.5203373Z ) 2025-03-14T05:41:26.5203689Z # Propagate download.pytorch.org IP to container 2025-03-14T05:41:26.5204351Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-03-14T05:41:26.5205042Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-03-14T05:41:26.5205461Z  2025-03-14T05:41:26.5205739Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-03-14T05:41:26.5206322Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-03-14T05:41:26.5206838Z fi 2025-03-14T05:41:26.5207071Z  2025-03-14T05:41:26.5207564Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-03-14T05:41:26.5216801Z shell: /usr/bin/bash -e {0} 2025-03-14T05:41:26.5217091Z env: 2025-03-14T05:41:26.5217333Z GIT_DEFAULT_BRANCH: main 2025-03-14T05:41:26.5217702Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:26.5218164Z BUILD_ENVIRONMENT: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:41:26.5218560Z PR_NUMBER: 2025-03-14T05:41:26.5218828Z GITHUB_REPOSITORY: pytorch/pytorch 2025-03-14T05:41:26.5219158Z GITHUB_WORKFLOW: inductor 2025-03-14T05:41:26.5219437Z GITHUB_JOB: test 2025-03-14T05:41:26.5219809Z GITHUB_RUN_ID: 13849515380 2025-03-14T05:41:26.5220091Z GITHUB_RUN_NUMBER: 122697 2025-03-14T05:41:26.5220368Z GITHUB_RUN_ATTEMPT: 1 2025-03-14T05:41:26.5220623Z JOB_ID: 38756916747 2025-03-14T05:41:26.5221096Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:26.5221606Z BRANCH: main 2025-03-14T05:41:26.5221885Z SHA1: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:26.5222277Z BASE_SHA: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:26.5222639Z TEST_CONFIG: inductor_torchbench 2025-03-14T05:41:26.5222937Z SHARD_NUMBER: 1 2025-03-14T05:41:26.5223189Z NUM_TEST_SHARDS: 2 2025-03-14T05:41:26.5223446Z REENABLED_ISSUES: 2025-03-14T05:41:26.5223717Z CONTINUE_THROUGH_ERROR: False 2025-03-14T05:41:26.5224003Z VERBOSE_TEST_LOGS: False 2025-03-14T05:41:26.5224283Z TEST_SHOWLOCALS: False 2025-03-14T05:41:26.5224556Z NO_TEST_TIMEOUT: False 2025-03-14T05:41:26.5224820Z NO_TD: False 2025-03-14T05:41:26.5225085Z TD_DISTRIBUTED: False 2025-03-14T05:41:26.5225414Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-03-14T05:41:26.5225793Z SCCACHE_REGION: us-east-1 2025-03-14T05:41:26.5226070Z SHM_SIZE: 2g 2025-03-14T05:41:26.5227356Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:41:26.5228211Z XLA_CUDA: 2025-03-14T05:41:26.5228585Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-03-14T05:41:26.5229045Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-03-14T05:41:26.5229377Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-03-14T05:41:26.5229692Z DASHBOARD_TAG: 2025-03-14T05:41:26.5230116Z HUGGING_FACE_HUB_TOKEN: *** 2025-03-14T05:41:26.5230540Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-03-14T05:41:26.5230844Z IS_A100_RUNNER: 0 2025-03-14T05:41:26.5231329Z ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T05:41:26.5231856Z ##[endgroup] 2025-03-14T05:41:26.5259459Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2025-03-14T05:41:26.5259898Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *onnx* ]] 2025-03-14T05:41:26.5260316Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-03-14T05:41:26.5262869Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-03-14T05:41:26.5285501Z + TOTAL_AVAILABLE_MEMORY_IN_GB='61.094 ' 2025-03-14T05:41:26.5285945Z + TOTAL_MEMORY_WITH_SWAP=64 2025-03-14T05:41:26.5286529Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]] 2025-03-14T05:41:26.5287319Z + SHM_OPTS=--shm-size=2g 2025-03-14T05:41:26.5287884Z + JENKINS_USER='--user jenkins' 2025-03-14T05:41:26.5288445Z + DOCKER_SHELL_CMD= 2025-03-14T05:41:26.5296719Z +++ nproc --ignore=2 2025-03-14T05:41:26.5347450Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --memory=61g --memory-swap=64g --env-file=/tmp/github_env_13849515380 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T05:41:32.6931637Z + container_name=f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T05:41:32.6938796Z + grep download.pytorch.org /etc/hosts 2025-03-14T05:41:32.6939520Z + docker exec -i f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de sudo bash -c '/bin/cat >> /etc/hosts' 2025-03-14T05:41:32.8376062Z + echo DOCKER_CONTAINER_ID=f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T05:41:32.8376708Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]] 2025-03-14T05:41:32.8380223Z ++ echo dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl 2025-03-14T05:41:32.8383302Z + docker exec -t f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de sh -c 'python3 -m pip install dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-03-14T05:41:33.3859594Z Processing ./dist/torch-2.8.0a0+gitaed0b7a-cp310-cp310-linux_x86_64.whl (from torch==2.8.0a0+gitaed0b7a) 2025-03-14T05:41:33.7362822Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.16.1) 2025-03-14T05:41:33.7366781Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (4.12.2) 2025-03-14T05:41:33.7854021Z Collecting sympy>=1.13.3 (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) 2025-03-14T05:41:33.7866281Z Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB) 2025-03-14T05:41:33.7883954Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (2.8.8) 2025-03-14T05:41:33.7887045Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.1.6) 2025-03-14T05:41:33.7890008Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (2024.10.0) 2025-03-14T05:41:33.7905142Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.3.0) 2025-03-14T05:41:33.7923125Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (1.22.4) 2025-03-14T05:41:33.7931115Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (1.3.0) 2025-03-14T05:41:33.8318531Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.8.0a0+gitaed0b7a->torch==2.8.0a0+gitaed0b7a) (3.0.2) 2025-03-14T05:41:33.8399174Z Using cached sympy-1.13.3-py3-none-any.whl (6.2 MB) 2025-03-14T05:41:34.4635901Z Installing collected packages: sympy, torch 2025-03-14T05:41:34.4638307Z Attempting uninstall: sympy 2025-03-14T05:41:34.4648508Z Found existing installation: sympy 1.13.1 2025-03-14T05:41:34.6652024Z Uninstalling sympy-1.13.1: 2025-03-14T05:41:35.8619325Z Successfully uninstalled sympy-1.13.1 2025-03-14T05:41:50.5187652Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-03-14T05:41:50.5188648Z timm 1.0.14 requires torchvision, which is not installed. 2025-03-14T05:41:50.5189227Z Successfully installed sympy-1.13.3 torch-2.8.0a0+gitaed0b7a 2025-03-14T05:41:50.6226491Z + export TERM=vt100 2025-03-14T05:41:50.6226940Z + TERM=vt100 2025-03-14T05:41:50.6234946Z ++ dirname .ci/pytorch/test.sh 2025-03-14T05:41:50.6248044Z + source .ci/pytorch/common.sh 2025-03-14T05:41:50.6252152Z +++ dirname .ci/pytorch/common.sh 2025-03-14T05:41:50.6262839Z ++ source .ci/pytorch/common_utils.sh 2025-03-14T05:41:50.6265005Z +++ declare -f -t trap_add 2025-03-14T05:41:50.6271739Z ++ set -ex -o pipefail 2025-03-14T05:41:50.6272207Z ++ [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-03-14T05:41:50.6272621Z ++ BUILD_TEST_LIBTORCH=0 2025-03-14T05:41:50.6282921Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *rocm* ]] 2025-03-14T05:41:50.6283419Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *s390x* ]] 2025-03-14T05:41:50.6283829Z + [[ -d /var/lib/jenkins/workspace ]] 2025-03-14T05:41:50.6284170Z ++ stat -c %u /var/lib/jenkins/workspace 2025-03-14T05:41:50.6319136Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-03-14T05:41:50.6319481Z + trap_add cleanup_workspace EXIT 2025-03-14T05:41:50.6319803Z + trap_add_cmd=cleanup_workspace 2025-03-14T05:41:50.6320094Z + shift 2025-03-14T05:41:50.6320341Z + for trap_add_name in "$@" 2025-03-14T05:41:50.6328122Z +++ trap -p EXIT 2025-03-14T05:41:50.6331696Z ++ eval 'extract_trap_cmd ' 2025-03-14T05:41:50.6331993Z +++ extract_trap_cmd 2025-03-14T05:41:50.6332250Z +++ printf '%s\n' '' 2025-03-14T05:41:50.6332525Z ++ printf '%s\n' cleanup_workspace 2025-03-14T05:41:50.6335457Z + trap -- ' 2025-03-14T05:41:50.6335715Z cleanup_workspace' EXIT 2025-03-14T05:41:50.6336037Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-03-14T05:41:51.4337475Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-03-14T05:41:51.4361780Z + echo 'Environment variables:' 2025-03-14T05:41:51.4362110Z Environment variables: 2025-03-14T05:41:51.4362371Z + env 2025-03-14T05:41:51.4387104Z INSTALLED_DB=yes 2025-03-14T05:41:51.4387496Z NV_LIBCUBLAS_VERSION=12.6.4.1-1 2025-03-14T05:41:51.4387979Z NVIDIA_VISIBLE_DEVICES=all 2025-03-14T05:41:51.4388401Z NV_NVML_DEV_VERSION=12.6.77-1 2025-03-14T05:41:51.4388912Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T05:41:51.4389430Z CONTINUE_THROUGH_ERROR=False 2025-03-14T05:41:51.4389776Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6 2025-03-14T05:41:51.4390151Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1 2025-03-14T05:41:51.4390556Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:41:51.4390939Z HOSTNAME=f31fd565c532 2025-03-14T05:41:51.4391503Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4392112Z GITHUB_ACTION=__self 2025-03-14T05:41:51.4392395Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-03-14T05:41:51.4397085Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 2025-03-14T05:41:51.4402124Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1 2025-03-14T05:41:51.4402498Z NV_NVTX_VERSION=12.6.77-1 2025-03-14T05:41:51.4402770Z GITHUB_RUN_NUMBER=122697 2025-03-14T05:41:51.4403054Z TEST_CONFIG=inductor_torchbench 2025-03-14T05:41:51.4403363Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-03-14T05:41:51.4403695Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-03-14T05:41:51.4404010Z IS_A100_RUNNER=0 2025-03-14T05:41:51.4404270Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1 2025-03-14T05:41:51.4404598Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1 2025-03-14T05:41:51.4405098Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-03-14T05:41:51.4405407Z NV_LIBNPP_VERSION=12.3.1.54-1 2025-03-14T05:41:51.4405723Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-03-14T05:41:51.4406096Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-03-14T05:41:51.4406452Z GITHUB_REF_TYPE=branch 2025-03-14T05:41:51.4406726Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-03-14T05:41:51.4407007Z NCCL_VERSION=2.23.4-1 2025-03-14T05:41:51.4407311Z BASE_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4407651Z XLA_CUDA= 2025-03-14T05:41:51.4407988Z HUGGING_FACE_HUB_TOKEN=*** 2025-03-14T05:41:51.4409837Z *** 2025-03-14T05:41:51.4410080Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-03-14T05:41:51.4410398Z GITHUB_REPOSITORY_ID=65600975 2025-03-14T05:41:51.4410678Z GITHUB_ACTIONS=true 2025-03-14T05:41:51.4410946Z NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:51.4411278Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1 2025-03-14T05:41:51.4411656Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1 2025-03-14T05:41:51.4412014Z SHA1=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4412366Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-03-14T05:41:51.4412727Z GITHUB_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4413239Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-03-14T05:41:51.4413706Z UCC_HOME=/usr 2025-03-14T05:41:51.4413959Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1 2025-03-14T05:41:51.4414268Z VERBOSE_TEST_LOGS=False 2025-03-14T05:41:51.4414541Z NVIDIA_PRODUCT_NAME=CUDA 2025-03-14T05:41:51.4414861Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6 2025-03-14T05:41:51.4415216Z GITHUB_REF=refs/heads/main 2025-03-14T05:41:51.4415504Z NV_CUDA_CUDART_VERSION=12.6.77-1 2025-03-14T05:41:51.4415793Z SHARD_NUMBER=1 2025-03-14T05:41:51.4416038Z GITHUB_REF_PROTECTED=true 2025-03-14T05:41:51.4416312Z HOME=/var/lib/jenkins 2025-03-14T05:41:51.4416599Z GITHUB_API_URL=https://api.github.com 2025-03-14T05:41:51.4416944Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-03-14T05:41:51.4417300Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-03-14T05:41:51.4417647Z CUDA_VERSION=12.6.3 2025-03-14T05:41:51.4417933Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1 2025-03-14T05:41:51.4418264Z NUM_TEST_SHARDS=2 2025-03-14T05:41:51.4418507Z UCX_HOME=/usr 2025-03-14T05:41:51.4418861Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1 2025-03-14T05:41:51.4419588Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4420407Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:51.4421202Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4421969Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-03-14T05:41:51.4422617Z GITHUB_EVENT_NAME=push 2025-03-14T05:41:51.4422879Z DASHBOARD_TAG= 2025-03-14T05:41:51.4423128Z GITHUB_RUN_ID=13849515380 2025-03-14T05:41:51.4423443Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1 2025-03-14T05:41:51.4423819Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6 2025-03-14T05:41:51.4424487Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4425137Z GITHUB_ACTOR=pytorchmergebot 2025-03-14T05:41:51.4425442Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1 2025-03-14T05:41:51.4425735Z PR_NUMBER= 2025-03-14T05:41:51.4426059Z GITHUB_RUN_ATTEMPT=1 2025-03-14T05:41:51.4426659Z ANACONDA_PYTHON_VERSION=3.10 2025-03-14T05:41:51.4427010Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-03-14T05:41:51.4427361Z TERM=vt100 2025-03-14T05:41:51.4427612Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1 2025-03-14T05:41:51.4427923Z INSTALLED_VISION=yes 2025-03-14T05:41:51.4428180Z BRANCH=main 2025-03-14T05:41:51.4428424Z SCCACHE_REGION=us-east-1 2025-03-14T05:41:51.4428712Z OPENSSL_ROOT_DIR=/opt/openssl 2025-03-14T05:41:51.4429020Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-03-14T05:41:51.4429346Z CUDA_PATH=/usr/local/cuda 2025-03-14T05:41:51.4429855Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-03-14T05:41:51.4430418Z GITHUB_SERVER_URL=https://github.com 2025-03-14T05:41:51.4430783Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-03-14T05:41:51.4431131Z REENABLED_ISSUES= 2025-03-14T05:41:51.4431366Z SHLVL=1 2025-03-14T05:41:51.4431576Z MAX_JOBS=14 2025-03-14T05:41:51.4431824Z NV_CUDA_LIB_VERSION=12.6.3-1 2025-03-14T05:41:51.4432100Z NVARCH=x86_64 2025-03-14T05:41:51.4432342Z GITHUB_ACTOR_ID=97764156 2025-03-14T05:41:51.4432694Z GITHUB_WORKFLOW_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4433079Z GITHUB_REF_NAME=main 2025-03-14T05:41:51.4433459Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-03-14T05:41:51.4433894Z GITHUB_JOB=test 2025-03-14T05:41:51.4434171Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6 2025-03-14T05:41:51.4434585Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-03-14T05:41:51.4434969Z NO_TEST_TIMEOUT=False 2025-03-14T05:41:51.4435230Z TD_DISTRIBUTED=False 2025-03-14T05:41:51.4435515Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1 2025-03-14T05:41:51.4435856Z GITHUB_REPOSITORY=pytorch/pytorch 2025-03-14T05:41:51.4436171Z NV_NVPROF_VERSION=12.6.80-1 2025-03-14T05:41:51.4436452Z GITHUB_RETENTION_DAYS=90 2025-03-14T05:41:51.4436730Z OPENSSL_DIR=/opt/openssl 2025-03-14T05:41:51.4437018Z GITHUB_ACTION_REPOSITORY= 2025-03-14T05:41:51.4437768Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T05:41:51.4438520Z GITHUB_BASE_REF= 2025-03-14T05:41:51.4438989Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T05:41:51.4439534Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-03-14T05:41:51.4439820Z CI=true 2025-03-14T05:41:51.4440066Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1 2025-03-14T05:41:51.4440387Z GITHUB_REPOSITORY_OWNER=pytorch 2025-03-14T05:41:51.4440677Z JOB_ID=38756916747 2025-03-14T05:41:51.4440932Z INSTALLED_PROTOBUF=yes 2025-03-14T05:41:51.4441194Z GITHUB_HEAD_REF= 2025-03-14T05:41:51.4441438Z GITHUB_ACTION_REF= 2025-03-14T05:41:51.4441741Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-03-14T05:41:51.4442128Z TEST_SHOWLOCALS=False 2025-03-14T05:41:51.4442422Z GITHUB_WORKFLOW=inductor 2025-03-14T05:41:51.4442702Z DEBIAN_FRONTEND=noninteractive 2025-03-14T05:41:51.4443307Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4443919Z NO_TD=False 2025-03-14T05:41:51.4444164Z SKIP_SCCACHE_INITIALIZATION=1 2025-03-14T05:41:51.4444448Z _=/usr/bin/env 2025-03-14T05:41:51.4444776Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-03-14T05:41:51.4565930Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-03-14T05:41:51.4566789Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-03-14T05:41:51.4567524Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-03-14T05:41:51.4568110Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-03-14T05:41:51.4568629Z + BUILD_DIR=build 2025-03-14T05:41:51.4568900Z + BUILD_RENAMED_DIR=build_renamed 2025-03-14T05:41:51.4569205Z + BUILD_BIN_DIR=build/bin 2025-03-14T05:41:51.4569683Z + SHARD_NUMBER=1 2025-03-14T05:41:51.4569938Z + NUM_TEST_SHARDS=2 2025-03-14T05:41:51.4570217Z + export TORCH_SERIALIZATION_DEBUG=1 2025-03-14T05:41:51.4570543Z + TORCH_SERIALIZATION_DEBUG=1 2025-03-14T05:41:51.4570834Z + export VALGRIND=ON 2025-03-14T05:41:51.4571172Z + VALGRIND=ON 2025-03-14T05:41:51.4571511Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *clang9* ]] 2025-03-14T05:41:51.4572040Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]] 2025-03-14T05:41:51.4572645Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *s390x* ]] 2025-03-14T05:41:51.4573066Z + [[ 0 == \1 ]] 2025-03-14T05:41:51.4573304Z + [[ False == \1 ]] 2025-03-14T05:41:51.4573614Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *bazel* ]] 2025-03-14T05:41:51.4574010Z ++ realpath build/custom_test_artifacts 2025-03-14T05:41:51.4594482Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-03-14T05:41:51.4595026Z + [[ -n '' ]] 2025-03-14T05:41:51.4595292Z + echo 'Environment variables' 2025-03-14T05:41:51.4595591Z Environment variables 2025-03-14T05:41:51.4595850Z + env 2025-03-14T05:41:51.4603450Z INSTALLED_DB=yes 2025-03-14T05:41:51.4603823Z NV_LIBCUBLAS_VERSION=12.6.4.1-1 2025-03-14T05:41:51.4604270Z NVIDIA_VISIBLE_DEVICES=all 2025-03-14T05:41:51.4604836Z NV_NVML_DEV_VERSION=12.6.77-1 2025-03-14T05:41:51.4605855Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T05:41:51.4606409Z CONTINUE_THROUGH_ERROR=False 2025-03-14T05:41:51.4606778Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6 2025-03-14T05:41:51.4607229Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1 2025-03-14T05:41:51.4607667Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-03-14T05:41:51.4608073Z HOSTNAME=f31fd565c532 2025-03-14T05:41:51.4608717Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4609353Z GITHUB_ACTION=__self 2025-03-14T05:41:51.4609650Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-03-14T05:41:51.4613997Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 2025-03-14T05:41:51.4618550Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1 2025-03-14T05:41:51.4619140Z NV_NVTX_VERSION=12.6.77-1 2025-03-14T05:41:51.4619430Z GITHUB_RUN_NUMBER=122697 2025-03-14T05:41:51.4619717Z TEST_CONFIG=inductor_torchbench 2025-03-14T05:41:51.4620027Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-03-14T05:41:51.4620369Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-03-14T05:41:51.4620687Z IS_A100_RUNNER=0 2025-03-14T05:41:51.4620948Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1 2025-03-14T05:41:51.4621280Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1 2025-03-14T05:41:51.4621752Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-03-14T05:41:51.4622078Z NV_LIBNPP_VERSION=12.3.1.54-1 2025-03-14T05:41:51.4622527Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-03-14T05:41:51.4622907Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-03-14T05:41:51.4623275Z GITHUB_REF_TYPE=branch 2025-03-14T05:41:51.4623551Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-03-14T05:41:51.4623839Z NCCL_VERSION=2.23.4-1 2025-03-14T05:41:51.4624143Z BASE_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4624488Z XLA_CUDA= 2025-03-14T05:41:51.4624840Z HUGGING_FACE_HUB_TOKEN=*** 2025-03-14T05:41:51.4625353Z *** 2025-03-14T05:41:51.4625589Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-03-14T05:41:51.4625914Z GITHUB_REPOSITORY_ID=65600975 2025-03-14T05:41:51.4626579Z GITHUB_ACTIONS=true 2025-03-14T05:41:51.4626861Z NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T05:41:51.4627201Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1 2025-03-14T05:41:51.4627576Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1 2025-03-14T05:41:51.4627937Z SHA1=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4628296Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-03-14T05:41:51.4628662Z GITHUB_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4629179Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main 2025-03-14T05:41:51.4629653Z UCC_HOME=/usr 2025-03-14T05:41:51.4629896Z TORCH_SERIALIZATION_DEBUG=1 2025-03-14T05:41:51.4630196Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1 2025-03-14T05:41:51.4630512Z VERBOSE_TEST_LOGS=False 2025-03-14T05:41:51.4630790Z NVIDIA_PRODUCT_NAME=CUDA 2025-03-14T05:41:51.4631114Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6 2025-03-14T05:41:51.4631471Z GITHUB_REF=refs/heads/main 2025-03-14T05:41:51.4631762Z NV_CUDA_CUDART_VERSION=12.6.77-1 2025-03-14T05:41:51.4632089Z SHARD_NUMBER=1 2025-03-14T05:41:51.4632376Z GITHUB_REF_PROTECTED=true 2025-03-14T05:41:51.4632657Z HOME=/var/lib/jenkins 2025-03-14T05:41:51.4632950Z GITHUB_API_URL=https://api.github.com 2025-03-14T05:41:51.4633297Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-03-14T05:41:51.4633656Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-03-14T05:41:51.4634012Z CUDA_VERSION=12.6.3 2025-03-14T05:41:51.4634461Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1 2025-03-14T05:41:51.4634795Z NUM_TEST_SHARDS=2 2025-03-14T05:41:51.4635048Z UCX_HOME=/usr 2025-03-14T05:41:51.4635416Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1 2025-03-14T05:41:51.4636147Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4636983Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T05:41:51.4637792Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4638578Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-03-14T05:41:51.4639085Z GITHUB_EVENT_NAME=push 2025-03-14T05:41:51.4639365Z DASHBOARD_TAG= 2025-03-14T05:41:51.4639614Z GITHUB_RUN_ID=13849515380 2025-03-14T05:41:51.4639935Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1 2025-03-14T05:41:51.4640321Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6 2025-03-14T05:41:51.4641001Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4641665Z GITHUB_ACTOR=pytorchmergebot 2025-03-14T05:41:51.4641984Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1 2025-03-14T05:41:51.4642447Z PR_NUMBER= 2025-03-14T05:41:51.4642682Z GITHUB_RUN_ATTEMPT=1 2025-03-14T05:41:51.4642934Z VALGRIND=ON 2025-03-14T05:41:51.4643180Z ANACONDA_PYTHON_VERSION=3.10 2025-03-14T05:41:51.4643531Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-03-14T05:41:51.4643885Z TERM=vt100 2025-03-14T05:41:51.4644133Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1 2025-03-14T05:41:51.4644445Z INSTALLED_VISION=yes 2025-03-14T05:41:51.4644701Z BRANCH=main 2025-03-14T05:41:51.4644940Z SCCACHE_REGION=us-east-1 2025-03-14T05:41:51.4645230Z OPENSSL_ROOT_DIR=/opt/openssl 2025-03-14T05:41:51.4645804Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-03-14T05:41:51.4646141Z CUDA_PATH=/usr/local/cuda 2025-03-14T05:41:51.4646655Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-03-14T05:41:51.4647221Z GITHUB_SERVER_URL=https://github.com 2025-03-14T05:41:51.4647601Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-03-14T05:41:51.4647967Z REENABLED_ISSUES= 2025-03-14T05:41:51.4648217Z SHLVL=1 2025-03-14T05:41:51.4648433Z MAX_JOBS=14 2025-03-14T05:41:51.4648682Z NV_CUDA_LIB_VERSION=12.6.3-1 2025-03-14T05:41:51.4648964Z NVARCH=x86_64 2025-03-14T05:41:51.4649219Z GITHUB_ACTOR_ID=97764156 2025-03-14T05:41:51.4649571Z GITHUB_WORKFLOW_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T05:41:51.4649962Z GITHUB_REF_NAME=main 2025-03-14T05:41:51.4650357Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-03-14T05:41:51.4650798Z GITHUB_JOB=test 2025-03-14T05:41:51.4651088Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6 2025-03-14T05:41:51.4651507Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-03-14T05:41:51.4651893Z NO_TEST_TIMEOUT=False 2025-03-14T05:41:51.4652158Z TD_DISTRIBUTED=False 2025-03-14T05:41:51.4652455Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1 2025-03-14T05:41:51.4652801Z GITHUB_REPOSITORY=pytorch/pytorch 2025-03-14T05:41:51.4653128Z NV_NVPROF_VERSION=12.6.80-1 2025-03-14T05:41:51.4653432Z GITHUB_RETENTION_DAYS=90 2025-03-14T05:41:51.4653718Z OPENSSL_DIR=/opt/openssl 2025-03-14T05:41:51.4654002Z GITHUB_ACTION_REPOSITORY= 2025-03-14T05:41:51.4654753Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T05:41:51.4655521Z GITHUB_BASE_REF= 2025-03-14T05:41:51.4655985Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T05:41:51.4656520Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-03-14T05:41:51.4656803Z CI=true 2025-03-14T05:41:51.4657053Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1 2025-03-14T05:41:51.4657376Z GITHUB_REPOSITORY_OWNER=pytorch 2025-03-14T05:41:51.4657666Z JOB_ID=38756916747 2025-03-14T05:41:51.4657922Z INSTALLED_PROTOBUF=yes 2025-03-14T05:41:51.4658191Z GITHUB_HEAD_REF= 2025-03-14T05:41:51.4658439Z GITHUB_ACTION_REF= 2025-03-14T05:41:51.4658747Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-03-14T05:41:51.4659116Z TEST_SHOWLOCALS=False 2025-03-14T05:41:51.4659384Z GITHUB_WORKFLOW=inductor 2025-03-14T05:41:51.4659671Z DEBIAN_FRONTEND=noninteractive 2025-03-14T05:41:51.4660275Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_18711753-8966-4b50-aafe-c558eed40d4f 2025-03-14T05:41:51.4660882Z NO_TD=False 2025-03-14T05:41:51.4661128Z SKIP_SCCACHE_INITIALIZATION=1 2025-03-14T05:41:51.4661419Z _=/usr/bin/env 2025-03-14T05:41:51.4661664Z + echo 'Testing pytorch' 2025-03-14T05:41:51.4661929Z Testing pytorch 2025-03-14T05:41:51.4662235Z + export LANG=C.UTF-8 2025-03-14T05:41:51.4662504Z + LANG=C.UTF-8 2025-03-14T05:41:51.4688085Z + PR_NUMBER= 2025-03-14T05:41:51.4688481Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2025-03-14T05:41:51.4689004Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-03-14T05:41:51.4689552Z + [[ inductor_torchbench == \s\l\o\w ]] 2025-03-14T05:41:51.4690034Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2025-03-14T05:41:51.4690683Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *cuda* ]] 2025-03-14T05:41:51.4691399Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-03-14T05:41:51.4691867Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-03-14T05:41:51.4692214Z + [[ inductor_torchbench == *crossref* ]] 2025-03-14T05:41:51.4692605Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-03-14T05:41:51.4693037Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]] 2025-03-14T05:41:51.4693474Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]] 2025-03-14T05:41:51.4693869Z + pip_install --user ninja==1.10.2 2025-03-14T05:41:51.4694374Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:41:51.4694866Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2025-03-14T05:41:52.0063876Z Collecting ninja==1.10.2 2025-03-14T05:41:52.0410560Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-03-14T05:41:52.0524246Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-03-14T05:41:52.6452241Z Installing collected packages: ninja 2025-03-14T05:41:52.6532981Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2025-03-14T05:41:52.6533893Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2025-03-14T05:41:52.6591357Z Successfully installed ninja-1.10.2 2025-03-14T05:41:52.7548479Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T05:41:52.7549987Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T05:41:52.7550930Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]] 2025-03-14T05:41:52.7551324Z + install_tlparse 2025-03-14T05:41:52.7551606Z + pip_install --user tlparse==0.3.30 2025-03-14T05:41:52.7552017Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:41:52.7552565Z + python3 -m pip install --progress-bar off --user tlparse==0.3.30 2025-03-14T05:41:53.2288949Z Collecting tlparse==0.3.30 2025-03-14T05:41:53.2635173Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB) 2025-03-14T05:41:53.2775009Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB) 2025-03-14T05:41:53.9054290Z Installing collected packages: tlparse 2025-03-14T05:41:53.9897514Z Successfully installed tlparse-0.3.30 2025-03-14T05:41:54.0867806Z ++ python -m site --user-base 2025-03-14T05:41:54.1055465Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-03-14T05:41:54.1058756Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *asan* ]] 2025-03-14T05:41:54.1059970Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-debug* ]] 2025-03-14T05:41:54.1061124Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]] 2025-03-14T05:41:54.1062571Z + echo 'We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass' 2025-03-14T05:41:54.1063430Z We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass 2025-03-14T05:41:54.1063974Z + cd test 2025-03-14T05:41:54.1064384Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-03-14T05:41:57.1560061Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-03-14T05:41:57.1560516Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-03-14T05:41:57.1566059Z + DYNAMO_BENCHMARK_FLAGS=() 2025-03-14T05:41:57.1567250Z + [[ inductor_torchbench == *pr_time_benchmarks* ]] 2025-03-14T05:41:57.1567712Z + [[ inductor_torchbench == *dynamo_eager* ]] 2025-03-14T05:41:57.1568476Z + [[ inductor_torchbench == *aot_eager* ]] 2025-03-14T05:41:57.1568850Z + [[ inductor_torchbench == *aot_inductor* ]] 2025-03-14T05:41:57.1569309Z + [[ inductor_torchbench == *max_autotune_inductor* ]] 2025-03-14T05:41:57.1569691Z + [[ inductor_torchbench == *inductor* ]] 2025-03-14T05:41:57.1570101Z + [[ inductor_torchbench != *perf* ]] 2025-03-14T05:41:57.1570462Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-03-14T05:41:57.1570846Z + [[ inductor_torchbench == *dynamic* ]] 2025-03-14T05:41:57.1571415Z + [[ inductor_torchbench == *cpu* ]] 2025-03-14T05:41:57.1571838Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-03-14T05:41:57.1601820Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]] 2025-03-14T05:41:57.1602454Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-bazel-* ]] 2025-03-14T05:41:57.1605813Z + cd test 2025-03-14T05:41:57.1606314Z + python -c 'import torch; print(torch.__config__.show())' 2025-03-14T05:41:58.7211233Z PyTorch built with: 2025-03-14T05:41:58.7211836Z - GCC 9.4 2025-03-14T05:41:58.7212306Z - C++ Version: 201703 2025-03-14T05:41:58.7213399Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-03-14T05:41:58.7214276Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-03-14T05:41:58.7214731Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-03-14T05:41:58.7215086Z - LAPACK is enabled (usually provided by MKL) 2025-03-14T05:41:58.7215432Z - NNPACK is enabled 2025-03-14T05:41:58.7215727Z - CPU capability usage: AVX2 2025-03-14T05:41:58.7216028Z - CUDA Runtime 12.6 2025-03-14T05:41:58.7216392Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2025-03-14T05:41:58.7216790Z - CuDNN 90.5.1 2025-03-14T05:41:58.7217038Z - Magma 2.6.1 2025-03-14T05:41:58.7221242Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=aed0b7a742a2d7b7901790622829cbd2135049a4, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2025-03-14T05:41:58.7225615Z 2025-03-14T05:41:59.0642211Z + cd test 2025-03-14T05:41:59.0642618Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-03-14T05:42:00.4742393Z ATen/Parallel: 2025-03-14T05:42:00.4742706Z at::get_num_threads() : 8 2025-03-14T05:42:00.4743015Z at::get_num_interop_threads() : 16 2025-03-14T05:42:00.4743348Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-03-14T05:42:00.4743656Z omp_get_max_threads() : 8 2025-03-14T05:42:00.4744206Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-03-14T05:42:00.4744808Z mkl_get_max_threads() : 8 2025-03-14T05:42:00.4745215Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-03-14T05:42:00.4745673Z std::thread::hardware_concurrency() : 16 2025-03-14T05:42:00.4746008Z Environment variables: 2025-03-14T05:42:00.4746287Z OMP_NUM_THREADS : [not set] 2025-03-14T05:42:00.4746646Z MKL_NUM_THREADS : [not set] 2025-03-14T05:42:00.4747176Z ATen parallel backend: OpenMP 2025-03-14T05:42:00.4747373Z 2025-03-14T05:42:00.7745230Z + [[ inductor_torchbench == *numpy_2* ]] 2025-03-14T05:42:00.7745654Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]] 2025-03-14T05:42:00.7746061Z + [[ inductor_torchbench == *backward* ]] 2025-03-14T05:42:00.7746402Z + [[ inductor_torchbench == *xla* ]] 2025-03-14T05:42:00.7746809Z + [[ inductor_torchbench == *executorch* ]] 2025-03-14T05:42:00.7747177Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2025-03-14T05:42:00.7747811Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]] 2025-03-14T05:42:00.7748219Z + [[ inductor_torchbench == distributed ]] 2025-03-14T05:42:00.7748605Z + [[ inductor_torchbench == *inductor_distributed* ]] 2025-03-14T05:42:00.7748994Z + [[ inductor_torchbench == *inductor-halide* ]] 2025-03-14T05:42:00.7749384Z + [[ inductor_torchbench == *inductor-triton-cpu* ]] 2025-03-14T05:42:00.7749807Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2025-03-14T05:42:00.7750208Z + [[ inductor_torchbench == *huggingface* ]] 2025-03-14T05:42:00.7750549Z + [[ inductor_torchbench == *timm* ]] 2025-03-14T05:42:00.7750878Z + [[ inductor_torchbench == cachebench ]] 2025-03-14T05:42:00.7751231Z + [[ inductor_torchbench == verify_cachebench ]] 2025-03-14T05:42:00.7751591Z + [[ inductor_torchbench == *torchbench* ]] 2025-03-14T05:42:00.7751925Z + [[ inductor_torchbench == *cpu* ]] 2025-03-14T05:42:00.7752239Z + install_torchaudio cuda 2025-03-14T05:42:00.7752504Z + local commit 2025-03-14T05:42:00.7752757Z ++ get_pinned_commit audio 2025-03-14T05:42:00.7753058Z ++ cat .github/ci_commit_pins/audio.txt 2025-03-14T05:42:00.7775715Z + commit=c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:00.7776093Z + [[ cuda == \c\u\d\a ]] 2025-03-14T05:42:00.7776470Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2025-03-14T05:42:00.7777077Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:00.7777765Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:42:00.7778556Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:01.1745186Z Collecting git+https://github.com/pytorch/audio.git@c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:01.1749207Z Cloning https://github.com/pytorch/audio.git (to revision c670ad81fda266b6598aeeef434583eb98197ae8) to /tmp/pip-req-build-dje_b90x 2025-03-14T05:42:01.1782487Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-dje_b90x 2025-03-14T05:42:02.0139258Z Running command git rev-parse -q --verify 'sha^c670ad81fda266b6598aeeef434583eb98197ae8' 2025-03-14T05:42:02.0167554Z Running command git fetch -q https://github.com/pytorch/audio.git c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:02.4363259Z Resolved https://github.com/pytorch/audio.git to commit c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T05:42:02.4364533Z Running command git submodule update --init --recursive -q 2025-03-14T05:42:05.0666362Z Preparing metadata (setup.py) ... [?25l- done 2025-03-14T05:42:05.0696408Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.6.0a0+c670ad8) (2.8.0a0+gitaed0b7a) 2025-03-14T05:42:05.0720080Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (3.16.1) 2025-03-14T05:42:05.0723625Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (4.12.2) 2025-03-14T05:42:05.0727702Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (1.13.3) 2025-03-14T05:42:05.0730861Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (2.8.8) 2025-03-14T05:42:05.0734019Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (3.1.6) 2025-03-14T05:42:05.0736917Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+c670ad8) (2024.10.0) 2025-03-14T05:42:05.0751910Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchaudio==2.6.0a0+c670ad8) (1.3.0) 2025-03-14T05:42:05.1219943Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.6.0a0+c670ad8) (3.0.2) 2025-03-14T05:42:05.1285527Z Building wheels for collected packages: torchaudio 2025-03-14T05:44:32.7582672Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2025-03-14T05:44:32.7621596Z [?25h Created wheel for torchaudio: filename=torchaudio-2.6.0a0+c670ad8-cp310-cp310-linux_x86_64.whl size=2396041 sha256=d8ffca102f781fd0edfadb4a4fee8b964f29056cfa566dfeb2fb78948107decc 2025-03-14T05:44:32.7623436Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d3/a5/a0/32c6aea783135ce13a14d2b2ccb6f8d5ce3b45e87b2c1918f9 2025-03-14T05:44:32.7654323Z Successfully built torchaudio 2025-03-14T05:44:33.2803266Z Installing collected packages: torchaudio 2025-03-14T05:44:33.4958247Z Successfully installed torchaudio-2.6.0a0+c670ad8 2025-03-14T05:44:33.6991257Z + install_torchvision 2025-03-14T05:44:33.6991598Z + local orig_preload 2025-03-14T05:44:33.6991857Z + local commit 2025-03-14T05:44:33.6995643Z ++ get_pinned_commit vision 2025-03-14T05:44:33.6996039Z ++ cat .github/ci_commit_pins/vision.txt 2025-03-14T05:44:33.7019250Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:33.7019746Z + orig_preload= 2025-03-14T05:44:33.7020080Z + '[' -n '' ']' 2025-03-14T05:44:33.7020721Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:33.7021424Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:44:33.7022223Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:34.1294491Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:34.1301456Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-nw02xaa2 2025-03-14T05:44:34.1334321Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-nw02xaa2 2025-03-14T05:44:35.6784491Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2025-03-14T05:44:35.6811942Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:37.0982076Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:37.4597948Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T05:44:40.1776929Z Preparing metadata (setup.py) ... [?25l- \ done 2025-03-14T05:44:40.1810216Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2025-03-14T05:44:40.1813594Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.8.0a0+gitaed0b7a) 2025-03-14T05:44:40.1818228Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0) 2025-03-14T05:44:40.1884723Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1) 2025-03-14T05:44:40.1888738Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2025-03-14T05:44:40.1892517Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.3) 2025-03-14T05:44:40.1895775Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2025-03-14T05:44:40.1898809Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.6) 2025-03-14T05:44:40.1901986Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0) 2025-03-14T05:44:40.1917168Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2025-03-14T05:44:40.2393111Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2) 2025-03-14T05:44:40.2460508Z Building wheels for collected packages: torchvision 2025-03-14T05:45:56.0946388Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2025-03-14T05:45:56.0978496Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2076332 sha256=d5bd3242b706adca5e07e3f4b77a5e8ac08285199a7d99b9ac42587589e54d02 2025-03-14T05:45:56.0981429Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2025-03-14T05:45:56.1016011Z Successfully built torchvision 2025-03-14T05:45:56.6177427Z Installing collected packages: torchvision 2025-03-14T05:45:57.0332442Z Successfully installed torchvision-0.19.0a0+d23a6e1 2025-03-14T05:45:57.1974126Z + '[' -n '' ']' 2025-03-14T05:45:57.1974426Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2025-03-14T05:45:57.1974812Z + pip_install git+https://github.com/pytorch/ao.git 2025-03-14T05:45:57.1975280Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:45:57.1975838Z + python3 -m pip install --progress-bar off git+https://github.com/pytorch/ao.git 2025-03-14T05:45:57.5922827Z Collecting git+https://github.com/pytorch/ao.git 2025-03-14T05:45:57.5927059Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-nfozqao2 2025-03-14T05:45:57.5958282Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-nfozqao2 2025-03-14T05:45:58.4590303Z Resolved https://github.com/pytorch/ao.git to commit 9259584f98db0760b27492a63050a2915c753dbe 2025-03-14T05:45:58.4591002Z Running command git submodule update --init --recursive -q 2025-03-14T05:46:04.0773737Z Preparing metadata (setup.py) ... [?25l- done 2025-03-14T05:46:04.0814065Z [?25hBuilding wheels for collected packages: torchao 2025-03-14T05:51:10.6901699Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / done 2025-03-14T05:51:10.6961995Z [?25h Created wheel for torchao: filename=torchao-0.10.0+git9259584f-cp39-abi3-linux_x86_64.whl size=5142683 sha256=2b495cdfc9b60b0e8e3a5fd66291f39b0b162db974c2dd9b4d99fb4c19b0f869 2025-03-14T05:51:10.6964579Z Stored in directory: /tmp/pip-ephem-wheel-cache-12zkiypv/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2025-03-14T05:51:10.7001514Z Successfully built torchao 2025-03-14T05:51:11.2755474Z Installing collected packages: torchao 2025-03-14T05:51:11.9395176Z Successfully installed torchao-0.10.0+git9259584f 2025-03-14T05:51:12.3984450Z + id=0 2025-03-14T05:51:12.3989861Z + pip_install opencv-python==4.8.0.74 2025-03-14T05:51:12.3990667Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T05:51:12.3991638Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74 2025-03-14T05:51:12.9382009Z Collecting opencv-python==4.8.0.74 2025-03-14T05:51:12.9733622Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2025-03-14T05:51:12.9813284Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2025-03-14T05:51:12.9880075Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2025-03-14T05:51:13.9770401Z Installing collected packages: opencv-python 2025-03-14T05:51:14.8429479Z Successfully installed opencv-python-4.8.0.74 2025-03-14T05:51:14.9525002Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2025-03-14T05:51:14.9528027Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2025-03-14T05:51:14.9528564Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2025-03-14T05:51:14.9528946Z + checkout_install_torchbench 2025-03-14T05:51:14.9533501Z + local commit 2025-03-14T05:51:14.9547465Z ++ get_pinned_commit torchbench 2025-03-14T05:51:14.9547816Z ++ cat .github/ci_commit_pins/torchbench.txt 2025-03-14T05:51:14.9612849Z + commit=373ffb19dc470f4423a3176a4133f8f4b3cdb5bd 2025-03-14T05:51:14.9613728Z + git clone https://github.com/pytorch/benchmark torchbench 2025-03-14T05:51:14.9630197Z Cloning into 'torchbench'... 2025-03-14T05:51:15.1524426Z remote: Enumerating objects: 35366, done. 2025-03-14T05:51:15.1525108Z remote: Counting objects: 0% (1/5443) 2025-03-14T05:51:15.1525698Z remote: Counting objects: 1% (55/5443) 2025-03-14T05:51:15.1526462Z remote: Counting objects: 2% (109/5443) 2025-03-14T05:51:15.1527100Z remote: Counting objects: 3% (164/5443) 2025-03-14T05:51:15.1527881Z remote: Counting objects: 4% (218/5443) 2025-03-14T05:51:15.1528460Z remote: Counting objects: 5% (273/5443) 2025-03-14T05:51:15.1528952Z remote: Counting objects: 6% (327/5443) 2025-03-14T05:51:15.1529415Z remote: Counting objects: 7% (382/5443) 2025-03-14T05:51:15.1530549Z remote: Counting objects: 8% (436/5443) 2025-03-14T05:51:15.1531108Z remote: Counting objects: 9% (490/5443) 2025-03-14T05:51:15.1531645Z remote: Counting objects: 10% (545/5443) 2025-03-14T05:51:15.1532052Z remote: Counting objects: 11% (599/5443) 2025-03-14T05:51:15.1532774Z remote: Counting objects: 12% (654/5443) 2025-03-14T05:51:15.1533214Z remote: Counting objects: 13% (708/5443) 2025-03-14T05:51:15.1533619Z remote: Counting objects: 14% (763/5443) 2025-03-14T05:51:15.1534019Z remote: Counting objects: 15% (817/5443) 2025-03-14T05:51:15.1534418Z remote: Counting objects: 16% (871/5443) 2025-03-14T05:51:15.1534822Z remote: Counting objects: 17% (926/5443) 2025-03-14T05:51:15.1535222Z remote: Counting objects: 18% (980/5443) 2025-03-14T05:51:15.1539318Z remote: Counting objects: 19% (1035/5443) 2025-03-14T05:51:15.1539784Z remote: Counting objects: 20% (1089/5443) 2025-03-14T05:51:15.1540201Z remote: Counting objects: 21% (1144/5443) 2025-03-14T05:51:15.1540613Z remote: Counting objects: 22% (1198/5443) 2025-03-14T05:51:15.1541018Z remote: Counting objects: 23% (1252/5443) 2025-03-14T05:51:15.1541427Z remote: Counting objects: 24% (1307/5443) 2025-03-14T05:51:15.1541840Z remote: Counting objects: 25% (1361/5443) 2025-03-14T05:51:15.1542510Z remote: Counting objects: 26% (1416/5443) 2025-03-14T05:51:15.1542912Z remote: Counting objects: 27% (1470/5443) 2025-03-14T05:51:15.1543318Z remote: Counting objects: 28% (1525/5443) 2025-03-14T05:51:15.1543726Z remote: Counting objects: 29% (1579/5443) 2025-03-14T05:51:15.1544128Z remote: Counting objects: 30% (1633/5443) 2025-03-14T05:51:15.1544527Z remote: Counting objects: 31% (1688/5443) 2025-03-14T05:51:15.1544928Z remote: Counting objects: 32% (1742/5443) 2025-03-14T05:51:15.1545328Z remote: Counting objects: 33% (1797/5443) 2025-03-14T05:51:15.1545918Z remote: Counting objects: 34% (1851/5443) 2025-03-14T05:51:15.1546338Z remote: Counting objects: 35% (1906/5443) 2025-03-14T05:51:15.1546852Z remote: Counting objects: 36% (1960/5443) 2025-03-14T05:51:15.1547258Z remote: Counting objects: 37% (2014/5443) 2025-03-14T05:51:15.1547664Z remote: Counting objects: 38% (2069/5443) 2025-03-14T05:51:15.1548063Z remote: Counting objects: 39% (2123/5443) 2025-03-14T05:51:15.1548471Z remote: Counting objects: 40% (2178/5443) 2025-03-14T05:51:15.1548873Z remote: Counting objects: 41% (2232/5443) 2025-03-14T05:51:15.1549276Z remote: Counting objects: 42% (2287/5443) 2025-03-14T05:51:15.1549682Z remote: Counting objects: 43% (2341/5443) 2025-03-14T05:51:15.1553890Z remote: Counting objects: 44% (2395/5443) 2025-03-14T05:51:15.1554514Z remote: Counting objects: 45% (2450/5443) 2025-03-14T05:51:15.1554932Z remote: Counting objects: 46% (2504/5443) 2025-03-14T05:51:15.1555359Z remote: Counting objects: 47% (2559/5443) 2025-03-14T05:51:15.1555829Z remote: Counting objects: 48% (2613/5443) 2025-03-14T05:51:15.1556234Z remote: Counting objects: 49% (2668/5443) 2025-03-14T05:51:15.1556638Z remote: Counting objects: 50% (2722/5443) 2025-03-14T05:51:15.1557040Z remote: Counting objects: 51% (2776/5443) 2025-03-14T05:51:15.1557528Z remote: Counting objects: 52% (2831/5443) 2025-03-14T05:51:15.1558064Z remote: Counting objects: 53% (2885/5443) 2025-03-14T05:51:15.1558491Z remote: Counting objects: 54% (2940/5443) 2025-03-14T05:51:15.1558896Z remote: Counting objects: 55% (2994/5443) 2025-03-14T05:51:15.1559315Z remote: Counting objects: 56% (3049/5443) 2025-03-14T05:51:15.1559789Z remote: Counting objects: 57% (3103/5443) 2025-03-14T05:51:15.1560205Z remote: Counting objects: 58% (3157/5443) 2025-03-14T05:51:15.1560662Z remote: Counting objects: 59% (3212/5443) 2025-03-14T05:51:15.1561071Z remote: Counting objects: 60% (3266/5443) 2025-03-14T05:51:15.1561480Z remote: Counting objects: 61% (3321/5443) 2025-03-14T05:51:15.1561893Z remote: Counting objects: 62% (3375/5443) 2025-03-14T05:51:15.1562301Z remote: Counting objects: 63% (3430/5443) 2025-03-14T05:51:15.1562715Z remote: Counting objects: 64% (3484/5443) 2025-03-14T05:51:15.1563118Z remote: Counting objects: 65% (3538/5443) 2025-03-14T05:51:15.1563556Z remote: Counting objects: 66% (3593/5443) 2025-03-14T05:51:15.1564064Z remote: Counting objects: 67% (3647/5443) 2025-03-14T05:51:15.1564472Z remote: Counting objects: 68% (3702/5443) 2025-03-14T05:51:15.1564884Z remote: Counting objects: 69% (3756/5443) 2025-03-14T05:51:15.1565303Z remote: Counting objects: 70% (3811/5443) 2025-03-14T05:51:15.1565785Z remote: Counting objects: 71% (3865/5443) 2025-03-14T05:51:15.1566197Z remote: Counting objects: 72% (3919/5443) 2025-03-14T05:51:15.1566606Z remote: Counting objects: 73% (3974/5443) 2025-03-14T05:51:15.1567010Z remote: Counting objects: 74% (4028/5443) 2025-03-14T05:51:15.1567417Z remote: Counting objects: 75% (4083/5443) 2025-03-14T05:51:15.1567821Z remote: Counting objects: 76% (4137/5443) 2025-03-14T05:51:15.1568228Z remote: Counting objects: 77% (4192/5443) 2025-03-14T05:51:15.1568634Z remote: Counting objects: 78% (4246/5443) 2025-03-14T05:51:15.1569039Z remote: Counting objects: 79% (4300/5443) 2025-03-14T05:51:15.1569444Z remote: Counting objects: 80% (4355/5443) 2025-03-14T05:51:15.1570918Z remote: Counting objects: 81% (4409/5443) 2025-03-14T05:51:15.1571325Z remote: Counting objects: 82% (4464/5443) 2025-03-14T05:51:15.1571730Z remote: Counting objects: 83% (4518/5443) 2025-03-14T05:51:15.1572133Z remote: Counting objects: 84% (4573/5443) 2025-03-14T05:51:15.1572536Z remote: Counting objects: 85% (4627/5443) 2025-03-14T05:51:15.1572940Z remote: Counting objects: 86% (4681/5443) 2025-03-14T05:51:15.1573341Z remote: Counting objects: 87% (4736/5443) 2025-03-14T05:51:15.1573870Z remote: Counting objects: 88% (4790/5443) 2025-03-14T05:51:15.1574278Z remote: Counting objects: 89% (4845/5443) 2025-03-14T05:51:15.1574685Z remote: Counting objects: 90% (4899/5443) 2025-03-14T05:51:15.1575094Z remote: Counting objects: 91% (4954/5443) 2025-03-14T05:51:15.1575498Z remote: Counting objects: 92% (5008/5443) 2025-03-14T05:51:15.1575892Z remote: Counting objects: 93% (5062/5443) 2025-03-14T05:51:15.1576309Z remote: Counting objects: 94% (5117/5443) 2025-03-14T05:51:15.1576710Z remote: Counting objects: 95% (5171/5443) 2025-03-14T05:51:15.1577115Z remote: Counting objects: 96% (5226/5443) 2025-03-14T05:51:15.1577523Z remote: Counting objects: 97% (5280/5443) 2025-03-14T05:51:15.1577938Z remote: Counting objects: 98% (5335/5443) 2025-03-14T05:51:15.1578347Z remote: Counting objects: 99% (5389/5443) 2025-03-14T05:51:15.1578754Z remote: Counting objects: 100% (5443/5443) 2025-03-14T05:51:15.1579198Z remote: Counting objects: 100% (5443/5443), done. 2025-03-14T05:51:15.1579649Z remote: Compressing objects: 0% (1/605) 2025-03-14T05:51:15.1598665Z remote: Compressing objects: 1% (7/605) 2025-03-14T05:51:15.1624158Z remote: Compressing objects: 2% (13/605) 2025-03-14T05:51:15.1641486Z remote: Compressing objects: 3% (19/605) 2025-03-14T05:51:15.1660432Z remote: Compressing objects: 4% (25/605) 2025-03-14T05:51:15.1698077Z remote: Compressing objects: 5% (31/605) 2025-03-14T05:51:15.1718293Z remote: Compressing objects: 6% (37/605) 2025-03-14T05:51:15.1741881Z remote: Compressing objects: 7% (43/605) 2025-03-14T05:51:15.1755874Z remote: Compressing objects: 8% (49/605) 2025-03-14T05:51:15.1767011Z remote: Compressing objects: 9% (55/605) 2025-03-14T05:51:15.1770565Z remote: Compressing objects: 10% (61/605) 2025-03-14T05:51:15.1911517Z remote: Compressing objects: 11% (67/605) 2025-03-14T05:51:15.2012371Z remote: Compressing objects: 12% (73/605) 2025-03-14T05:51:15.2034642Z remote: Compressing objects: 13% (79/605) 2025-03-14T05:51:15.2152218Z remote: Compressing objects: 14% (85/605) 2025-03-14T05:51:15.2211383Z remote: Compressing objects: 15% (91/605) 2025-03-14T05:51:15.2240246Z remote: Compressing objects: 16% (97/605) 2025-03-14T05:51:15.2282018Z remote: Compressing objects: 17% (103/605) 2025-03-14T05:51:15.2329043Z remote: Compressing objects: 18% (109/605) 2025-03-14T05:51:15.2363814Z remote: Compressing objects: 19% (115/605) 2025-03-14T05:51:15.2379347Z remote: Compressing objects: 20% (121/605) 2025-03-14T05:51:15.2414915Z remote: Compressing objects: 21% (128/605) 2025-03-14T05:51:15.2444705Z remote: Compressing objects: 22% (134/605) 2025-03-14T05:51:15.2477711Z remote: Compressing objects: 23% (140/605) 2025-03-14T05:51:15.2499751Z remote: Compressing objects: 24% (146/605) 2025-03-14T05:51:15.2522670Z remote: Compressing objects: 25% (152/605) 2025-03-14T05:51:15.2540777Z remote: Compressing objects: 26% (158/605) 2025-03-14T05:51:15.2559830Z remote: Compressing objects: 27% (164/605) 2025-03-14T05:51:15.2573941Z remote: Compressing objects: 28% (170/605) 2025-03-14T05:51:15.2593627Z remote: Compressing objects: 29% (176/605) 2025-03-14T05:51:15.2608681Z remote: Compressing objects: 30% (182/605) 2025-03-14T05:51:15.2627001Z remote: Compressing objects: 31% (188/605) 2025-03-14T05:51:15.2642473Z remote: Compressing objects: 32% (194/605) 2025-03-14T05:51:15.2655223Z remote: Compressing objects: 33% (200/605) 2025-03-14T05:51:15.2669234Z remote: Compressing objects: 34% (206/605) 2025-03-14T05:51:15.2680060Z remote: Compressing objects: 35% (212/605) 2025-03-14T05:51:15.2689335Z remote: Compressing objects: 36% (218/605) 2025-03-14T05:51:15.2694725Z remote: Compressing objects: 37% (224/605) 2025-03-14T05:51:15.2703626Z remote: Compressing objects: 38% (230/605) 2025-03-14T05:51:15.2708803Z remote: Compressing objects: 39% (236/605) 2025-03-14T05:51:15.2712701Z remote: Compressing objects: 40% (242/605) 2025-03-14T05:51:15.2715203Z remote: Compressing objects: 41% (249/605) 2025-03-14T05:51:15.2716167Z remote: Compressing objects: 42% (255/605) 2025-03-14T05:51:15.2716596Z remote: Compressing objects: 43% (261/605) 2025-03-14T05:51:15.2719231Z remote: Compressing objects: 44% (267/605) 2025-03-14T05:51:15.2719765Z remote: Compressing objects: 45% (273/605) 2025-03-14T05:51:15.2721805Z remote: Compressing objects: 46% (279/605) 2025-03-14T05:51:15.2725195Z remote: Compressing objects: 47% (285/605) 2025-03-14T05:51:15.2737497Z remote: Compressing objects: 48% (291/605) 2025-03-14T05:51:15.2743049Z remote: Compressing objects: 49% (297/605) 2025-03-14T05:51:15.2745740Z remote: Compressing objects: 50% (303/605) 2025-03-14T05:51:15.2746271Z remote: Compressing objects: 51% (309/605) 2025-03-14T05:51:15.2747726Z remote: Compressing objects: 52% (315/605) 2025-03-14T05:51:15.2750874Z remote: Compressing objects: 53% (321/605) 2025-03-14T05:51:15.2754649Z remote: Compressing objects: 54% (327/605) 2025-03-14T05:51:15.2755210Z remote: Compressing objects: 55% (333/605) 2025-03-14T05:51:15.2757773Z remote: Compressing objects: 56% (339/605) 2025-03-14T05:51:15.2759833Z remote: Compressing objects: 57% (345/605) 2025-03-14T05:51:15.2762014Z remote: Compressing objects: 58% (351/605) 2025-03-14T05:51:15.2764500Z remote: Compressing objects: 59% (357/605) 2025-03-14T05:51:15.2766510Z remote: Compressing objects: 60% (363/605) 2025-03-14T05:51:15.2770464Z remote: Compressing objects: 61% (370/605) 2025-03-14T05:51:15.2771560Z remote: Compressing objects: 62% (376/605) 2025-03-14T05:51:15.2775101Z remote: Compressing objects: 63% (382/605) 2025-03-14T05:51:15.2776927Z remote: Compressing objects: 64% (388/605) 2025-03-14T05:51:15.2779499Z remote: Compressing objects: 65% (394/605) 2025-03-14T05:51:15.2783488Z remote: Compressing objects: 66% (400/605) 2025-03-14T05:51:15.2784639Z remote: Compressing objects: 67% (406/605) 2025-03-14T05:51:15.2786985Z remote: Compressing objects: 68% (412/605) 2025-03-14T05:51:15.2788394Z remote: Compressing objects: 69% (418/605) 2025-03-14T05:51:15.2790836Z remote: Compressing objects: 70% (424/605) 2025-03-14T05:51:15.2793342Z remote: Compressing objects: 71% (430/605) 2025-03-14T05:51:15.2795654Z remote: Compressing objects: 72% (436/605) 2025-03-14T05:51:15.2796814Z remote: Compressing objects: 73% (442/605) 2025-03-14T05:51:15.2799462Z remote: Compressing objects: 74% (448/605) 2025-03-14T05:51:15.2801949Z remote: Compressing objects: 75% (454/605) 2025-03-14T05:51:15.2804316Z remote: Compressing objects: 76% (460/605) 2025-03-14T05:51:15.2805104Z remote: Compressing objects: 77% (466/605) 2025-03-14T05:51:15.2805647Z remote: Compressing objects: 78% (472/605) 2025-03-14T05:51:15.2806418Z remote: Compressing objects: 79% (478/605) 2025-03-14T05:51:15.2807612Z remote: Compressing objects: 80% (484/605) 2025-03-14T05:51:15.2809019Z remote: Compressing objects: 81% (491/605) 2025-03-14T05:51:15.2809757Z remote: Compressing objects: 82% (497/605) 2025-03-14T05:51:15.2811437Z remote: Compressing objects: 83% (503/605) 2025-03-14T05:51:15.2812972Z remote: Compressing objects: 84% (509/605) 2025-03-14T05:51:15.2813485Z remote: Compressing objects: 85% (515/605) 2025-03-14T05:51:15.2813904Z remote: Compressing objects: 86% (521/605) 2025-03-14T05:51:15.2814621Z remote: Compressing objects: 87% (527/605) 2025-03-14T05:51:15.2819343Z remote: Compressing objects: 88% (533/605) 2025-03-14T05:51:15.2819779Z remote: Compressing objects: 89% (539/605) 2025-03-14T05:51:15.2820198Z remote: Compressing objects: 90% (545/605) 2025-03-14T05:51:15.2820623Z remote: Compressing objects: 91% (551/605) 2025-03-14T05:51:15.2824086Z remote: Compressing objects: 92% (557/605) 2025-03-14T05:51:15.2825821Z remote: Compressing objects: 93% (563/605) 2025-03-14T05:51:15.2828495Z remote: Compressing objects: 94% (569/605) 2025-03-14T05:51:15.2829518Z remote: Compressing objects: 95% (575/605) 2025-03-14T05:51:15.2830284Z remote: Compressing objects: 96% (581/605) 2025-03-14T05:51:15.2831271Z remote: Compressing objects: 97% (587/605) 2025-03-14T05:51:15.2832387Z remote: Compressing objects: 98% (593/605) 2025-03-14T05:51:15.2832895Z remote: Compressing objects: 99% (599/605) 2025-03-14T05:51:15.2833688Z remote: Compressing objects: 100% (605/605) 2025-03-14T05:51:15.2834793Z remote: Compressing objects: 100% (605/605), done. 2025-03-14T05:51:15.2940888Z Receiving objects: 0% (1/35366) 2025-03-14T05:51:15.2988555Z Receiving objects: 1% (354/35366) 2025-03-14T05:51:15.3036125Z Receiving objects: 2% (708/35366) 2025-03-14T05:51:15.3076909Z Receiving objects: 3% (1061/35366) 2025-03-14T05:51:15.3117553Z Receiving objects: 4% (1415/35366) 2025-03-14T05:51:15.3156831Z Receiving objects: 5% (1769/35366) 2025-03-14T05:51:15.3197905Z Receiving objects: 6% (2122/35366) 2025-03-14T05:51:15.3240634Z Receiving objects: 7% (2476/35366) 2025-03-14T05:51:15.3276787Z Receiving objects: 8% (2830/35366) 2025-03-14T05:51:15.3326790Z Receiving objects: 9% (3183/35366) 2025-03-14T05:51:15.3384790Z Receiving objects: 10% (3537/35366) 2025-03-14T05:51:15.3463057Z Receiving objects: 11% (3891/35366) 2025-03-14T05:51:15.3596479Z Receiving objects: 12% (4244/35366) 2025-03-14T05:51:15.3753926Z Receiving objects: 13% (4598/35366) 2025-03-14T05:51:15.3825867Z Receiving objects: 14% (4952/35366) 2025-03-14T05:51:15.3870855Z Receiving objects: 15% (5305/35366) 2025-03-14T05:51:15.3902569Z Receiving objects: 16% (5659/35366) 2025-03-14T05:51:15.3952043Z Receiving objects: 17% (6013/35366) 2025-03-14T05:51:15.3985324Z Receiving objects: 18% (6366/35366) 2025-03-14T05:51:15.4014072Z Receiving objects: 19% (6720/35366) 2025-03-14T05:51:15.6840446Z Receiving objects: 20% (7074/35366) 2025-03-14T05:51:16.2861101Z Receiving objects: 21% (7427/35366) 2025-03-14T05:51:17.0514330Z Receiving objects: 21% (7501/35366), 69.41 MiB | 69.41 MiB/s 2025-03-14T05:51:17.1386597Z Receiving objects: 22% (7781/35366), 106.49 MiB | 70.99 MiB/s 2025-03-14T05:51:17.2272209Z Receiving objects: 23% (8135/35366), 106.49 MiB | 70.99 MiB/s 2025-03-14T05:51:17.2862302Z Receiving objects: 24% (8488/35366), 106.49 MiB | 70.99 MiB/s 2025-03-14T05:51:17.3164263Z Receiving objects: 24% (8722/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.4049281Z Receiving objects: 25% (8842/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.4942424Z Receiving objects: 26% (9196/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.5832868Z Receiving objects: 27% (9549/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.6720478Z Receiving objects: 28% (9903/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.9436456Z Receiving objects: 29% (10257/35366), 140.44 MiB | 70.22 MiB/s 2025-03-14T05:51:17.9583113Z Receiving objects: 30% (10610/35366), 176.20 MiB | 70.48 MiB/s 2025-03-14T05:51:18.2657753Z Receiving objects: 31% (10964/35366), 176.20 MiB | 70.48 MiB/s 2025-03-14T05:51:18.2861673Z Receiving objects: 32% (11318/35366), 176.20 MiB | 70.48 MiB/s 2025-03-14T05:51:18.4882974Z Receiving objects: 32% (11328/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4900526Z Receiving objects: 33% (11671/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4919516Z Receiving objects: 34% (12025/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4933395Z Receiving objects: 35% (12379/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4944445Z Receiving objects: 36% (12732/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4961939Z Receiving objects: 37% (13086/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4975383Z Receiving objects: 38% (13440/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.4982188Z Receiving objects: 39% (13793/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5007260Z Receiving objects: 40% (14147/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5022157Z Receiving objects: 41% (14501/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5227600Z Receiving objects: 42% (14854/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5234019Z Receiving objects: 43% (15208/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5242229Z Receiving objects: 44% (15562/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5259085Z Receiving objects: 45% (15915/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5272084Z Receiving objects: 46% (16269/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5290824Z Receiving objects: 47% (16623/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5300553Z Receiving objects: 48% (16976/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5445282Z Receiving objects: 49% (17330/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5453326Z Receiving objects: 50% (17683/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5463141Z Receiving objects: 51% (18037/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.5722183Z Receiving objects: 52% (18391/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6144202Z Receiving objects: 53% (18744/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6155658Z Receiving objects: 54% (19098/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6171238Z Receiving objects: 55% (19452/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6187323Z Receiving objects: 56% (19805/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6199581Z Receiving objects: 57% (20159/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6221095Z Receiving objects: 58% (20513/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6229615Z Receiving objects: 59% (20866/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6239467Z Receiving objects: 60% (21220/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6460453Z Receiving objects: 61% (21574/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6493987Z Receiving objects: 62% (21927/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6507163Z Receiving objects: 63% (22281/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6518304Z Receiving objects: 64% (22635/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6573871Z Receiving objects: 65% (22988/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6718479Z Receiving objects: 66% (23342/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6725956Z Receiving objects: 67% (23696/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6734878Z Receiving objects: 68% (24049/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6743201Z Receiving objects: 69% (24403/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6749744Z Receiving objects: 70% (24757/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6864588Z Receiving objects: 71% (25110/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6880978Z Receiving objects: 72% (25464/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6895068Z Receiving objects: 73% (25818/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6916606Z Receiving objects: 74% (26171/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6948314Z Receiving objects: 75% (26525/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6967640Z Receiving objects: 76% (26879/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.6993454Z Receiving objects: 77% (27232/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7012974Z Receiving objects: 78% (27586/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7038947Z Receiving objects: 79% (27940/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7135420Z Receiving objects: 80% (28293/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7151914Z Receiving objects: 81% (28647/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7173331Z Receiving objects: 82% (29001/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7209079Z Receiving objects: 83% (29354/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7306014Z Receiving objects: 84% (29708/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7313349Z Receiving objects: 85% (30062/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7317690Z Receiving objects: 86% (30415/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7330869Z Receiving objects: 87% (30769/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7331480Z Receiving objects: 88% (31123/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7331946Z Receiving objects: 89% (31476/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7333183Z Receiving objects: 90% (31830/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7338806Z Receiving objects: 91% (32184/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7344957Z Receiving objects: 92% (32537/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7349219Z Receiving objects: 93% (32891/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7353103Z Receiving objects: 94% (33245/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7375089Z Receiving objects: 95% (33598/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7441441Z Receiving objects: 96% (33952/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7458322Z Receiving objects: 97% (34306/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7496139Z Receiving objects: 98% (34659/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7536446Z Receiving objects: 99% (35013/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7537250Z remote: Total 35366 (delta 5141), reused 4845 (delta 4838), pack-reused 29923 (from 2) 2025-03-14T05:51:18.7554218Z Receiving objects: 100% (35366/35366), 204.63 MiB | 68.21 MiB/s 2025-03-14T05:51:18.7554703Z Receiving objects: 100% (35366/35366), 233.84 MiB | 67.41 MiB/s, done. 2025-03-14T05:51:18.7588644Z Resolving deltas: 0% (0/19971) 2025-03-14T05:51:18.7605689Z Resolving deltas: 1% (230/19971) 2025-03-14T05:51:18.7651307Z Resolving deltas: 2% (411/19971) 2025-03-14T05:51:18.7680654Z Resolving deltas: 3% (609/19971) 2025-03-14T05:51:18.7712205Z Resolving deltas: 4% (800/19971) 2025-03-14T05:51:18.7782604Z Resolving deltas: 5% (1008/19971) 2025-03-14T05:51:18.7824442Z Resolving deltas: 6% (1208/19971) 2025-03-14T05:51:18.7847468Z Resolving deltas: 7% (1409/19971) 2025-03-14T05:51:18.7861217Z Resolving deltas: 8% (1639/19971) 2025-03-14T05:51:18.7893491Z Resolving deltas: 9% (1799/19971) 2025-03-14T05:51:18.7906923Z Resolving deltas: 10% (2004/19971) 2025-03-14T05:51:18.7928121Z Resolving deltas: 11% (2199/19971) 2025-03-14T05:51:18.7960957Z Resolving deltas: 12% (2430/19971) 2025-03-14T05:51:18.7983687Z Resolving deltas: 13% (2605/19971) 2025-03-14T05:51:18.8004269Z Resolving deltas: 14% (2800/19971) 2025-03-14T05:51:18.8031685Z Resolving deltas: 15% (3005/19971) 2025-03-14T05:51:18.8053619Z Resolving deltas: 16% (3197/19971) 2025-03-14T05:51:18.8080972Z Resolving deltas: 17% (3396/19971) 2025-03-14T05:51:18.8108720Z Resolving deltas: 18% (3605/19971) 2025-03-14T05:51:18.8135477Z Resolving deltas: 19% (3799/19971) 2025-03-14T05:51:18.8185216Z Resolving deltas: 20% (3996/19971) 2025-03-14T05:51:18.8210754Z Resolving deltas: 21% (4217/19971) 2025-03-14T05:51:18.8226710Z Resolving deltas: 22% (4394/19971) 2025-03-14T05:51:18.8241104Z Resolving deltas: 23% (4595/19971) 2025-03-14T05:51:18.8266318Z Resolving deltas: 24% (4794/19971) 2025-03-14T05:51:18.8286858Z Resolving deltas: 25% (4994/19971) 2025-03-14T05:51:18.8299984Z Resolving deltas: 26% (5234/19971) 2025-03-14T05:51:18.8308592Z Resolving deltas: 27% (5396/19971) 2025-03-14T05:51:18.8315430Z Resolving deltas: 28% (5611/19971) 2025-03-14T05:51:18.8329811Z Resolving deltas: 29% (5803/19971) 2025-03-14T05:51:18.8342196Z Resolving deltas: 30% (6007/19971) 2025-03-14T05:51:18.8356935Z Resolving deltas: 31% (6216/19971) 2025-03-14T05:51:18.8367828Z Resolving deltas: 32% (6392/19971) 2025-03-14T05:51:18.8379955Z Resolving deltas: 33% (6594/19971) 2025-03-14T05:51:18.8394501Z Resolving deltas: 34% (6796/19971) 2025-03-14T05:51:18.8407069Z Resolving deltas: 35% (6990/19971) 2025-03-14T05:51:18.8423097Z Resolving deltas: 36% (7190/19971) 2025-03-14T05:51:18.8442621Z Resolving deltas: 37% (7403/19971) 2025-03-14T05:51:18.8455509Z Resolving deltas: 38% (7589/19971) 2025-03-14T05:51:18.8464607Z Resolving deltas: 39% (7789/19971) 2025-03-14T05:51:18.8473777Z Resolving deltas: 40% (7989/19971) 2025-03-14T05:51:18.8482042Z Resolving deltas: 41% (8195/19971) 2025-03-14T05:51:18.8489856Z Resolving deltas: 42% (8394/19971) 2025-03-14T05:51:18.8498522Z Resolving deltas: 43% (8589/19971) 2025-03-14T05:51:18.8506534Z Resolving deltas: 44% (8798/19971) 2025-03-14T05:51:18.8515989Z Resolving deltas: 45% (8989/19971) 2025-03-14T05:51:18.8523092Z Resolving deltas: 46% (9209/19971) 2025-03-14T05:51:18.8537942Z Resolving deltas: 47% (9397/19971) 2025-03-14T05:51:18.8544129Z Resolving deltas: 48% (9669/19971) 2025-03-14T05:51:18.8559905Z Resolving deltas: 49% (9806/19971) 2025-03-14T05:51:18.8567626Z Resolving deltas: 50% (9996/19971) 2025-03-14T05:51:18.8580024Z Resolving deltas: 51% (10186/19971) 2025-03-14T05:51:18.8590878Z Resolving deltas: 52% (10388/19971) 2025-03-14T05:51:18.8600456Z Resolving deltas: 53% (10595/19971) 2025-03-14T05:51:18.8615029Z Resolving deltas: 54% (10793/19971) 2025-03-14T05:51:18.8633872Z Resolving deltas: 55% (10999/19971) 2025-03-14T05:51:18.8649518Z Resolving deltas: 56% (11219/19971) 2025-03-14T05:51:18.8661645Z Resolving deltas: 57% (11400/19971) 2025-03-14T05:51:18.8673360Z Resolving deltas: 58% (11590/19971) 2025-03-14T05:51:18.8688957Z Resolving deltas: 59% (11785/19971) 2025-03-14T05:51:18.8702786Z Resolving deltas: 60% (11983/19971) 2025-03-14T05:51:18.8714850Z Resolving deltas: 61% (12191/19971) 2025-03-14T05:51:18.8723973Z Resolving deltas: 62% (12407/19971) 2025-03-14T05:51:18.8735451Z Resolving deltas: 63% (12582/19971) 2025-03-14T05:51:18.8748506Z Resolving deltas: 64% (12783/19971) 2025-03-14T05:51:18.8760972Z Resolving deltas: 65% (12986/19971) 2025-03-14T05:51:18.8769008Z Resolving deltas: 66% (13220/19971) 2025-03-14T05:51:18.8782486Z Resolving deltas: 67% (13384/19971) 2025-03-14T05:51:18.8793532Z Resolving deltas: 68% (13581/19971) 2025-03-14T05:51:18.8803871Z Resolving deltas: 69% (13780/19971) 2025-03-14T05:51:18.8819922Z Resolving deltas: 70% (13982/19971) 2025-03-14T05:51:18.8830481Z Resolving deltas: 71% (14225/19971) 2025-03-14T05:51:18.8843288Z Resolving deltas: 72% (14380/19971) 2025-03-14T05:51:18.8873865Z Resolving deltas: 73% (14579/19971) 2025-03-14T05:51:18.8930592Z Resolving deltas: 74% (14781/19971) 2025-03-14T05:51:18.8938561Z Resolving deltas: 77% (15530/19971) 2025-03-14T05:51:18.8946022Z Resolving deltas: 78% (15658/19971) 2025-03-14T05:51:18.8962766Z Resolving deltas: 79% (15808/19971) 2025-03-14T05:51:18.8981647Z Resolving deltas: 80% (15984/19971) 2025-03-14T05:51:18.8995847Z Resolving deltas: 81% (16179/19971) 2025-03-14T05:51:18.9023147Z Resolving deltas: 82% (16389/19971) 2025-03-14T05:51:18.9044112Z Resolving deltas: 84% (16945/19971) 2025-03-14T05:51:18.9051314Z Resolving deltas: 86% (17355/19971) 2025-03-14T05:51:18.9060228Z Resolving deltas: 87% (17479/19971) 2025-03-14T05:51:18.9068759Z Resolving deltas: 88% (17641/19971) 2025-03-14T05:51:18.9084266Z Resolving deltas: 89% (17776/19971) 2025-03-14T05:51:18.9095137Z Resolving deltas: 90% (18029/19971) 2025-03-14T05:51:18.9104040Z Resolving deltas: 91% (18210/19971) 2025-03-14T05:51:18.9112244Z Resolving deltas: 92% (18399/19971) 2025-03-14T05:51:18.9130132Z Resolving deltas: 93% (18574/19971) 2025-03-14T05:51:18.9142729Z Resolving deltas: 94% (18795/19971) 2025-03-14T05:51:18.9165445Z Resolving deltas: 95% (18977/19971) 2025-03-14T05:51:18.9182763Z Resolving deltas: 96% (19205/19971) 2025-03-14T05:51:18.9230333Z Resolving deltas: 97% (19387/19971) 2025-03-14T05:51:18.9312295Z Resolving deltas: 98% (19592/19971) 2025-03-14T05:51:18.9490249Z Resolving deltas: 99% (19777/19971) 2025-03-14T05:51:18.9490865Z Resolving deltas: 100% (19971/19971) 2025-03-14T05:51:18.9491278Z Resolving deltas: 100% (19971/19971), done. 2025-03-14T05:51:19.9594725Z + pushd torchbench 2025-03-14T05:51:19.9595053Z ~/workspace/torchbench ~/workspace 2025-03-14T05:51:19.9595706Z + git checkout 373ffb19dc470f4423a3176a4133f8f4b3cdb5bd 2025-03-14T05:51:20.0018849Z Note: switching to '373ffb19dc470f4423a3176a4133f8f4b3cdb5bd'. 2025-03-14T05:51:20.0019174Z 2025-03-14T05:51:20.0019393Z You are in 'detached HEAD' state. You can look around, make experimental 2025-03-14T05:51:20.0019928Z changes and commit them, and you can discard any commits you make in this 2025-03-14T05:51:20.0020480Z state without impacting any branches by switching back to a branch. 2025-03-14T05:51:20.0020790Z 2025-03-14T05:51:20.0021006Z If you want to create a new branch to retain commits you create, you may 2025-03-14T05:51:20.0021505Z do so (now or later) by using -c with the switch command. Example: 2025-03-14T05:51:20.0021788Z 2025-03-14T05:51:20.0021914Z git switch -c 2025-03-14T05:51:20.0022113Z 2025-03-14T05:51:20.0022234Z Or undo this operation with: 2025-03-14T05:51:20.0022418Z 2025-03-14T05:51:20.0022523Z git switch - 2025-03-14T05:51:20.0022663Z 2025-03-14T05:51:20.0022913Z Turn off this advice by setting config variable advice.detachedHead to false 2025-03-14T05:51:20.0023250Z 2025-03-14T05:51:20.0023469Z HEAD is now at 373ffb19 Copy model before benchmark warmup runs (#145858) 2025-03-14T05:51:20.0025397Z + '[' '' ']' 2025-03-14T05:51:20.0025664Z + python install.py --continue_on_fail 2025-03-14T05:51:28.1763108Z checking packages numpy, torch, torchvision, torchaudio are installed, generating constaints...OK 2025-03-14T05:52:02.9217843Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2025-03-14T05:52:18.2773158Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2025-03-14T05:52:33.7095949Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2025-03-14T05:52:48.7997257Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2025-03-14T05:53:02.2011003Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2025-03-14T05:53:23.8199126Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2025-03-14T05:53:40.1189037Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2025-03-14T05:53:56.3812836Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2025-03-14T05:54:12.6537409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2025-03-14T05:54:12.6539087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2025-03-14T05:54:27.1766058Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2025-03-14T05:54:43.3993525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2025-03-14T05:54:56.7957104Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2025-03-14T05:56:08.2114293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2025-03-14T05:56:31.7733050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2025-03-14T05:56:54.2033389Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2025-03-14T05:57:16.2456112Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2025-03-14T05:57:39.7192717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2025-03-14T05:58:01.8435332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2025-03-14T05:58:23.4061188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2025-03-14T05:58:45.5395479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2025-03-14T05:59:07.7986013Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2025-03-14T05:59:30.1710585Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2025-03-14T05:59:52.2619247Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2025-03-14T06:00:14.5174365Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2025-03-14T06:00:29.3985209Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2025-03-14T06:01:05.4308959Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2025-03-14T06:01:25.9078316Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2025-03-14T06:01:45.0063243Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2025-03-14T06:02:09.4007985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2025-03-14T06:02:24.3396627Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2025-03-14T06:02:39.6395883Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2025-03-14T06:03:01.6477784Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2025-03-14T06:03:23.9481902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2025-03-14T06:03:44.7577971Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2025-03-14T06:04:07.7082092Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2025-03-14T06:04:28.9417728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2025-03-14T06:04:49.2489823Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2025-03-14T06:05:10.7459161Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2025-03-14T06:05:42.1024479Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2025-03-14T06:06:03.5875728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2025-03-14T06:06:22.8432401Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2025-03-14T06:06:47.8615864Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Roberta_base...OK 2025-03-14T06:07:07.9792322Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2025-03-14T06:07:30.0795888Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2025-03-14T06:07:30.0798463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2025-03-14T06:07:58.2934110Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2025-03-14T06:08:16.2413751Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2025-03-14T06:08:16.2416838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2025-03-14T06:08:39.1002659Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2025-03-14T06:08:54.1385442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2025-03-14T06:09:09.1867210Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2025-03-14T06:10:08.1511853Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2025-03-14T06:11:50.6868800Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2025-03-14T06:12:04.4234203Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2025-03-14T06:12:19.8153495Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2025-03-14T06:12:19.8158886Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/microbench_unbacked_tolist_sum...SKIP - No install.py is found 2025-03-14T06:12:33.5827265Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2025-03-14T06:12:47.3188397Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2025-03-14T06:13:01.0750799Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2025-03-14T06:13:14.8036562Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2025-03-14T06:13:28.5457016Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2025-03-14T06:14:04.4781312Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2025-03-14T06:14:04.4784030Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2025-03-14T06:14:19.5263847Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2025-03-14T06:14:35.9376965Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2025-03-14T06:14:49.6871163Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2025-03-14T06:15:03.4057723Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2025-03-14T06:15:17.1639112Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2025-03-14T06:15:30.9252676Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2025-03-14T06:15:44.6524267Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2025-03-14T06:16:05.4102678Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2025-03-14T06:16:20.6704016Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2025-03-14T06:16:38.2538258Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2025-03-14T06:16:52.0040464Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2025-03-14T06:17:05.7597456Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2025-03-14T06:17:19.5163014Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2025-03-14T06:17:33.2603463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2025-03-14T06:17:47.0257231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2025-03-14T06:18:10.0920219Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2025-03-14T06:18:34.0841209Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2025-03-14T06:18:47.8595723Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2025-03-14T06:18:47.8599559Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2025-03-14T06:18:47.8602602Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2025-03-14T06:19:04.4096845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2025-03-14T06:19:19.9176233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2025-03-14T06:19:33.5804613Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2025-03-14T06:20:12.2836062Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2025-03-14T06:20:33.5474204Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2025-03-14T06:20:52.0171401Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2025-03-14T06:21:12.8236638Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2025-03-14T06:21:26.5030692Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2025-03-14T06:21:40.1913914Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2025-03-14T06:21:53.8501209Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2025-03-14T06:22:07.5313173Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2025-03-14T06:22:21.2111990Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2025-03-14T06:22:34.8750809Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2025-03-14T06:22:48.5529286Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2025-03-14T06:23:08.5932815Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2025-03-14T06:23:28.2284911Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2025-03-14T06:23:41.8917771Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2025-03-14T06:23:57.3122114Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2025-03-14T06:24:13.0503449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2025-03-14T06:24:19.8619212Z installed torchbench with package constraints: {'numpy': '1.22.4', 'torch': '2.8.0a0+gitaed0b7a', 'torchvision': '0.19.0a0+d23a6e1', 'torchaudio': '2.6.0a0+c670ad8'} 2025-03-14T06:24:20.1756167Z + pip install transformers==4.38.1 2025-03-14T06:24:20.6971368Z Collecting transformers==4.38.1 2025-03-14T06:24:20.7354695Z Downloading transformers-4.38.1-py3-none-any.whl.metadata (131 kB) 2025-03-14T06:24:20.9405638Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (3.16.1) 2025-03-14T06:24:20.9409506Z Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.29.3) 2025-03-14T06:24:20.9413589Z Requirement already satisfied: numpy>=1.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (1.22.4) 2025-03-14T06:24:20.9417775Z Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (24.2) 2025-03-14T06:24:20.9421464Z Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (6.0.2) 2025-03-14T06:24:20.9425557Z Requirement already satisfied: regex!=2019.12.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2024.11.6) 2025-03-14T06:24:20.9429532Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2.32.3) 2025-03-14T06:24:21.0925560Z Collecting tokenizers<0.19,>=0.14 (from transformers==4.38.1) 2025-03-14T06:24:21.0940119Z Using cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) 2025-03-14T06:24:21.0965194Z Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.5.3) 2025-03-14T06:24:21.0969080Z Requirement already satisfied: tqdm>=4.27 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (4.67.1) 2025-03-14T06:24:21.1186370Z Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (2024.10.0) 2025-03-14T06:24:21.1193449Z Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (4.12.2) 2025-03-14T06:24:21.1372266Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.4.1) 2025-03-14T06:24:21.1376975Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.10) 2025-03-14T06:24:21.1379760Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2.3.0) 2025-03-14T06:24:21.1382743Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2025.1.31) 2025-03-14T06:24:21.1824300Z Downloading transformers-4.38.1-py3-none-any.whl (8.5 MB) 2025-03-14T06:24:21.2351965Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.5 MB ? eta -:--:-- 2025-03-14T06:24:21.2352838Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 MB 176.8 MB/s eta 0:00:00 2025-03-14T06:24:21.2369861Z [?25hUsing cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) 2025-03-14T06:24:22.2376669Z Installing collected packages: tokenizers, transformers 2025-03-14T06:24:22.2377204Z Attempting uninstall: tokenizers 2025-03-14T06:24:22.2393362Z Found existing installation: tokenizers 0.19.1 2025-03-14T06:24:22.2437432Z Uninstalling tokenizers-0.19.1: 2025-03-14T06:24:22.2452734Z Successfully uninstalled tokenizers-0.19.1 2025-03-14T06:24:22.3196919Z Attempting uninstall: transformers 2025-03-14T06:24:22.3221752Z Found existing installation: transformers 4.44.2 2025-03-14T06:24:22.5318722Z Uninstalling transformers-4.44.2: 2025-03-14T06:24:22.5642469Z Successfully uninstalled transformers-4.44.2 2025-03-14T06:24:26.6910397Z Successfully installed tokenizers-0.15.2 transformers-4.38.1 2025-03-14T06:24:26.8400039Z + echo 'Print all dependencies after TorchBench is installed' 2025-03-14T06:24:26.8400771Z Print all dependencies after TorchBench is installed 2025-03-14T06:24:26.8401372Z + python -mpip freeze 2025-03-14T06:24:27.2829244Z absl-py==2.1.0 2025-03-14T06:24:27.2830455Z accelerate==1.5.1 2025-03-14T06:24:27.2830948Z aiohappyeyeballs==2.6.1 2025-03-14T06:24:27.2832324Z aiohttp==3.11.13 2025-03-14T06:24:27.2833187Z aiosignal==1.3.2 2025-03-14T06:24:27.2833705Z annotated-types==0.7.0 2025-03-14T06:24:27.2834445Z antlr4-python3-runtime==4.9.3 2025-03-14T06:24:27.2834971Z anyascii==0.3.2 2025-03-14T06:24:27.2835629Z astroid==3.3.9 2025-03-14T06:24:27.2836089Z astunparse==1.6.3 2025-03-14T06:24:27.2836650Z async-timeout==5.0.1 2025-03-14T06:24:27.2837087Z attrs==23.1.0 2025-03-14T06:24:27.2837552Z audioread==3.0.1 2025-03-14T06:24:27.2838081Z babel==2.17.0 2025-03-14T06:24:27.2838725Z beautifulsoup4==4.13.3 2025-03-14T06:24:27.2840292Z -e git+https://github.com/pytorch/benchmark@373ffb19dc470f4423a3176a4133f8f4b3cdb5bd#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2025-03-14T06:24:27.2841447Z black==25.1.0 2025-03-14T06:24:27.2841844Z blinker==1.9.0 2025-03-14T06:24:27.2842412Z blis==1.2.0 2025-03-14T06:24:27.2842845Z blobfile==3.0.0 2025-03-14T06:24:27.2843430Z bokeh==3.7.0 2025-03-14T06:24:27.2843884Z boto3==1.35.42 2025-03-14T06:24:27.2844326Z botocore==1.35.99 2025-03-14T06:24:27.2844845Z bs4==0.0.2 2025-03-14T06:24:27.2845245Z cachetools==5.5.2 2025-03-14T06:24:27.2845599Z cardboardlint==1.3.1 2025-03-14T06:24:27.2846114Z catalogue==2.0.10 2025-03-14T06:24:27.2846477Z certifi==2025.1.31 2025-03-14T06:24:27.2846935Z cffi==1.17.1 2025-03-14T06:24:27.2847311Z charset-normalizer==3.4.1 2025-03-14T06:24:27.2847739Z click==8.1.8 2025-03-14T06:24:27.2848129Z cloudpathlib==0.21.0 2025-03-14T06:24:27.2848575Z cloudpickle==3.1.1 2025-03-14T06:24:27.2848950Z colorama==0.4.6 2025-03-14T06:24:27.2849503Z confection==0.1.5 2025-03-14T06:24:27.2849870Z contourpy==1.2.1 2025-03-14T06:24:27.2850262Z coremltools==5.0b5 2025-03-14T06:24:27.2850768Z cryptography==44.0.2 2025-03-14T06:24:27.2851213Z csvw==3.5.1 2025-03-14T06:24:27.2851576Z cycler==0.12.1 2025-03-14T06:24:27.2851997Z cymem==2.0.11 2025-03-14T06:24:27.2852351Z Cython==3.0.12 2025-03-14T06:24:27.2852754Z DALL-E==0.1 2025-03-14T06:24:27.2853167Z dataclasses-json==0.6.7 2025-03-14T06:24:27.2853515Z datasets==3.3.2 2025-03-14T06:24:27.2854119Z decorator==5.2.1 2025-03-14T06:24:27.2854437Z defusedxml==0.7.1 2025-03-14T06:24:27.2854843Z Deprecated==1.2.18 2025-03-14T06:24:27.2855659Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@0df2d73d0013db7de629602c23cc120219b4f2b8 2025-03-14T06:24:27.2856418Z diffusers==0.30.3 2025-03-14T06:24:27.2856803Z dill==0.3.8 2025-03-14T06:24:27.2857226Z diskcache==5.6.3 2025-03-14T06:24:27.2857588Z distro==1.9.0 2025-03-14T06:24:27.2858249Z dlinfo==2.0.0 2025-03-14T06:24:27.2858601Z docker-pycreds==0.4.0 2025-03-14T06:24:27.2859004Z dominate==2.9.1 2025-03-14T06:24:27.2859908Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2025-03-14T06:24:27.2860628Z einops==0.8.1 2025-03-14T06:24:27.2860964Z exceptiongroup==1.2.2 2025-03-14T06:24:27.2861230Z execnet==2.1.1 2025-03-14T06:24:27.2872025Z expecttest==0.3.0 2025-03-14T06:24:27.2872364Z FastNLP==0.6.0 2025-03-14T06:24:27.2872608Z fbscribelogger==0.1.7 2025-03-14T06:24:27.2872872Z ffmpeg-python==0.2.0 2025-03-14T06:24:27.2873129Z filelock==3.16.1 2025-03-14T06:24:27.2873356Z Flask==3.1.0 2025-03-14T06:24:27.2873584Z flatbuffers==2.0 2025-03-14T06:24:27.2873895Z fonttools==4.56.0 2025-03-14T06:24:27.2874130Z frozenlist==1.5.0 2025-03-14T06:24:27.2874357Z fsspec==2024.10.0 2025-03-14T06:24:27.2874581Z ftfy==6.3.1 2025-03-14T06:24:27.2875037Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2025-03-14T06:24:27.2875553Z future==1.0.0 2025-03-14T06:24:27.2875789Z fvcore==0.1.5.post20221221 2025-03-14T06:24:27.2876128Z gdown==5.2.0 2025-03-14T06:24:27.2876356Z ghstack==0.8.0 2025-03-14T06:24:27.2876586Z gitdb==4.0.12 2025-03-14T06:24:27.2876810Z GitPython==3.1.44 2025-03-14T06:24:27.2877047Z google-auth==2.38.0 2025-03-14T06:24:27.2877304Z google-auth-oauthlib==1.0.0 2025-03-14T06:24:27.2877578Z grpcio==1.71.0 2025-03-14T06:24:27.2877805Z gym==0.26.2 2025-03-14T06:24:27.2878034Z gym-notices==0.0.8 2025-03-14T06:24:27.2878274Z h5py==3.13.0 2025-03-14T06:24:27.2878640Z higher==0.2.1 2025-03-14T06:24:27.2878883Z huggingface-hub==0.29.3 2025-03-14T06:24:27.2879145Z hydra-core==1.3.2 2025-03-14T06:24:27.2879393Z hypothesis==5.35.1 2025-03-14T06:24:27.2879632Z idna==3.10 2025-03-14T06:24:27.2879904Z imageio==2.37.0 2025-03-14T06:24:27.2880196Z importlib_metadata==8.6.1 2025-03-14T06:24:27.2880467Z inflect==7.5.0 2025-03-14T06:24:27.2880702Z iniconfig==2.0.0 2025-03-14T06:24:27.2880940Z iopath==0.1.9 2025-03-14T06:24:27.2881169Z isodate==0.7.2 2025-03-14T06:24:27.2881398Z isort==6.0.1 2025-03-14T06:24:27.2881680Z itsdangerous==2.2.0 2025-03-14T06:24:27.2882034Z Jinja2==3.1.6 2025-03-14T06:24:27.2882278Z jmespath==1.0.1 2025-03-14T06:24:27.2882518Z joblib==1.4.2 2025-03-14T06:24:27.2882749Z jsonpatch==1.33 2025-03-14T06:24:27.2882990Z jsonpointer==3.0.0 2025-03-14T06:24:27.2883241Z jsonschema==4.23.0 2025-03-14T06:24:27.2883512Z jsonschema-specifications==2024.10.1 2025-03-14T06:24:27.2883836Z junitparser==2.1.1 2025-03-14T06:24:27.2884079Z kaldi-io==0.9.8 2025-03-14T06:24:27.2884328Z kiwisolver==1.4.8 2025-03-14T06:24:27.2884570Z kornia==0.8.0 2025-03-14T06:24:27.2884797Z kornia_rs==0.1.8 2025-03-14T06:24:27.2885037Z lameenc==1.8.1 2025-03-14T06:24:27.2885264Z langcodes==3.5.0 2025-03-14T06:24:27.2885502Z langdetect==1.0.9 2025-03-14T06:24:27.2885748Z language-tags==1.2.0 2025-03-14T06:24:27.2886005Z language_data==1.3.0 2025-03-14T06:24:27.2886253Z lark==0.12.0 2025-03-14T06:24:27.2886480Z lazy_loader==0.4 2025-03-14T06:24:27.2886723Z libcst==1.7.0 2025-03-14T06:24:27.2887009Z librosa==0.9.2 2025-03-14T06:24:27.2887279Z lintrunner==0.12.7 2025-03-14T06:24:27.2887525Z llvmlite==0.38.1 2025-03-14T06:24:27.2887759Z lxml==5.3.0 2025-03-14T06:24:27.2887984Z marisa-trie==1.2.1 2025-03-14T06:24:27.2888221Z Markdown==3.7 2025-03-14T06:24:27.2888463Z markdown-it-py==3.0.0 2025-03-14T06:24:27.2888723Z MarkupSafe==3.0.2 2025-03-14T06:24:27.2888968Z marshmallow==3.26.1 2025-03-14T06:24:27.2889215Z matplotlib==3.8.4 2025-03-14T06:24:27.2889453Z mccabe==0.7.0 2025-03-14T06:24:27.2889687Z mdurl==0.1.2 2025-03-14T06:24:27.2889913Z ml_dtypes==0.5.1 2025-03-14T06:24:27.2890161Z MonkeyType==23.3.0 2025-03-14T06:24:27.2890433Z more-itertools==10.6.0 2025-03-14T06:24:27.2890699Z mpmath==1.3.0 2025-03-14T06:24:27.2890939Z msgpack==1.1.0 2025-03-14T06:24:27.2891179Z multidict==6.1.0 2025-03-14T06:24:27.2891431Z multiprocess==0.70.16 2025-03-14T06:24:27.2891686Z murmurhash==1.0.12 2025-03-14T06:24:27.2891934Z musdb==0.4.2 2025-03-14T06:24:27.2892168Z museval==0.4.1 2025-03-14T06:24:27.2892404Z mypy==1.14.0 2025-03-14T06:24:27.2892646Z mypy-extensions==1.0.0 2025-03-14T06:24:27.2892915Z narwhals==1.30.0 2025-03-14T06:24:27.2893163Z networkx==2.8.8 2025-03-14T06:24:27.2893403Z ninja==1.10.2 2025-03-14T06:24:27.2893635Z nose==1.3.7 2025-03-14T06:24:27.2893858Z numba==0.55.2 2025-03-14T06:24:27.2894089Z numpy==1.22.4 2025-03-14T06:24:27.2894338Z nvidia-cublas-cu12==12.4.5.8 2025-03-14T06:24:27.2894651Z nvidia-cuda-cupti-cu12==12.4.127 2025-03-14T06:24:27.2894970Z nvidia-cuda-nvrtc-cu12==12.4.127 2025-03-14T06:24:27.2895289Z nvidia-cuda-runtime-cu12==12.4.127 2025-03-14T06:24:27.2895608Z nvidia-cudnn-cu12==9.1.0.70 2025-03-14T06:24:27.2895901Z nvidia-cufft-cu12==11.2.1.3 2025-03-14T06:24:27.2896194Z nvidia-curand-cu12==10.3.5.147 2025-03-14T06:24:27.2896497Z nvidia-cusolver-cu12==11.6.1.9 2025-03-14T06:24:27.2896803Z nvidia-cusparse-cu12==12.3.1.170 2025-03-14T06:24:27.2897118Z nvidia-cusparselt-cu12==0.6.2 2025-03-14T06:24:27.2897418Z nvidia-ml-py==12.570.86 2025-03-14T06:24:27.2897692Z nvidia-nccl-cu12==2.21.5 2025-03-14T06:24:27.2897983Z nvidia-nvjitlink-cu12==12.4.127 2025-03-14T06:24:27.2898296Z nvidia-nvtx-cu12==12.4.127 2025-03-14T06:24:27.2898576Z oauthlib==3.2.2 2025-03-14T06:24:27.2898820Z omegaconf==2.3.0 2025-03-14T06:24:27.2899058Z onnx==1.17.0 2025-03-14T06:24:27.2899288Z onnxscript==0.2.2 2025-03-14T06:24:27.2899537Z opacus==1.5.3 2025-03-14T06:24:27.2899996Z opencv-python==4.8.0.74 2025-03-14T06:24:27.2900278Z opt-einsum==3.3.0 2025-03-14T06:24:27.2900609Z optree==0.13.0 2025-03-14T06:24:27.2900971Z packaging==24.2 2025-03-14T06:24:27.2901220Z pandas==2.0.3 2025-03-14T06:24:27.2901473Z parameterized==0.8.1 2025-03-14T06:24:27.2901734Z patch==1.16 2025-03-14T06:24:27.2901972Z pathspec==0.12.1 2025-03-14T06:24:27.2902226Z phonemizer==3.3.0 2025-03-14T06:24:27.2902476Z pillow==11.0.0 2025-03-14T06:24:27.2902724Z platformdirs==4.3.6 2025-03-14T06:24:27.2902985Z pluggy==1.5.0 2025-03-14T06:24:27.2903217Z ply==3.11 2025-03-14T06:24:27.2903442Z pooch==1.8.2 2025-03-14T06:24:27.2903684Z portalocker==3.1.1 2025-03-14T06:24:27.2903942Z preshed==3.0.9 2025-03-14T06:24:27.2904187Z prettytable==3.15.1 2025-03-14T06:24:27.2904527Z propcache==0.3.0 2025-03-14T06:24:27.2904792Z protobuf==3.20.2 2025-03-14T06:24:27.2905036Z psutil==7.0.0 2025-03-14T06:24:27.2905267Z PuLP==2.9.0 2025-03-14T06:24:27.2905492Z pwlf==2.2.1 2025-03-14T06:24:27.2905723Z py-cpuinfo==9.0.0 2025-03-14T06:24:27.2905969Z pyaml==25.1.0 2025-03-14T06:24:27.2906202Z pyarrow==19.0.1 2025-03-14T06:24:27.2906431Z pyasn1==0.6.1 2025-03-14T06:24:27.2906734Z pyasn1_modules==0.4.1 2025-03-14T06:24:27.2907001Z pyclipper==1.3.0.post6 2025-03-14T06:24:27.2907273Z pycocotools==2.0.8 2025-03-14T06:24:27.2907524Z pycparser==2.22 2025-03-14T06:24:27.2907774Z pycryptodomex==3.21.0 2025-03-14T06:24:27.2908033Z pydantic==2.10.6 2025-03-14T06:24:27.2908278Z pydantic_core==2.27.2 2025-03-14T06:24:27.2908534Z pyDOE==0.3.8 2025-03-14T06:24:27.2908761Z pydot==3.0.4 2025-03-14T06:24:27.2908991Z pygame==2.6.1 2025-03-14T06:24:27.2909223Z PyGithub==2.3.0 2025-03-14T06:24:27.2909463Z Pygments==2.15.0 2025-03-14T06:24:27.2909695Z PyJWT==2.10.1 2025-03-14T06:24:27.2909934Z pylint==3.3.5 2025-03-14T06:24:27.2910164Z PyNaCl==1.5.0 2025-03-14T06:24:27.2910395Z pynvml==12.0.0 2025-03-14T06:24:27.2910643Z pyparsing==3.2.1 2025-03-14T06:24:27.2910930Z pypdfium2==4.30.1 2025-03-14T06:24:27.2911169Z pysbd==0.3.4 2025-03-14T06:24:27.2911398Z PySocks==1.7.1 2025-03-14T06:24:27.2911632Z pytest==8.3.5 2025-03-14T06:24:27.2911877Z pytest-benchmark==5.1.0 2025-03-14T06:24:27.2912155Z pytest-cpp==2.3.0 2025-03-14T06:24:27.2912428Z pytest-flakefinder==1.1.0 2025-03-14T06:24:27.2912724Z pytest-rerunfailures==14.0 2025-03-14T06:24:27.2913010Z pytest-subtests==0.13.1 2025-03-14T06:24:27.2913282Z pytest-xdist==3.3.1 2025-03-14T06:24:27.2913556Z python-dateutil==2.9.0.post0 2025-03-14T06:24:27.2913849Z python-doctr==0.11.0 2025-03-14T06:24:27.2914548Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@e6aadeb86f3ae1f58c3f98e2a91e251716e0f2aa 2025-03-14T06:24:27.2915279Z pytz==2025.1 2025-03-14T06:24:27.2915516Z PyWavelets==1.4.1 2025-03-14T06:24:27.2915765Z PyYAML==6.0.2 2025-03-14T06:24:27.2916002Z pyzstd==0.16.2 2025-03-14T06:24:27.2916248Z RapidFuzz==3.12.2 2025-03-14T06:24:27.2916477Z rdflib==7.1.3 2025-03-14T06:24:27.2916710Z redis==5.2.1 2025-03-14T06:24:27.2916949Z referencing==0.36.2 2025-03-14T06:24:27.2917198Z regex==2024.11.6 2025-03-14T06:24:27.2917437Z requests==2.32.3 2025-03-14T06:24:27.2917692Z requests-oauthlib==2.0.0 2025-03-14T06:24:27.2917975Z resampy==0.4.3 2025-03-14T06:24:27.2918211Z rfc3986==1.5.0 2025-03-14T06:24:27.2918439Z rich==13.9.4 2025-03-14T06:24:27.2918672Z rpds-py==0.23.1 2025-03-14T06:24:27.2918915Z rsa==4.9 2025-03-14T06:24:27.2919138Z s3transfer==0.10.4 2025-03-14T06:24:27.2919386Z safetensors==0.5.3 2025-03-14T06:24:27.2919630Z scikit-image==0.22.0 2025-03-14T06:24:27.2919893Z scikit-learn==1.6.1 2025-03-14T06:24:27.2920145Z scipy==1.10.1 2025-03-14T06:24:27.2920378Z segments==2.3.0 2025-03-14T06:24:27.2920648Z sentencepiece==0.2.0 2025-03-14T06:24:27.2920929Z sentry-sdk==2.22.0 2025-03-14T06:24:27.2921183Z setproctitle==1.3.5 2025-03-14T06:24:27.2921428Z shapely==2.0.7 2025-03-14T06:24:27.2921674Z shellingham==1.5.4 2025-03-14T06:24:27.2921922Z simplejson==3.20.1 2025-03-14T06:24:27.2922160Z six==1.17.0 2025-03-14T06:24:27.2922392Z smart-open==7.1.0 2025-03-14T06:24:27.2922635Z smmap==5.0.2 2025-03-14T06:24:27.2922870Z sortedcontainers==2.4.0 2025-03-14T06:24:27.2923134Z soundfile==0.13.1 2025-03-14T06:24:27.2923483Z soupsieve==2.6 2025-03-14T06:24:27.2923721Z soxr==0.5.0.post1 2025-03-14T06:24:27.2923957Z spacy==3.8.4 2025-03-14T06:24:27.2924194Z spacy-legacy==3.0.12 2025-03-14T06:24:27.2924454Z spacy-loggers==1.0.5 2025-03-14T06:24:27.2924707Z srsly==2.5.1 2025-03-14T06:24:27.2924935Z stempeg==0.2.3 2025-03-14T06:24:27.2925174Z submitit==1.5.2 2025-03-14T06:24:27.2925416Z sympy==1.13.3 2025-03-14T06:24:27.2925649Z tabulate==0.9.0 2025-03-14T06:24:27.2925894Z tb-nightly==2.13.0a20230426 2025-03-14T06:24:27.2926514Z tensorboard==2.13.0 2025-03-14T06:24:27.2926823Z tensorboard-data-server==0.7.2 2025-03-14T06:24:27.2927295Z tensorboardX==2.6.2.2 2025-03-14T06:24:27.2927566Z termcolor==2.5.0 2025-03-14T06:24:27.2927812Z thinc==8.3.4 2025-03-14T06:24:27.2928055Z threadpoolctl==3.6.0 2025-03-14T06:24:27.2928313Z thriftpy2==0.5.2 2025-03-14T06:24:27.2928561Z tifffile==2025.2.18 2025-03-14T06:24:27.2929091Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2025-03-14T06:24:27.2929674Z tlparse==0.3.30 2025-03-14T06:24:27.2929916Z tokenizers==0.15.2 2025-03-14T06:24:27.2930160Z tomli==2.2.1 2025-03-14T06:24:27.2930398Z tomlkit==0.13.2 2025-03-14T06:24:27.2931225Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.8.0a0%2Bgitaed0b7a-cp310-cp310-linux_x86_64.whl#sha256=02752e54cf563c69dc1c59b75a999c1c0b4f2b10ebfff89223d069c58f048fa3 2025-03-14T06:24:27.2932340Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2025-03-14T06:24:27.2933113Z torchao @ git+https://github.com/pytorch/ao.git@9259584f98db0760b27492a63050a2915c753dbe 2025-03-14T06:24:27.2933814Z torchaudio @ git+https://github.com/pytorch/audio.git@c670ad81fda266b6598aeeef434583eb98197ae8 2025-03-14T06:24:27.2934640Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@6569fcc03450c2360b50d772bf9b18ec3487fcf4 2025-03-14T06:24:27.2935460Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-03-14T06:24:27.2935980Z tornado==6.4.2 2025-03-14T06:24:27.2936216Z tqdm==4.67.1 2025-03-14T06:24:27.2936444Z transformers==4.38.1 2025-03-14T06:24:27.2936701Z treetable==0.2.5 2025-03-14T06:24:27.2936986Z triton @ file:///var/lib/jenkins/triton/python 2025-03-14T06:24:27.2937324Z typeguard==4.4.2 2025-03-14T06:24:27.2937559Z typer==0.15.2 2025-03-14T06:24:27.2937799Z typing-inspect==0.9.0 2025-03-14T06:24:27.2938071Z typing_extensions==4.12.2 2025-03-14T06:24:27.2938341Z tzdata==2025.1 2025-03-14T06:24:27.2938576Z Unidecode==1.3.8 2025-03-14T06:24:27.2938839Z unittest-xml-reporting==3.2.0 2025-03-14T06:24:27.2939144Z uritemplate==4.1.1 2025-03-14T06:24:27.2939391Z urllib3==2.3.0 2025-03-14T06:24:27.2939625Z visdom==0.2.4 2025-03-14T06:24:27.2939856Z wandb==0.19.8 2025-03-14T06:24:27.2940086Z wasabi==1.1.3 2025-03-14T06:24:27.2940314Z wcwidth==0.2.13 2025-03-14T06:24:27.2940547Z weasel==0.4.1 2025-03-14T06:24:27.2940787Z websocket-client==1.8.0 2025-03-14T06:24:27.2941051Z Werkzeug==3.1.3 2025-03-14T06:24:27.2941295Z wrapt==1.17.2 2025-03-14T06:24:27.2941532Z xdoctest==1.1.0 2025-03-14T06:24:27.2941765Z xxhash==3.5.0 2025-03-14T06:24:27.2942001Z xyzservices==2025.1.0 2025-03-14T06:24:27.2942254Z yacs==0.1.8 2025-03-14T06:24:27.2942473Z yarl==1.18.3 2025-03-14T06:24:27.2942706Z z3-solver==4.12.6.0 2025-03-14T06:24:27.2942948Z zipp==3.21.0 2025-03-14T06:24:27.3133332Z + popd 2025-03-14T06:24:27.3133999Z ~/workspace 2025-03-14T06:24:27.3134343Z + [[ inductor_torchbench != *cpu* ]] 2025-03-14T06:24:27.3134691Z + install_torchrec_and_fbgemm 2025-03-14T06:24:27.3143435Z + local torchrec_commit 2025-03-14T06:24:27.3148476Z ++ get_pinned_commit torchrec 2025-03-14T06:24:27.3154285Z ++ cat .github/ci_commit_pins/torchrec.txt 2025-03-14T06:24:27.3186740Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T06:24:27.3187155Z + local fbgemm_commit 2025-03-14T06:24:27.3190591Z ++ get_pinned_commit fbgemm 2025-03-14T06:24:27.3216038Z ++ cat .github/ci_commit_pins/fbgemm.txt 2025-03-14T06:24:27.3216679Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2025-03-14T06:24:27.3217119Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-03-14T06:24:27.3217504Z + pip_uninstall torchrec-nightly 2025-03-14T06:24:27.3217837Z + pip3 uninstall -y torchrec-nightly 2025-03-14T06:24:27.7559202Z WARNING: Skipping torchrec-nightly as it is not installed. 2025-03-14T06:24:27.7930739Z + pip_uninstall fbgemm-gpu-nightly 2025-03-14T06:24:27.7931168Z + pip3 uninstall -y fbgemm-gpu-nightly 2025-03-14T06:24:28.2263054Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2025-03-14T06:24:28.2618233Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2025-03-14T06:24:28.2618843Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T06:24:28.2619471Z + python3 -m pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2025-03-14T06:24:28.7641190Z Collecting setuptools-git-versioning 2025-03-14T06:24:28.7986975Z Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl.metadata (6.1 kB) 2025-03-14T06:24:28.8211226Z Collecting scikit-build 2025-03-14T06:24:28.8242880Z Downloading scikit_build-0.18.1-py3-none-any.whl.metadata (18 kB) 2025-03-14T06:24:28.8582817Z Collecting pyre-extensions 2025-03-14T06:24:28.8615784Z Downloading pyre_extensions-0.0.32-py3-none-any.whl.metadata (4.0 kB) 2025-03-14T06:24:28.8685869Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.2) 2025-03-14T06:24:28.8688561Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (75.8.0) 2025-03-14T06:24:28.8692864Z Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (2.2.1) 2025-03-14T06:24:28.8703389Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2025-03-14T06:24:28.8708740Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.45.1) 2025-03-14T06:24:28.8715099Z Requirement already satisfied: typing-inspect in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (0.9.0) 2025-03-14T06:24:28.8718065Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2025-03-14T06:24:28.8805207Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2025-03-14T06:24:28.8845921Z Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl (10 kB) 2025-03-14T06:24:28.8936950Z Downloading scikit_build-0.18.1-py3-none-any.whl (85 kB) 2025-03-14T06:24:28.9112443Z Downloading pyre_extensions-0.0.32-py3-none-any.whl (12 kB) 2025-03-14T06:24:29.7864656Z Installing collected packages: setuptools-git-versioning, scikit-build, pyre-extensions 2025-03-14T06:24:29.8969599Z Successfully installed pyre-extensions-0.0.32 scikit-build-0.18.1 setuptools-git-versioning-2.1.0 2025-03-14T06:24:30.0230031Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-03-14T06:24:30.0231020Z + CUDA_PATH=/usr/local/cuda-12.1 2025-03-14T06:24:30.0231861Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2025-03-14T06:24:30.0232755Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T06:24:30.0233724Z + python3 -m pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2025-03-14T06:24:30.4668272Z Collecting fbgemm-gpu 2025-03-14T06:24:30.4670305Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-jdmz58br/fbgemm-gpu_e9a454f396c74d13b9624f181ce2b01d 2025-03-14T06:24:30.4701375Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-jdmz58br/fbgemm-gpu_e9a454f396c74d13b9624f181ce2b01d 2025-03-14T06:24:31.5849130Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2025-03-14T06:24:31.5880395Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2025-03-14T06:24:32.3124758Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2025-03-14T06:24:32.8180389Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2025-03-14T06:24:32.8181059Z Running command git submodule update --init --recursive -q 2025-03-14T06:24:41.4178445Z Preparing metadata (setup.py) ... [?25l- done 2025-03-14T06:24:41.4197367Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.22.4) 2025-03-14T06:24:41.4204504Z Building wheels for collected packages: fbgemm-gpu 2025-03-14T07:09:18.6647008Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2025-03-14T07:09:18.8494935Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253784739 sha256=1882d00531dc73346231b8c289a37cb0e8311677ccb5ffc6f018c7a72940cfda 2025-03-14T07:09:18.8497417Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2025-03-14T07:09:18.8516366Z Successfully built fbgemm-gpu 2025-03-14T07:09:19.7265760Z Installing collected packages: fbgemm-gpu 2025-03-14T07:09:22.9809184Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2025-03-14T07:09:23.9665332Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:23.9666074Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-03-14T07:09:23.9667006Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:24.3973825Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:24.3976615Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-9jmlac3b 2025-03-14T07:09:24.4006789Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-9jmlac3b 2025-03-14T07:09:25.1622626Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2025-03-14T07:09:25.1652532Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:25.5590925Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:25.9398399Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-03-14T07:09:26.2133872Z Preparing metadata (setup.py) ... [?25l- done 2025-03-14T07:09:26.2169576Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2025-03-14T07:09:26.2173013Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2025-03-14T07:09:26.2175738Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2025-03-14T07:09:26.2759887Z Collecting torchmetrics (from torchrec==0.3.2) 2025-03-14T07:09:26.3137761Z Downloading torchmetrics-1.6.3-py3-none-any.whl.metadata (20 kB) 2025-03-14T07:09:26.3562786Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.67.1) 2025-03-14T07:09:26.3565657Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2025-03-14T07:09:26.3575627Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.22.4) 2025-03-14T07:09:26.3584822Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (3.1.1) 2025-03-14T07:09:26.3862898Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2025-03-14T07:09:26.3866091Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.1) 2025-03-14T07:09:26.3870002Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.1) 2025-03-14T07:09:26.3893546Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.2) 2025-03-14T07:09:26.3898377Z Requirement already satisfied: torch>=2.0.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.8.0a0+gitaed0b7a) 2025-03-14T07:09:26.4021760Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2025-03-14T07:09:26.4051987Z Downloading lightning_utilities-0.14.0-py3-none-any.whl.metadata (5.6 kB) 2025-03-14T07:09:26.4143601Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (75.8.0) 2025-03-14T07:09:26.4147152Z Requirement already satisfied: typing_extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2025-03-14T07:09:26.4169349Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.17.0) 2025-03-14T07:09:26.4200475Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.16.1) 2025-03-14T07:09:26.4204976Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.13.3) 2025-03-14T07:09:26.4208087Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2025-03-14T07:09:26.4211276Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.1.6) 2025-03-14T07:09:26.4214440Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2024.10.0) 2025-03-14T07:09:26.4265325Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2025-03-14T07:09:26.4651696Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.0.2) 2025-03-14T07:09:26.4834671Z Downloading torchmetrics-1.6.3-py3-none-any.whl (931 kB) 2025-03-14T07:09:26.5085953Z Downloading lightning_utilities-0.14.0-py3-none-any.whl (28 kB) 2025-03-14T07:09:26.5179909Z Building wheels for collected packages: torchrec 2025-03-14T07:09:27.0827992Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2025-03-14T07:09:27.0837923Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374547 sha256=596f9246827441385d7ac4325f33fd133834567f5296af43cd3aa5513a8fa499 2025-03-14T07:09:27.0839583Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2025-03-14T07:09:27.0863147Z Successfully built torchrec 2025-03-14T07:09:27.9099720Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2025-03-14T07:09:28.5493759Z Successfully installed lightning-utilities-0.14.0 torchmetrics-1.6.3 torchrec-0.3.2 2025-03-14T07:09:28.7117251Z ++ pwd 2025-03-14T07:09:28.7119288Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2025-03-14T07:09:28.7119693Z + test_dynamo_benchmark torchbench 0 2025-03-14T07:09:28.7141094Z ++ pwd 2025-03-14T07:09:28.7144665Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-03-14T07:09:28.7145231Z + local suite=torchbench 2025-03-14T07:09:28.7145500Z + shift 2025-03-14T07:09:28.7145723Z + local shard_id=0 2025-03-14T07:09:28.7146037Z + shift 2025-03-14T07:09:28.7146303Z + [[ inductor_torchbench == *perf_compare* ]] 2025-03-14T07:09:28.7146791Z + [[ inductor_torchbench == *perf* ]] 2025-03-14T07:09:28.7147172Z + [[ inductor_torchbench == *cpu* ]] 2025-03-14T07:09:28.7147502Z + [[ inductor_torchbench == *aot_inductor* ]] 2025-03-14T07:09:28.7147979Z + [[ inductor_torchbench == *max_autotune_inductor* ]] 2025-03-14T07:09:28.7148505Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2025-03-14T07:09:28.7195237Z ++ pwd 2025-03-14T07:09:28.7198024Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-03-14T07:09:28.7198502Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-03-14T07:09:28.7251395Z + local name=inference 2025-03-14T07:09:28.7251672Z + shift 2025-03-14T07:09:28.7251903Z + local suite=torchbench 2025-03-14T07:09:28.7252161Z + shift 2025-03-14T07:09:28.7252376Z + local shard_id=0 2025-03-14T07:09:28.7252606Z + shift 2025-03-14T07:09:28.7252831Z + partition_flags=() 2025-03-14T07:09:28.7253097Z + local partition_flags 2025-03-14T07:09:28.7253358Z + [[ -n 2 ]] 2025-03-14T07:09:28.7253581Z + [[ -n 0 ]] 2025-03-14T07:09:28.7253985Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-03-14T07:09:28.7254506Z + [[ inductor_torchbench == *perf_compare* ]] 2025-03-14T07:09:28.7254858Z + [[ inductor_torchbench == *perf* ]] 2025-03-14T07:09:28.7255186Z + [[ inductor_torchbench == *_avx2* ]] 2025-03-14T07:09:28.7255514Z + [[ inductor_torchbench == *_avx512* ]] 2025-03-14T07:09:28.7256678Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2025-03-14T07:09:36.0280723Z 2025-03-14T07:09:37.3249089Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:09:37.3249455Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:09:37.3249797Z cuda eval torchrec_dlrm 2025-03-14T07:09:37.3280459Z Traceback (most recent call last): 2025-03-14T07:09:37.3281015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1913, in validate_model 2025-03-14T07:09:37.3281596Z self.model_iter_fn(model, example_inputs) 2025-03-14T07:09:37.3282534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 459, in forward_pass 2025-03-14T07:09:37.3283069Z return mod(*inputs) 2025-03-14T07:09:37.3283657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:09:37.3284292Z return self._call_impl(*args, **kwargs) 2025-03-14T07:09:37.3284891Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:09:37.3285496Z return forward_call(*args, **kwargs) 2025-03-14T07:09:37.3286060Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2025-03-14T07:09:37.3286719Z logits = self.model(batch.dense_features, batch.sparse_features) 2025-03-14T07:09:37.3287429Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:09:37.3288072Z return self._call_impl(*args, **kwargs) 2025-03-14T07:09:37.3288660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:09:37.3289249Z return forward_call(*args, **kwargs) 2025-03-14T07:09:37.3289812Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2025-03-14T07:09:37.3290386Z concatenated_dense = self.inter_arch( 2025-03-14T07:09:37.3291018Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:09:37.3291644Z return self._call_impl(*args, **kwargs) 2025-03-14T07:09:37.3292230Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:09:37.3292817Z return forward_call(*args, **kwargs) 2025-03-14T07:09:37.3293383Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2025-03-14T07:09:37.3294033Z return self.crossnet(combined_values.reshape([B, -1])) 2025-03-14T07:09:37.3294717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:09:37.3295346Z return self._call_impl(*args, **kwargs) 2025-03-14T07:09:37.3295934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:09:37.3296523Z return forward_call(*args, **kwargs) 2025-03-14T07:09:37.3297110Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2025-03-14T07:09:37.3297777Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2025-03-14T07:09:37.3298383Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2025-03-14T07:09:37.3298764Z 2025-03-14T07:09:37.3298985Z The above exception was the direct cause of the following exception: 2025-03-14T07:09:37.3299299Z 2025-03-14T07:09:37.3299423Z Traceback (most recent call last): 2025-03-14T07:09:37.3299895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3991, in run 2025-03-14T07:09:37.3300376Z ) = runner.load_model( 2025-03-14T07:09:37.3300862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2025-03-14T07:09:37.3301412Z self.validate_model(model, example_inputs) 2025-03-14T07:09:37.3302120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1915, in validate_model 2025-03-14T07:09:37.3302681Z raise RuntimeError("Eager run failed") from e 2025-03-14T07:09:37.3303034Z RuntimeError: Eager run failed 2025-03-14T07:09:37.3303224Z 2025-03-14T07:09:37.3303328Z eager_fail_to_run 2025-03-14T07:09:37.8549283Z [rank0]:[W314 07:09:37.831770834 ProcessGroupNCCL.cpp:1497] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-03-14T07:09:41.6337285Z 2025-03-14T07:09:43.9299100Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:09:43.9299473Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:09:43.9360941Z cuda eval BERT_pytorch 2025-03-14T07:10:13.3355237Z Compilation time (from dynamo_timed): 26.812997044 2025-03-14T07:10:13.3357495Z pass 2025-03-14T07:10:13.3397231Z TIMING: _recursive_pre_grad_passes:0.01556 pad_mm_benchmark:0.46962 _recursive_joint_graph_passes:1.03021 _recursive_post_grad_passes:0.67677 async_compile.wait:1.94717 code_gen:4.18624 inductor_compile:7.15297 backend_compile:23.80437 entire_frame_compile:26.813 gc:0.00044 cudagraphify.get_container:0.25353 CachingAutotuner.benchmark_all_configs:0.02984 CUDAGraphNode.record:0.2519 total_wall_time:26.813 2025-03-14T07:10:13.3399211Z STATS: call_* op count: 543 | FakeTensor.__torch_dispatch__:2100 | FakeTensorMode.__torch_dispatch__:17631 | ProxyTorchDispatchMode.__torch_dispatch__:7466 2025-03-14T07:10:13.3400032Z Dynamo produced 1 graphs covering 543 ops with 0 graph breaks (0 unique) 2025-03-14T07:10:18.9120462Z 2025-03-14T07:10:21.7406177Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:10:21.7406849Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:10:21.7457989Z cuda eval Background_Matting 2025-03-14T07:10:21.7690111Z Compilation time (from dynamo_timed): 0 2025-03-14T07:10:21.7690482Z pass_due_to_skip 2025-03-14T07:10:21.8187641Z TIMING: total_wall_time:0 2025-03-14T07:10:21.8188123Z STATS: call_* op count: 0 2025-03-14T07:10:21.8188552Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:10:26.1144984Z 2025-03-14T07:10:29.3069766Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:10:29.3070124Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:10:29.3096542Z cuda eval LearningToPaint 2025-03-14T07:10:37.4001839Z Compilation time (from dynamo_timed): 5.805097813 2025-03-14T07:10:37.4002212Z pass 2025-03-14T07:10:37.4010521Z TIMING: _recursive_pre_grad_passes:0.0053 pad_mm_benchmark:0.06163 _recursive_joint_graph_passes:0.40908 _recursive_post_grad_passes:0.08005 async_compile.wait:0.54925 code_gen:1.44111 inductor_compile:2.38512 backend_compile:4.44954 entire_frame_compile:5.8051 gc:0.00067 cudagraphify.get_container:0.22479 CUDAGraphNode.record:0.21607 total_wall_time:5.8051 2025-03-14T07:10:37.4012419Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4575 | ProxyTorchDispatchMode.__torch_dispatch__:1728 | FakeTensor.__torch_dispatch__:933 2025-03-14T07:10:37.4013237Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique) 2025-03-14T07:10:42.9304690Z 2025-03-14T07:10:44.3905817Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2025-03-14T07:10:44.4240653Z 2025-03-14T07:10:44.4240986Z 2025-03-14T07:10:44.5245172Z 0% 0.00/528M [00:00 0).to(torch.long) * num_buckets 2025-03-14T07:25:53.0502082Z gt: "b8[2048, 2048][2048, 1]cuda:0" = relative_position > 0 2025-03-14T07:25:53.0502529Z to_3: "i64[2048, 2048][2048, 1]cuda:0" = gt.to(torch.int64); gt = None 2025-03-14T07:25:53.0502993Z mul_3: "i64[2048, 2048][2048, 1]cuda:0" = to_3 * 16; to_3 = None 2025-03-14T07:25:53.0503490Z relative_buckets: "i64[2048, 2048][2048, 1]cuda:0" = 0 + mul_3; mul_3 = None 2025-03-14T07:25:53.0503900Z 2025-03-14T07:25:53.0504689Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:415 in _relative_position_bucket, code: relative_position = torch.abs(relative_position) 2025-03-14T07:25:53.0505770Z relative_position_1: "i64[2048, 2048][2048, 1]cuda:0" = torch.abs(relative_position); relative_position = None 2025-03-14T07:25:53.0506280Z 2025-03-14T07:25:53.0507082Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:422 in _relative_position_bucket, code: is_small = relative_position < max_exact 2025-03-14T07:25:53.0508084Z is_small: "b8[2048, 2048][2048, 1]cuda:0" = relative_position_1 < 8 2025-03-14T07:25:53.0508461Z 2025-03-14T07:25:53.0509243Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:426 in _relative_position_bucket, code: torch.log(relative_position.float() / max_exact) 2025-03-14T07:25:53.0510263Z float_1: "f32[2048, 2048][2048, 1]cuda:0" = relative_position_1.float() 2025-03-14T07:25:53.0510757Z truediv: "f32[2048, 2048][2048, 1]cuda:0" = float_1 / 8; float_1 = None 2025-03-14T07:25:53.0511254Z log: "f32[2048, 2048][2048, 1]cuda:0" = torch.log(truediv); truediv = None 2025-03-14T07:25:53.0511774Z truediv_1: "f32[2048, 2048][2048, 1]cuda:0" = log / 2.772588722239781; log = None 2025-03-14T07:25:53.0512283Z mul_4: "f32[2048, 2048][2048, 1]cuda:0" = truediv_1 * 8; truediv_1 = None 2025-03-14T07:25:53.0512672Z 2025-03-14T07:25:53.0513344Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:429 in _relative_position_bucket, code: ).to(torch.long) 2025-03-14T07:25:53.0514179Z to_4: "i64[2048, 2048][2048, 1]cuda:0" = mul_4.to(torch.int64); mul_4 = None 2025-03-14T07:25:53.0514578Z 2025-03-14T07:25:53.0515338Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:425 in _relative_position_bucket, code: relative_position_if_large = max_exact + ( 2025-03-14T07:25:53.0516308Z relative_position_if_large: "i64[2048, 2048][2048, 1]cuda:0" = 8 + to_4; to_4 = None 2025-03-14T07:25:53.0516743Z 2025-03-14T07:25:53.0517663Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:431 in _relative_position_bucket, code: relative_position_if_large, torch.full_like(relative_position_if_large, num_buckets - 1) 2025-03-14T07:25:53.0518824Z full_like: "i64[2048, 2048][2048, 1]cuda:0" = torch.full_like(relative_position_if_large, 15) 2025-03-14T07:25:53.0519279Z 2025-03-14T07:25:53.0520030Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:430 in _relative_position_bucket, code: relative_position_if_large = torch.min( 2025-03-14T07:25:53.0521233Z relative_position_if_large_1: "i64[2048, 2048][2048, 1]cuda:0" = torch.min(relative_position_if_large, full_like); relative_position_if_large = full_like = None 2025-03-14T07:25:53.0521895Z 2025-03-14T07:25:53.0522801Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:434 in _relative_position_bucket, code: relative_buckets += torch.where(is_small, relative_position, relative_position_if_large) 2025-03-14T07:25:53.0524228Z where: "i64[2048, 2048][2048, 1]cuda:0" = torch.where(is_small, relative_position_1, relative_position_if_large_1); is_small = relative_position_1 = relative_position_if_large_1 = None 2025-03-14T07:25:53.0525263Z relative_buckets += where; relative_buckets_1: "i64[2048, 2048][2048, 1]cuda:0" = relative_buckets; relative_buckets = where = None 2025-03-14T07:25:53.0525842Z 2025-03-14T07:25:53.0527149Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:450 in compute_bias, code: values = self.relative_attention_bias(relative_position_bucket) # shape (query_length, key_length, num_heads) 2025-03-14T07:25:53.0529535Z values: "bf16[2048, 2048, 8][16384, 8, 1]cuda:0" = torch.nn.functional.embedding(relative_buckets_1, l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_relative_attention_bias_parameters_weight_, None, None, 2.0, False, False); relative_buckets_1 = l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_relative_attention_bias_parameters_weight_ = None 2025-03-14T07:25:53.0531339Z 2025-03-14T07:25:53.0532209Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:451 in compute_bias, code: values = values.permute([2, 0, 1]).unsqueeze(0) # shape (1, num_heads, query_length, key_length) 2025-03-14T07:25:53.0533300Z permute: "bf16[8, 2048, 2048][1, 16384, 8]cuda:0" = values.permute([2, 0, 1]); values = None 2025-03-14T07:25:53.0534036Z values_1: "bf16[1, 8, 2048, 2048][8, 1, 16384, 8]cuda:0" = permute.unsqueeze(0); permute = None 2025-03-14T07:25:53.0534480Z 2025-03-14T07:25:53.0535312Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:552 in forward, code: position_bias = position_bias + mask # (batch_size, n_heads, seq_length, key_length) 2025-03-14T07:25:53.0536539Z position_bias: "bf16[4, 8, 2048, 2048][33554432, 1, 16384, 8]cuda:0" = values_1 + extended_attention_mask_2; values_1 = extended_attention_mask_2 = None 2025-03-14T07:25:53.0537159Z 2025-03-14T07:25:53.0537817Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0538774Z scores += position_bias; scores_1: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores; scores = None 2025-03-14T07:25:53.0539296Z 2025-03-14T07:25:53.0540084Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.0541056Z float_2: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_1.float() 2025-03-14T07:25:53.0541736Z softmax: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_2, dim = -1); float_2 = None 2025-03-14T07:25:53.0542553Z attn_weights: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax.type_as(scores_1); softmax = scores_1 = None 2025-03-14T07:25:53.0543097Z 2025-03-14T07:25:53.0543779Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.0544930Z attn_weights_1: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights, p = 0.1, training = False); attn_weights = None 2025-03-14T07:25:53.0545600Z 2025-03-14T07:25:53.0546560Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.0547818Z matmul_1: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_1, hidden_states_4); attn_weights_1 = hidden_states_4 = None 2025-03-14T07:25:53.0548418Z 2025-03-14T07:25:53.0549240Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.0550315Z transpose_4: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_1.transpose(1, 2); matmul_1 = None 2025-03-14T07:25:53.0550992Z contiguous: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_4.contiguous(); transpose_4 = None 2025-03-14T07:25:53.0551668Z attn_output: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous.view(4, -1, 512); contiguous = None 2025-03-14T07:25:53.0552147Z 2025-03-14T07:25:53.0552814Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.0554717Z attn_output_1: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output, l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output = l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.0556041Z 2025-03-14T07:25:53.0556886Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.0558037Z dropout_2: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_1, 0.1, False, False); attn_output_1 = None 2025-03-14T07:25:53.0558871Z hidden_states_5: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states + dropout_2; hidden_states = dropout_2 = None 2025-03-14T07:25:53.0559401Z 2025-03-14T07:25:53.0560197Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0561174Z to_5: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_5.to(torch.float32) 2025-03-14T07:25:53.0561688Z pow_2: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_5.pow(2); to_5 = None 2025-03-14T07:25:53.0562245Z variance_1: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_2.mean(-1, keepdim = True); pow_2 = None 2025-03-14T07:25:53.0562685Z 2025-03-14T07:25:53.0563506Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0564501Z add_5: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_1 + 1e-06; variance_1 = None 2025-03-14T07:25:53.0565027Z rsqrt_1: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_5); add_5 = None 2025-03-14T07:25:53.0565614Z hidden_states_6: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_5 * rsqrt_1; rsqrt_1 = None 2025-03-14T07:25:53.0566081Z 2025-03-14T07:25:53.0566807Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0567850Z hidden_states_7: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_6.to(torch.bfloat16); hidden_states_6 = None 2025-03-14T07:25:53.0568375Z 2025-03-14T07:25:53.0569039Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0570700Z forwarded_states: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_7; l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_7 = None 2025-03-14T07:25:53.0571898Z 2025-03-14T07:25:53.0572573Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.0574444Z hidden_states_8: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states, l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states = l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.0575853Z 2025-03-14T07:25:53.0576531Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.0577701Z hidden_states_9: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_8, inplace = False); hidden_states_8 = None 2025-03-14T07:25:53.0578314Z 2025-03-14T07:25:53.0579017Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.0580214Z hidden_states_10: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_9, 0.1, False, False); hidden_states_9 = None 2025-03-14T07:25:53.0580849Z 2025-03-14T07:25:53.0581524Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.0583373Z hidden_states_11: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_10, l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_10 = l_self_modules_encoder_modules_block_modules_0_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.0584753Z 2025-03-14T07:25:53.0585538Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.0586805Z dropout_4: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_11, 0.1, False, False); hidden_states_11 = None 2025-03-14T07:25:53.0587839Z hidden_states_12: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_5 + dropout_4; hidden_states_5 = dropout_4 = None 2025-03-14T07:25:53.0588472Z 2025-03-14T07:25:53.0589413Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0590579Z to_7: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_12.to(torch.float32) 2025-03-14T07:25:53.0591160Z pow_3: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_7.pow(2); to_7 = None 2025-03-14T07:25:53.0591777Z variance_2: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_3.mean(-1, keepdim = True); pow_3 = None 2025-03-14T07:25:53.0592263Z 2025-03-14T07:25:53.0593217Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0594373Z add_7: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_2 + 1e-06; variance_2 = None 2025-03-14T07:25:53.0594938Z rsqrt_2: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_7); add_7 = None 2025-03-14T07:25:53.0595600Z hidden_states_13: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_12 * rsqrt_2; rsqrt_2 = None 2025-03-14T07:25:53.0596123Z 2025-03-14T07:25:53.0596961Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0598200Z hidden_states_14: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_13.to(torch.bfloat16); hidden_states_13 = None 2025-03-14T07:25:53.0598797Z 2025-03-14T07:25:53.0599557Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0601603Z normed_hidden_states_1: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_14; l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_14 = None 2025-03-14T07:25:53.0602908Z 2025-03-14T07:25:53.0603772Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.0605801Z linear_6: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_1, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.0607117Z 2025-03-14T07:25:53.0607968Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0609036Z view_5: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_6.view(4, -1, 8, 64); linear_6 = None 2025-03-14T07:25:53.0609656Z query_states_1: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_5.transpose(1, 2); view_5 = None 2025-03-14T07:25:53.0610111Z 2025-03-14T07:25:53.0610840Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0612655Z linear_7: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_1, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.0613968Z 2025-03-14T07:25:53.0614817Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0615888Z view_6: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_7.view(4, -1, 8, 64); linear_7 = None 2025-03-14T07:25:53.0616532Z hidden_states_15: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_6.transpose(1, 2); view_6 = None 2025-03-14T07:25:53.0617000Z 2025-03-14T07:25:53.0617723Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0619622Z linear_8: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_1, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_1 = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.0621014Z 2025-03-14T07:25:53.0621859Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0622930Z view_7: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_8.view(4, -1, 8, 64); linear_8 = None 2025-03-14T07:25:53.0623567Z hidden_states_16: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_7.transpose(1, 2); view_7 = None 2025-03-14T07:25:53.0624036Z 2025-03-14T07:25:53.0624734Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.0625807Z transpose_8: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_15.transpose(3, 2); hidden_states_15 = None 2025-03-14T07:25:53.0626749Z 2025-03-14T07:25:53.0627389Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.0628579Z scores_2: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.matmul(query_states_1, transpose_8); query_states_1 = transpose_8 = None 2025-03-14T07:25:53.0629196Z 2025-03-14T07:25:53.0629845Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0630805Z scores_2 += position_bias; scores_3: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_2; scores_2 = None 2025-03-14T07:25:53.0631346Z 2025-03-14T07:25:53.0632135Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.0633109Z float_3: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_3.float() 2025-03-14T07:25:53.0633802Z softmax_1: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_3, dim = -1); float_3 = None 2025-03-14T07:25:53.0634629Z attn_weights_2: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax_1.type_as(scores_3); softmax_1 = scores_3 = None 2025-03-14T07:25:53.0635183Z 2025-03-14T07:25:53.0635868Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.0637017Z attn_weights_3: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_2, p = 0.1, training = False); attn_weights_2 = None 2025-03-14T07:25:53.0637696Z 2025-03-14T07:25:53.0638573Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.0639853Z matmul_3: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_3, hidden_states_16); attn_weights_3 = hidden_states_16 = None 2025-03-14T07:25:53.0640461Z 2025-03-14T07:25:53.0641281Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.0642360Z transpose_9: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_3.transpose(1, 2); matmul_3 = None 2025-03-14T07:25:53.0643043Z contiguous_1: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_9.contiguous(); transpose_9 = None 2025-03-14T07:25:53.0643728Z attn_output_2: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous_1.view(4, -1, 512); contiguous_1 = None 2025-03-14T07:25:53.0644215Z 2025-03-14T07:25:53.0644898Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.0646692Z attn_output_3: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_2, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_2 = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.0648145Z 2025-03-14T07:25:53.0648917Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.0650068Z dropout_6: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_3, 0.1, False, False); attn_output_3 = None 2025-03-14T07:25:53.0651010Z hidden_states_17: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_12 + dropout_6; hidden_states_12 = dropout_6 = None 2025-03-14T07:25:53.0651561Z 2025-03-14T07:25:53.0652357Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0653345Z to_9: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_17.to(torch.float32) 2025-03-14T07:25:53.0653870Z pow_4: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_9.pow(2); to_9 = None 2025-03-14T07:25:53.0654423Z variance_3: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_4.mean(-1, keepdim = True); pow_4 = None 2025-03-14T07:25:53.0654863Z 2025-03-14T07:25:53.0655690Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0656695Z add_9: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_3 + 1e-06; variance_3 = None 2025-03-14T07:25:53.0657208Z rsqrt_3: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_9); add_9 = None 2025-03-14T07:25:53.0657796Z hidden_states_18: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_17 * rsqrt_3; rsqrt_3 = None 2025-03-14T07:25:53.0658272Z 2025-03-14T07:25:53.0659010Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0660056Z hidden_states_19: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_18.to(torch.bfloat16); hidden_states_18 = None 2025-03-14T07:25:53.0660586Z 2025-03-14T07:25:53.0661263Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0662941Z forwarded_states_1: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_19; l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_19 = None 2025-03-14T07:25:53.0664153Z 2025-03-14T07:25:53.0664837Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.0666797Z hidden_states_20: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_1, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_1 = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.0668211Z 2025-03-14T07:25:53.0668912Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.0670010Z hidden_states_21: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_20, inplace = False); hidden_states_20 = None 2025-03-14T07:25:53.0670652Z 2025-03-14T07:25:53.0671454Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.0672599Z hidden_states_22: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_21, 0.1, False, False); hidden_states_21 = None 2025-03-14T07:25:53.0673251Z 2025-03-14T07:25:53.0673938Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.0675876Z hidden_states_23: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_22, l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_22 = l_self_modules_encoder_modules_block_modules_1_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.0677276Z 2025-03-14T07:25:53.0678069Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.0679244Z dropout_8: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_23, 0.1, False, False); hidden_states_23 = None 2025-03-14T07:25:53.0680139Z hidden_states_24: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_17 + dropout_8; hidden_states_17 = dropout_8 = None 2025-03-14T07:25:53.0680693Z 2025-03-14T07:25:53.0681498Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0682505Z to_11: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_24.to(torch.float32) 2025-03-14T07:25:53.0683048Z pow_5: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_11.pow(2); to_11 = None 2025-03-14T07:25:53.0683607Z variance_4: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_5.mean(-1, keepdim = True); pow_5 = None 2025-03-14T07:25:53.0684048Z 2025-03-14T07:25:53.0684870Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0685859Z add_11: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_4 + 1e-06; variance_4 = None 2025-03-14T07:25:53.0686374Z rsqrt_4: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_11); add_11 = None 2025-03-14T07:25:53.0686966Z hidden_states_25: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_24 * rsqrt_4; rsqrt_4 = None 2025-03-14T07:25:53.0687429Z 2025-03-14T07:25:53.0688170Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0689220Z hidden_states_26: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_25.to(torch.bfloat16); hidden_states_25 = None 2025-03-14T07:25:53.0689755Z 2025-03-14T07:25:53.0690428Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0692108Z normed_hidden_states_2: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_26; l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_26 = None 2025-03-14T07:25:53.0693334Z 2025-03-14T07:25:53.0694200Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.0696243Z linear_12: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_2, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.0697545Z 2025-03-14T07:25:53.0698526Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0699610Z view_9: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_12.view(4, -1, 8, 64); linear_12 = None 2025-03-14T07:25:53.0700255Z query_states_2: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_9.transpose(1, 2); view_9 = None 2025-03-14T07:25:53.0700718Z 2025-03-14T07:25:53.0701452Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0703288Z linear_13: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_2, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.0704602Z 2025-03-14T07:25:53.0705450Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0706618Z view_10: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_13.view(4, -1, 8, 64); linear_13 = None 2025-03-14T07:25:53.0707273Z hidden_states_27: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_10.transpose(1, 2); view_10 = None 2025-03-14T07:25:53.0707744Z 2025-03-14T07:25:53.0708470Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0710384Z linear_14: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_2, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_2 = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.0711788Z 2025-03-14T07:25:53.0712645Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0713729Z view_11: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_14.view(4, -1, 8, 64); linear_14 = None 2025-03-14T07:25:53.0714384Z hidden_states_28: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_11.transpose(1, 2); view_11 = None 2025-03-14T07:25:53.0714861Z 2025-03-14T07:25:53.0715570Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.0716564Z transpose_13: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_27.transpose(3, 2); hidden_states_27 = None 2025-03-14T07:25:53.0717083Z 2025-03-14T07:25:53.0717720Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.0718839Z scores_4: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.matmul(query_states_2, transpose_13); query_states_2 = transpose_13 = None 2025-03-14T07:25:53.0719464Z 2025-03-14T07:25:53.0720121Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0721181Z scores_4 += position_bias; scores_5: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_4; scores_4 = None 2025-03-14T07:25:53.0721713Z 2025-03-14T07:25:53.0722500Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.0723470Z float_4: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_5.float() 2025-03-14T07:25:53.0724169Z softmax_2: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_4, dim = -1); float_4 = None 2025-03-14T07:25:53.0725000Z attn_weights_4: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax_2.type_as(scores_5); softmax_2 = scores_5 = None 2025-03-14T07:25:53.0725550Z 2025-03-14T07:25:53.0726480Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.0727693Z attn_weights_5: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_4, p = 0.1, training = False); attn_weights_4 = None 2025-03-14T07:25:53.0728379Z 2025-03-14T07:25:53.0729261Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.0730549Z matmul_5: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_5, hidden_states_28); attn_weights_5 = hidden_states_28 = None 2025-03-14T07:25:53.0731156Z 2025-03-14T07:25:53.0731992Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.0733074Z transpose_14: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_5.transpose(1, 2); matmul_5 = None 2025-03-14T07:25:53.0733763Z contiguous_2: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_14.contiguous(); transpose_14 = None 2025-03-14T07:25:53.0734464Z attn_output_4: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous_2.view(4, -1, 512); contiguous_2 = None 2025-03-14T07:25:53.0734958Z 2025-03-14T07:25:53.0735626Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.0737430Z attn_output_5: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_4, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_4 = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.0738821Z 2025-03-14T07:25:53.0739610Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.0740773Z dropout_10: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_5, 0.1, False, False); attn_output_5 = None 2025-03-14T07:25:53.0741783Z hidden_states_29: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_24 + dropout_10; hidden_states_24 = dropout_10 = None 2025-03-14T07:25:53.0742334Z 2025-03-14T07:25:53.0743124Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0744242Z to_13: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_29.to(torch.float32) 2025-03-14T07:25:53.0744769Z pow_6: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_13.pow(2); to_13 = None 2025-03-14T07:25:53.0745319Z variance_5: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_6.mean(-1, keepdim = True); pow_6 = None 2025-03-14T07:25:53.0745756Z 2025-03-14T07:25:53.0746647Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0747644Z add_13: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_5 + 1e-06; variance_5 = None 2025-03-14T07:25:53.0748158Z rsqrt_5: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_13); add_13 = None 2025-03-14T07:25:53.0748740Z hidden_states_30: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_29 * rsqrt_5; rsqrt_5 = None 2025-03-14T07:25:53.0749202Z 2025-03-14T07:25:53.0749939Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0750980Z hidden_states_31: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_30.to(torch.bfloat16); hidden_states_30 = None 2025-03-14T07:25:53.0751509Z 2025-03-14T07:25:53.0752176Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0753845Z forwarded_states_2: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_31; l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_31 = None 2025-03-14T07:25:53.0755052Z 2025-03-14T07:25:53.0755731Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.0757617Z hidden_states_32: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_2, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_2 = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.0759026Z 2025-03-14T07:25:53.0759708Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.0760799Z hidden_states_33: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_32, inplace = False); hidden_states_32 = None 2025-03-14T07:25:53.0761432Z 2025-03-14T07:25:53.0762133Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.0763245Z hidden_states_34: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_33, 0.1, False, False); hidden_states_33 = None 2025-03-14T07:25:53.0771691Z 2025-03-14T07:25:53.0772423Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.0774390Z hidden_states_35: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_34, l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_34 = l_self_modules_encoder_modules_block_modules_2_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.0775772Z 2025-03-14T07:25:53.0776539Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.0777705Z dropout_12: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_35, 0.1, False, False); hidden_states_35 = None 2025-03-14T07:25:53.0778589Z hidden_states_36: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_29 + dropout_12; hidden_states_29 = dropout_12 = None 2025-03-14T07:25:53.0779181Z 2025-03-14T07:25:53.0779981Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0780976Z to_15: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_36.to(torch.float32) 2025-03-14T07:25:53.0781493Z pow_7: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_15.pow(2); to_15 = None 2025-03-14T07:25:53.0782041Z variance_6: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_7.mean(-1, keepdim = True); pow_7 = None 2025-03-14T07:25:53.0782470Z 2025-03-14T07:25:53.0783286Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0784278Z add_15: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_6 + 1e-06; variance_6 = None 2025-03-14T07:25:53.0784790Z rsqrt_6: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_15); add_15 = None 2025-03-14T07:25:53.0785382Z hidden_states_37: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_36 * rsqrt_6; rsqrt_6 = None 2025-03-14T07:25:53.0785846Z 2025-03-14T07:25:53.0786672Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0787711Z hidden_states_38: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_37.to(torch.bfloat16); hidden_states_37 = None 2025-03-14T07:25:53.0788233Z 2025-03-14T07:25:53.0788899Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0790560Z normed_hidden_states_3: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_38; l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_38 = None 2025-03-14T07:25:53.0791763Z 2025-03-14T07:25:53.0792623Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.0794574Z linear_18: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_3, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.0795967Z 2025-03-14T07:25:53.0796801Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0797945Z view_13: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_18.view(4, -1, 8, 64); linear_18 = None 2025-03-14T07:25:53.0798583Z query_states_3: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_13.transpose(1, 2); view_13 = None 2025-03-14T07:25:53.0799046Z 2025-03-14T07:25:53.0799755Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0801565Z linear_19: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_3, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.0802863Z 2025-03-14T07:25:53.0803704Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0804768Z view_14: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_19.view(4, -1, 8, 64); linear_19 = None 2025-03-14T07:25:53.0805406Z hidden_states_39: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_14.transpose(1, 2); view_14 = None 2025-03-14T07:25:53.0805874Z 2025-03-14T07:25:53.0806586Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0808500Z linear_20: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_3, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_3 = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.0809904Z 2025-03-14T07:25:53.0810740Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0811816Z view_15: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_20.view(4, -1, 8, 64); linear_20 = None 2025-03-14T07:25:53.0812457Z hidden_states_40: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_15.transpose(1, 2); view_15 = None 2025-03-14T07:25:53.0812927Z 2025-03-14T07:25:53.0813619Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.0814601Z transpose_18: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_39.transpose(3, 2); hidden_states_39 = None 2025-03-14T07:25:53.0815112Z 2025-03-14T07:25:53.0815734Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.0816750Z scores_6: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.matmul(query_states_3, transpose_18); query_states_3 = transpose_18 = None 2025-03-14T07:25:53.0817468Z 2025-03-14T07:25:53.0818115Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0819082Z scores_6 += position_bias; scores_7: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_6; scores_6 = None 2025-03-14T07:25:53.0819607Z 2025-03-14T07:25:53.0820476Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.0821443Z float_5: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_7.float() 2025-03-14T07:25:53.0822127Z softmax_3: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_5, dim = -1); float_5 = None 2025-03-14T07:25:53.0822957Z attn_weights_6: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax_3.type_as(scores_7); softmax_3 = scores_7 = None 2025-03-14T07:25:53.0823515Z 2025-03-14T07:25:53.0824198Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.0825340Z attn_weights_7: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_6, p = 0.1, training = False); attn_weights_6 = None 2025-03-14T07:25:53.0826019Z 2025-03-14T07:25:53.0827209Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.0828472Z matmul_7: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_7, hidden_states_40); attn_weights_7 = hidden_states_40 = None 2025-03-14T07:25:53.0829087Z 2025-03-14T07:25:53.0829899Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.0830978Z transpose_19: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_7.transpose(1, 2); matmul_7 = None 2025-03-14T07:25:53.0831663Z contiguous_3: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_19.contiguous(); transpose_19 = None 2025-03-14T07:25:53.0832368Z attn_output_6: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous_3.view(4, -1, 512); contiguous_3 = None 2025-03-14T07:25:53.0832848Z 2025-03-14T07:25:53.0833510Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.0835300Z attn_output_7: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_6, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_6 = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.0836642Z 2025-03-14T07:25:53.0837420Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.0838573Z dropout_14: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_7, 0.1, False, False); attn_output_7 = None 2025-03-14T07:25:53.0839431Z hidden_states_41: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_36 + dropout_14; hidden_states_36 = dropout_14 = None 2025-03-14T07:25:53.0839969Z 2025-03-14T07:25:53.0840763Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0841891Z to_17: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_41.to(torch.float32) 2025-03-14T07:25:53.0842408Z pow_8: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_17.pow(2); to_17 = None 2025-03-14T07:25:53.0842958Z variance_7: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_8.mean(-1, keepdim = True); pow_8 = None 2025-03-14T07:25:53.0843397Z 2025-03-14T07:25:53.0844340Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0845340Z add_17: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_7 + 1e-06; variance_7 = None 2025-03-14T07:25:53.0845856Z rsqrt_7: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_17); add_17 = None 2025-03-14T07:25:53.0846447Z hidden_states_42: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_41 * rsqrt_7; rsqrt_7 = None 2025-03-14T07:25:53.0846915Z 2025-03-14T07:25:53.0847651Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0848695Z hidden_states_43: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_42.to(torch.bfloat16); hidden_states_42 = None 2025-03-14T07:25:53.0849225Z 2025-03-14T07:25:53.0849889Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0851544Z forwarded_states_3: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_43; l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_43 = None 2025-03-14T07:25:53.0852757Z 2025-03-14T07:25:53.0853439Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.0855322Z hidden_states_44: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_3, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_3 = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.0856729Z 2025-03-14T07:25:53.0857404Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.0858487Z hidden_states_45: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_44, inplace = False); hidden_states_44 = None 2025-03-14T07:25:53.0859103Z 2025-03-14T07:25:53.0859792Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.0860917Z hidden_states_46: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_45, 0.1, False, False); hidden_states_45 = None 2025-03-14T07:25:53.0861551Z 2025-03-14T07:25:53.0862221Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.0864053Z hidden_states_47: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_46, l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_46 = l_self_modules_encoder_modules_block_modules_3_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.0865500Z 2025-03-14T07:25:53.0866254Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.0867560Z dropout_16: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_47, 0.1, False, False); hidden_states_47 = None 2025-03-14T07:25:53.0868433Z hidden_states_48: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_41 + dropout_16; hidden_states_41 = dropout_16 = None 2025-03-14T07:25:53.0868970Z 2025-03-14T07:25:53.0869762Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0870750Z to_19: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_48.to(torch.float32) 2025-03-14T07:25:53.0871271Z pow_9: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_19.pow(2); to_19 = None 2025-03-14T07:25:53.0871823Z variance_8: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_9.mean(-1, keepdim = True); pow_9 = None 2025-03-14T07:25:53.0872257Z 2025-03-14T07:25:53.0873072Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0874060Z add_19: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_8 + 1e-06; variance_8 = None 2025-03-14T07:25:53.0874575Z rsqrt_8: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_19); add_19 = None 2025-03-14T07:25:53.0875170Z hidden_states_49: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_48 * rsqrt_8; rsqrt_8 = None 2025-03-14T07:25:53.0875637Z 2025-03-14T07:25:53.0876378Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0877415Z hidden_states_50: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_49.to(torch.bfloat16); hidden_states_49 = None 2025-03-14T07:25:53.0877939Z 2025-03-14T07:25:53.0878600Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0880259Z normed_hidden_states_4: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_50; l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_50 = None 2025-03-14T07:25:53.0881476Z 2025-03-14T07:25:53.0882323Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.0884270Z linear_24: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_4, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.0885567Z 2025-03-14T07:25:53.0886405Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0887575Z view_17: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_24.view(4, -1, 8, 64); linear_24 = None 2025-03-14T07:25:53.0888210Z query_states_4: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_17.transpose(1, 2); view_17 = None 2025-03-14T07:25:53.0888663Z 2025-03-14T07:25:53.0889461Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0891274Z linear_25: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_4, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.0892579Z 2025-03-14T07:25:53.0893414Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0894476Z view_18: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_25.view(4, -1, 8, 64); linear_25 = None 2025-03-14T07:25:53.0895118Z hidden_states_51: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_18.transpose(1, 2); view_18 = None 2025-03-14T07:25:53.0895587Z 2025-03-14T07:25:53.0896303Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0898192Z linear_26: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_4, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_4 = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.0899582Z 2025-03-14T07:25:53.0900413Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0901479Z view_19: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_26.view(4, -1, 8, 64); linear_26 = None 2025-03-14T07:25:53.0902118Z hidden_states_52: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_19.transpose(1, 2); view_19 = None 2025-03-14T07:25:53.0902577Z 2025-03-14T07:25:53.0903267Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.0904252Z transpose_23: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_51.transpose(3, 2); hidden_states_51 = None 2025-03-14T07:25:53.0904761Z 2025-03-14T07:25:53.0905383Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.0906458Z scores_8: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.matmul(query_states_4, transpose_23); query_states_4 = transpose_23 = None 2025-03-14T07:25:53.0907068Z 2025-03-14T07:25:53.0907713Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0908667Z scores_8 += position_bias; scores_9: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_8; scores_8 = None 2025-03-14T07:25:53.0909303Z 2025-03-14T07:25:53.0910087Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.0911054Z float_6: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_9.float() 2025-03-14T07:25:53.0911735Z softmax_4: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_6, dim = -1); float_6 = None 2025-03-14T07:25:53.0912645Z attn_weights_8: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax_4.type_as(scores_9); softmax_4 = scores_9 = None 2025-03-14T07:25:53.0913200Z 2025-03-14T07:25:53.0913877Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.0915033Z attn_weights_9: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_8, p = 0.1, training = False); attn_weights_8 = None 2025-03-14T07:25:53.0915712Z 2025-03-14T07:25:53.0916582Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.0917850Z matmul_9: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_9, hidden_states_52); attn_weights_9 = hidden_states_52 = None 2025-03-14T07:25:53.0918459Z 2025-03-14T07:25:53.0919275Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.0920354Z transpose_24: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_9.transpose(1, 2); matmul_9 = None 2025-03-14T07:25:53.0921058Z contiguous_4: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_24.contiguous(); transpose_24 = None 2025-03-14T07:25:53.0921751Z attn_output_8: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous_4.view(4, -1, 512); contiguous_4 = None 2025-03-14T07:25:53.0922240Z 2025-03-14T07:25:53.0922911Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.0924706Z attn_output_9: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_8, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_8 = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.0926040Z 2025-03-14T07:25:53.0927045Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.0928199Z dropout_18: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_9, 0.1, False, False); attn_output_9 = None 2025-03-14T07:25:53.0929060Z hidden_states_53: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_48 + dropout_18; hidden_states_48 = dropout_18 = None 2025-03-14T07:25:53.0929610Z 2025-03-14T07:25:53.0930411Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0931398Z to_21: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_53.to(torch.float32) 2025-03-14T07:25:53.0932042Z pow_10: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_21.pow(2); to_21 = None 2025-03-14T07:25:53.0932597Z variance_9: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_10.mean(-1, keepdim = True); pow_10 = None 2025-03-14T07:25:53.0933044Z 2025-03-14T07:25:53.0933865Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0934962Z add_21: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_9 + 1e-06; variance_9 = None 2025-03-14T07:25:53.0935483Z rsqrt_9: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_21); add_21 = None 2025-03-14T07:25:53.0936073Z hidden_states_54: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_53 * rsqrt_9; rsqrt_9 = None 2025-03-14T07:25:53.0936539Z 2025-03-14T07:25:53.0937272Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0938321Z hidden_states_55: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_54.to(torch.bfloat16); hidden_states_54 = None 2025-03-14T07:25:53.0938847Z 2025-03-14T07:25:53.0939510Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0941169Z forwarded_states_4: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_55; l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_55 = None 2025-03-14T07:25:53.0942367Z 2025-03-14T07:25:53.0943047Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.0944922Z hidden_states_56: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_4, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_4 = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.0946331Z 2025-03-14T07:25:53.0947079Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.0948173Z hidden_states_57: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_56, inplace = False); hidden_states_56 = None 2025-03-14T07:25:53.0948796Z 2025-03-14T07:25:53.0949490Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.0950619Z hidden_states_58: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_57, 0.1, False, False); hidden_states_57 = None 2025-03-14T07:25:53.0951262Z 2025-03-14T07:25:53.0951936Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.0953784Z hidden_states_59: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_58, l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_58 = l_self_modules_encoder_modules_block_modules_4_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.0955237Z 2025-03-14T07:25:53.0956004Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.0957164Z dropout_20: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_59, 0.1, False, False); hidden_states_59 = None 2025-03-14T07:25:53.0958043Z hidden_states_60: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_53 + dropout_20; hidden_states_53 = dropout_20 = None 2025-03-14T07:25:53.0958718Z 2025-03-14T07:25:53.0959514Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.0960502Z to_23: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_60.to(torch.float32) 2025-03-14T07:25:53.0961036Z pow_11: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_23.pow(2); to_23 = None 2025-03-14T07:25:53.0961599Z variance_10: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_11.mean(-1, keepdim = True); pow_11 = None 2025-03-14T07:25:53.0962052Z 2025-03-14T07:25:53.0962874Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.0963880Z add_23: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_10 + 1e-06; variance_10 = None 2025-03-14T07:25:53.0964410Z rsqrt_10: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_23); add_23 = None 2025-03-14T07:25:53.0965006Z hidden_states_61: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_60 * rsqrt_10; rsqrt_10 = None 2025-03-14T07:25:53.0965478Z 2025-03-14T07:25:53.0966219Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.0967270Z hidden_states_62: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_61.to(torch.bfloat16); hidden_states_61 = None 2025-03-14T07:25:53.0967799Z 2025-03-14T07:25:53.0968471Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.0970148Z normed_hidden_states_5: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_62; l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_62 = None 2025-03-14T07:25:53.0971359Z 2025-03-14T07:25:53.0972215Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.0974163Z linear_30: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_5, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.0975479Z 2025-03-14T07:25:53.0976336Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0977419Z view_21: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_30.view(4, -1, 8, 64); linear_30 = None 2025-03-14T07:25:53.0978060Z query_states_5: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_21.transpose(1, 2); view_21 = None 2025-03-14T07:25:53.0978612Z 2025-03-14T07:25:53.0979340Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0981250Z linear_31: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_5, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.0982555Z 2025-03-14T07:25:53.0983399Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0984481Z view_22: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_31.view(4, -1, 8, 64); linear_31 = None 2025-03-14T07:25:53.0985114Z hidden_states_63: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_22.transpose(1, 2); view_22 = None 2025-03-14T07:25:53.0985575Z 2025-03-14T07:25:53.0986289Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.0988247Z linear_32: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_5, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_5 = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.0989674Z 2025-03-14T07:25:53.0990509Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.0991575Z view_23: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_32.view(4, -1, 8, 64); linear_32 = None 2025-03-14T07:25:53.0992222Z hidden_states_64: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_23.transpose(1, 2); view_23 = None 2025-03-14T07:25:53.0992690Z 2025-03-14T07:25:53.0993385Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.0994366Z transpose_28: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_63.transpose(3, 2); hidden_states_63 = None 2025-03-14T07:25:53.0994879Z 2025-03-14T07:25:53.0995508Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.0996538Z scores_10: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.matmul(query_states_5, transpose_28); query_states_5 = transpose_28 = None 2025-03-14T07:25:53.0997150Z 2025-03-14T07:25:53.0997797Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.0998818Z scores_10 += position_bias; scores_11: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_10; scores_10 = position_bias = None 2025-03-14T07:25:53.0999400Z 2025-03-14T07:25:53.1000184Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1001258Z float_7: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = scores_11.float() 2025-03-14T07:25:53.1001943Z softmax_5: "f32[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_7, dim = -1); float_7 = None 2025-03-14T07:25:53.1002789Z attn_weights_10: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = softmax_5.type_as(scores_11); softmax_5 = scores_11 = None 2025-03-14T07:25:53.1003347Z 2025-03-14T07:25:53.1004103Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1005261Z attn_weights_11: "bf16[4, 8, 2048, 2048][33554432, 4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_10, p = 0.1, training = False); attn_weights_10 = None 2025-03-14T07:25:53.1005962Z 2025-03-14T07:25:53.1006835Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1008121Z matmul_11: "bf16[4, 8, 2048, 64][1048576, 131072, 64, 1]cuda:0" = torch.matmul(attn_weights_11, hidden_states_64); attn_weights_11 = hidden_states_64 = None 2025-03-14T07:25:53.1008735Z 2025-03-14T07:25:53.1009558Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1010645Z transpose_29: "bf16[4, 2048, 8, 64][1048576, 64, 131072, 1]cuda:0" = matmul_11.transpose(1, 2); matmul_11 = None 2025-03-14T07:25:53.1011339Z contiguous_5: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = transpose_29.contiguous(); transpose_29 = None 2025-03-14T07:25:53.1012032Z attn_output_10: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = contiguous_5.view(4, -1, 512); contiguous_5 = None 2025-03-14T07:25:53.1012528Z 2025-03-14T07:25:53.1013186Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1014990Z attn_output_11: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_10, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_10 = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1016329Z 2025-03-14T07:25:53.1017107Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1018263Z dropout_22: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_11, 0.1, False, False); attn_output_11 = None 2025-03-14T07:25:53.1019129Z hidden_states_65: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_60 + dropout_22; hidden_states_60 = dropout_22 = None 2025-03-14T07:25:53.1019669Z 2025-03-14T07:25:53.1020463Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1021449Z to_25: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_65.to(torch.float32) 2025-03-14T07:25:53.1021966Z pow_12: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_25.pow(2); to_25 = None 2025-03-14T07:25:53.1022518Z variance_11: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_12.mean(-1, keepdim = True); pow_12 = None 2025-03-14T07:25:53.1022957Z 2025-03-14T07:25:53.1023775Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1024850Z add_25: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_11 + 1e-06; variance_11 = None 2025-03-14T07:25:53.1025376Z rsqrt_11: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_25); add_25 = None 2025-03-14T07:25:53.1025969Z hidden_states_66: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_65 * rsqrt_11; rsqrt_11 = None 2025-03-14T07:25:53.1026873Z 2025-03-14T07:25:53.1027611Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1028650Z hidden_states_67: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_66.to(torch.bfloat16); hidden_states_66 = None 2025-03-14T07:25:53.1029189Z 2025-03-14T07:25:53.1029842Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1031509Z forwarded_states_5: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_67; l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_67 = None 2025-03-14T07:25:53.1032717Z 2025-03-14T07:25:53.1033396Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1035271Z hidden_states_68: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_5, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_5 = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1036688Z 2025-03-14T07:25:53.1037376Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1038481Z hidden_states_69: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_68, inplace = False); hidden_states_68 = None 2025-03-14T07:25:53.1039109Z 2025-03-14T07:25:53.1039813Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1040945Z hidden_states_70: "bf16[4, 2048, 2048][4194304, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_69, 0.1, False, False); hidden_states_69 = None 2025-03-14T07:25:53.1041588Z 2025-03-14T07:25:53.1042264Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1044111Z hidden_states_71: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_70, l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_70 = l_self_modules_encoder_modules_block_modules_5_modules_layer_modules_1_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1045487Z 2025-03-14T07:25:53.1046264Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1047437Z dropout_24: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_71, 0.1, False, False); hidden_states_71 = None 2025-03-14T07:25:53.1048431Z hidden_states_72: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_65 + dropout_24; hidden_states_65 = dropout_24 = None 2025-03-14T07:25:53.1048974Z 2025-03-14T07:25:53.1049778Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1050852Z to_27: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_72.to(torch.float32) 2025-03-14T07:25:53.1051377Z pow_13: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = to_27.pow(2); to_27 = None 2025-03-14T07:25:53.1051941Z variance_12: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = pow_13.mean(-1, keepdim = True); pow_13 = None 2025-03-14T07:25:53.1052388Z 2025-03-14T07:25:53.1053217Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1054215Z add_27: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = variance_12 + 1e-06; variance_12 = None 2025-03-14T07:25:53.1054746Z rsqrt_12: "f32[4, 2048, 1][2048, 1, 1]cuda:0" = torch.rsqrt(add_27); add_27 = None 2025-03-14T07:25:53.1055420Z hidden_states_73: "f32[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_72 * rsqrt_12; hidden_states_72 = rsqrt_12 = None 2025-03-14T07:25:53.1055964Z 2025-03-14T07:25:53.1056705Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1057762Z hidden_states_74: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = hidden_states_73.to(torch.bfloat16); hidden_states_73 = None 2025-03-14T07:25:53.1058305Z 2025-03-14T07:25:53.1058998Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1060365Z hidden_states_75: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = l_self_modules_encoder_modules_final_layer_norm_parameters_weight_ * hidden_states_74; l_self_modules_encoder_modules_final_layer_norm_parameters_weight_ = hidden_states_74 = None 2025-03-14T07:25:53.1061292Z 2025-03-14T07:25:53.1062015Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1158 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1063151Z hidden_states_76: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_75, 0.1, False, False); hidden_states_75 = None 2025-03-14T07:25:53.1063794Z 2025-03-14T07:25:53.1064744Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:554 in _prepare_decoder_input_ids_for_generation, code: torch.ones((batch_size, 1), dtype=torch.long, device=device) * decoder_start_token_id 2025-03-14T07:25:53.1065933Z ones: "i64[4, 1][1, 1]cuda:0" = torch.ones((4, 1), dtype = torch.int64, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1066599Z decoder_input_ids_start: "i64[4, 1][1, 1]cuda:0" = ones * 0; ones = None 2025-03-14T07:25:53.1066998Z 2025-03-14T07:25:53.1067912Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2357 in greedy_search, code: eos_token_id_tensor = torch.tensor(eos_token_id).to(input_ids.device) if eos_token_id is not None else None 2025-03-14T07:25:53.1068913Z tensor: "i64[1][1]cpu" = torch.tensor([1]) 2025-03-14T07:25:53.1069421Z eos_token_id_tensor: "i64[1][1]cuda:0" = tensor.to(device(type='cuda', index=0)); tensor = None 2025-03-14T07:25:53.1069965Z 2025-03-14T07:25:53.1070863Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2386 in greedy_search, code: unfinished_sequences = torch.ones(input_ids.shape[0], dtype=torch.long, device=input_ids.device) 2025-03-14T07:25:53.1078063Z unfinished_sequences: "i64[4][1]cuda:0" = torch.ones(4, dtype = torch.int64, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1078596Z 2025-03-14T07:25:53.1079523Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1011 in forward, code: input_ids = input_ids.view(-1, input_shape[-1]) 2025-03-14T07:25:53.1080414Z input_ids_1: "i64[4, 1][1, 1]cuda:0" = decoder_input_ids_start.view(-1, 1) 2025-03-14T07:25:53.1080810Z 2025-03-14T07:25:53.1081511Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1021 in forward, code: inputs_embeds = self.embed_tokens(input_ids) 2025-03-14T07:25:53.1082843Z inputs_embeds_1: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.embedding(input_ids_1, l_self_modules_encoder_modules_embed_tokens_parameters_weight_, None, None, 2.0, False, False); input_ids_1 = None 2025-03-14T07:25:53.1083671Z 2025-03-14T07:25:53.1084504Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1037 in forward, code: attention_mask = torch.ones(batch_size, mask_seq_length, device=inputs_embeds.device) 2025-03-14T07:25:53.1085590Z attention_mask: "f32[4, 1][1, 1]cuda:0" = torch.ones(4, 1, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1086041Z 2025-03-14T07:25:53.1086850Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:922 in create_extended_attention_mask_for_decoder, code: seq_ids = torch.arange(seq_length, device=device) 2025-03-14T07:25:53.1087850Z seq_ids: "i64[1][1]cuda:0" = torch.arange(1, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1088280Z 2025-03-14T07:25:53.1089263Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:923 in create_extended_attention_mask_for_decoder, code: causal_mask = seq_ids[None, None, :].repeat(batch_size, seq_length, 1) <= seq_ids[None, :, None] 2025-03-14T07:25:53.1090392Z getitem_3: "i64[1, 1, 1][1, 1, 1]cuda:0" = seq_ids[(None, None, slice(None, None, None))] 2025-03-14T07:25:53.1090920Z repeat: "i64[4, 1, 1][1, 1, 1]cuda:0" = getitem_3.repeat(4, 1, 1); getitem_3 = None 2025-03-14T07:25:53.1091486Z getitem_4: "i64[1, 1, 1][1, 1, 1]cuda:0" = seq_ids[(None, slice(None, None, None), None)]; seq_ids = None 2025-03-14T07:25:53.1092075Z causal_mask: "b8[4, 1, 1][1, 1, 1]cuda:0" = repeat <= getitem_4; repeat = getitem_4 = None 2025-03-14T07:25:53.1092489Z 2025-03-14T07:25:53.1093305Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:926 in create_extended_attention_mask_for_decoder, code: causal_mask = causal_mask.to(attention_mask.dtype) 2025-03-14T07:25:53.1094333Z causal_mask_1: "f32[4, 1, 1][1, 1, 1]cuda:0" = causal_mask.to(torch.float32); causal_mask = None 2025-03-14T07:25:53.1094770Z 2025-03-14T07:25:53.1095694Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:938 in create_extended_attention_mask_for_decoder, code: extended_attention_mask = causal_mask[:, None, :, :] * attention_mask[:, None, None, :] 2025-03-14T07:25:53.1097011Z getitem_5: "f32[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = causal_mask_1[(slice(None, None, None), None, slice(None, None, None), slice(None, None, None))]; causal_mask_1 = None 2025-03-14T07:25:53.1097914Z getitem_6: "f32[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = attention_mask[(slice(None, None, None), None, None, slice(None, None, None))]; attention_mask = None 2025-03-14T07:25:53.1098784Z extended_attention_mask_3: "f32[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = getitem_5 * getitem_6; getitem_5 = getitem_6 = None 2025-03-14T07:25:53.1099271Z 2025-03-14T07:25:53.1100154Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:989 in get_extended_attention_mask, code: extended_attention_mask = extended_attention_mask.to(dtype=dtype) # fp16 compatibility 2025-03-14T07:25:53.1101516Z extended_attention_mask_4: "bf16[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = extended_attention_mask_3.to(dtype = torch.bfloat16); extended_attention_mask_3 = None 2025-03-14T07:25:53.1102120Z 2025-03-14T07:25:53.1102985Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:990 in get_extended_attention_mask, code: extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min 2025-03-14T07:25:53.1104084Z sub_2: "bf16[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = 1.0 - extended_attention_mask_4; extended_attention_mask_4 = None 2025-03-14T07:25:53.1104732Z extended_attention_mask_5: "bf16[4, 1, 1, 1][1, 1, 1, 1]cuda:0" = sub_2 * -3.3895313892515355e+38; sub_2 = None 2025-03-14T07:25:53.1105195Z 2025-03-14T07:25:53.1106021Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:902 in invert_attention_mask, code: encoder_extended_attention_mask = encoder_attention_mask[:, None, None, :] 2025-03-14T07:25:53.1107379Z encoder_extended_attention_mask: "i64[4, 1, 1, 2048][2048, 2048, 2048, 1]cuda:0" = l_stack0_[(slice(None, None, None), None, None, slice(None, None, None))]; l_stack0_ = None 2025-03-14T07:25:53.1108031Z 2025-03-14T07:25:53.1108964Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:908 in invert_attention_mask, code: encoder_extended_attention_mask = encoder_extended_attention_mask.to(dtype=self.dtype) # fp16 compatibility 2025-03-14T07:25:53.1110414Z encoder_extended_attention_mask_1: "bf16[4, 1, 1, 2048][2048, 2048, 2048, 1]cuda:0" = encoder_extended_attention_mask.to(dtype = torch.bfloat16); encoder_extended_attention_mask = None 2025-03-14T07:25:53.1111135Z 2025-03-14T07:25:53.1112041Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/modeling_utils.py:909 in invert_attention_mask, code: encoder_extended_attention_mask = (1.0 - encoder_extended_attention_mask) * torch.finfo(self.dtype).min 2025-03-14T07:25:53.1113288Z sub_3: "bf16[4, 1, 1, 2048][2048, 2048, 2048, 1]cuda:0" = 1.0 - encoder_extended_attention_mask_1; encoder_extended_attention_mask_1 = None 2025-03-14T07:25:53.1114095Z encoder_extended_attention_mask_2: "bf16[4, 1, 1, 2048][2048, 2048, 2048, 1]cuda:0" = sub_3 * -3.3895313892515355e+38; sub_3 = None 2025-03-14T07:25:53.1114625Z 2025-03-14T07:25:53.1115322Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1073 in forward, code: hidden_states = self.dropout(inputs_embeds) 2025-03-14T07:25:53.1116411Z hidden_states_77: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(inputs_embeds_1, 0.1, False, False); inputs_embeds_1 = None 2025-03-14T07:25:53.1117001Z 2025-03-14T07:25:53.1117790Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1118799Z to_33: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_77.to(torch.float32) 2025-03-14T07:25:53.1119276Z pow_14: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_33.pow(2); to_33 = None 2025-03-14T07:25:53.1119778Z variance_13: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_14.mean(-1, keepdim = True); pow_14 = None 2025-03-14T07:25:53.1120197Z 2025-03-14T07:25:53.1121013Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1122065Z add_28: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_13 + 1e-06; variance_13 = None 2025-03-14T07:25:53.1122543Z rsqrt_13: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_28); add_28 = None 2025-03-14T07:25:53.1123086Z hidden_states_78: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_77 * rsqrt_13; rsqrt_13 = None 2025-03-14T07:25:53.1123530Z 2025-03-14T07:25:53.1124356Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1125377Z hidden_states_79: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_78.to(torch.bfloat16); hidden_states_78 = None 2025-03-14T07:25:53.1125872Z 2025-03-14T07:25:53.1126814Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1128465Z normed_hidden_states_6: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_79; l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_79 = None 2025-03-14T07:25:53.1129638Z 2025-03-14T07:25:53.1130492Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1132409Z linear_36: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_6, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1133681Z 2025-03-14T07:25:53.1134514Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1135559Z view_26: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_36.view(4, -1, 8, 64); linear_36 = None 2025-03-14T07:25:53.1136143Z query_states_6: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_26.transpose(1, 2); view_26 = None 2025-03-14T07:25:53.1136573Z 2025-03-14T07:25:53.1137288Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1139064Z linear_37: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_6, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1140330Z 2025-03-14T07:25:53.1141173Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1142208Z view_27: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_37.view(4, -1, 8, 64); linear_37 = None 2025-03-14T07:25:53.1142799Z hidden_states_80: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_27.transpose(1, 2); view_27 = None 2025-03-14T07:25:53.1143232Z 2025-03-14T07:25:53.1143951Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1145961Z linear_38: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_6, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_6 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1147378Z 2025-03-14T07:25:53.1148336Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1149381Z view_28: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_38.view(4, -1, 8, 64); linear_38 = None 2025-03-14T07:25:53.1149958Z hidden_states_81: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_28.transpose(1, 2); view_28 = None 2025-03-14T07:25:53.1150405Z 2025-03-14T07:25:53.1151095Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1152053Z transpose_33: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_80.transpose(3, 2); hidden_states_80 = None 2025-03-14T07:25:53.1152530Z 2025-03-14T07:25:53.1153159Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1154103Z scores_12: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_6, transpose_33); query_states_6 = transpose_33 = None 2025-03-14T07:25:53.1154631Z 2025-03-14T07:25:53.1155510Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:441 in compute_bias, code: context_position = torch.arange(query_length, dtype=torch.long, device=device)[:, None] 2025-03-14T07:25:53.1156641Z arange_3: "i64[1][1]cuda:0" = torch.arange(1, dtype = torch.int64, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1157314Z context_position_1: "i64[1, 1][1, 1]cuda:0" = arange_3[(slice(None, None, None), None)]; arange_3 = None 2025-03-14T07:25:53.1157782Z 2025-03-14T07:25:53.1158642Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:442 in compute_bias, code: memory_position = torch.arange(key_length, dtype=torch.long, device=device)[None, :] 2025-03-14T07:25:53.1159760Z arange_4: "i64[1][1]cuda:0" = torch.arange(1, dtype = torch.int64, device = device(type='cuda', index=0)) 2025-03-14T07:25:53.1160440Z memory_position_1: "i64[1, 1][1, 1]cuda:0" = arange_4[(None, slice(None, None, None))]; arange_4 = None 2025-03-14T07:25:53.1160898Z 2025-03-14T07:25:53.1161797Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:443 in compute_bias, code: relative_position = memory_position - context_position # shape (query_length, key_length) 2025-03-14T07:25:53.1163035Z relative_position_2: "i64[1, 1][1, 1]cuda:0" = memory_position_1 - context_position_1; memory_position_1 = context_position_1 = None 2025-03-14T07:25:53.1163598Z 2025-03-14T07:25:53.1164511Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:417 in _relative_position_bucket, code: relative_position = -torch.min(relative_position, torch.zeros_like(relative_position)) 2025-03-14T07:25:53.1165590Z zeros_like: "i64[1, 1][1, 1]cuda:0" = torch.zeros_like(relative_position_2) 2025-03-14T07:25:53.1166217Z min_2: "i64[1, 1][1, 1]cuda:0" = torch.min(relative_position_2, zeros_like); relative_position_2 = zeros_like = None 2025-03-14T07:25:53.1166943Z relative_position_3: "i64[1, 1][1, 1]cuda:0" = -min_2; min_2 = None 2025-03-14T07:25:53.1167335Z 2025-03-14T07:25:53.1168082Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:422 in _relative_position_bucket, code: is_small = relative_position < max_exact 2025-03-14T07:25:53.1168970Z is_small_1: "b8[1, 1][1, 1]cuda:0" = relative_position_3 < 16 2025-03-14T07:25:53.1169322Z 2025-03-14T07:25:53.1170247Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:426 in _relative_position_bucket, code: torch.log(relative_position.float() / max_exact) 2025-03-14T07:25:53.1171171Z float_8: "f32[1, 1][1, 1]cuda:0" = relative_position_3.float() 2025-03-14T07:25:53.1171618Z truediv_2: "f32[1, 1][1, 1]cuda:0" = float_8 / 16; float_8 = None 2025-03-14T07:25:53.1172070Z log_1: "f32[1, 1][1, 1]cuda:0" = torch.log(truediv_2); truediv_2 = None 2025-03-14T07:25:53.1172539Z truediv_3: "f32[1, 1][1, 1]cuda:0" = log_1 / 2.0794415416798357; log_1 = None 2025-03-14T07:25:53.1172992Z mul_35: "f32[1, 1][1, 1]cuda:0" = truediv_3 * 16; truediv_3 = None 2025-03-14T07:25:53.1173346Z 2025-03-14T07:25:53.1174014Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:429 in _relative_position_bucket, code: ).to(torch.long) 2025-03-14T07:25:53.1174821Z to_35: "i64[1, 1][1, 1]cuda:0" = mul_35.to(torch.int64); mul_35 = None 2025-03-14T07:25:53.1175180Z 2025-03-14T07:25:53.1175937Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:425 in _relative_position_bucket, code: relative_position_if_large = max_exact + ( 2025-03-14T07:25:53.1176889Z relative_position_if_large_2: "i64[1, 1][1, 1]cuda:0" = 16 + to_35; to_35 = None 2025-03-14T07:25:53.1177314Z 2025-03-14T07:25:53.1178249Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:431 in _relative_position_bucket, code: relative_position_if_large, torch.full_like(relative_position_if_large, num_buckets - 1) 2025-03-14T07:25:53.1179401Z full_like_1: "i64[1, 1][1, 1]cuda:0" = torch.full_like(relative_position_if_large_2, 31) 2025-03-14T07:25:53.1179818Z 2025-03-14T07:25:53.1180657Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:430 in _relative_position_bucket, code: relative_position_if_large = torch.min( 2025-03-14T07:25:53.1182024Z relative_position_if_large_3: "i64[1, 1][1, 1]cuda:0" = torch.min(relative_position_if_large_2, full_like_1); relative_position_if_large_2 = full_like_1 = None 2025-03-14T07:25:53.1182663Z 2025-03-14T07:25:53.1183567Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:434 in _relative_position_bucket, code: relative_buckets += torch.where(is_small, relative_position, relative_position_if_large) 2025-03-14T07:25:53.1184987Z where_1: "i64[1, 1][1, 1]cuda:0" = torch.where(is_small_1, relative_position_3, relative_position_if_large_3); is_small_1 = relative_position_3 = relative_position_if_large_3 = None 2025-03-14T07:25:53.1185804Z relative_buckets_2: "i64[1, 1][1, 1]cuda:0" = 0 + where_1; where_1 = None 2025-03-14T07:25:53.1186195Z 2025-03-14T07:25:53.1187216Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:450 in compute_bias, code: values = self.relative_attention_bias(relative_position_bucket) # shape (query_length, key_length, num_heads) 2025-03-14T07:25:53.1189610Z values_2: "bf16[1, 1, 8][8, 8, 1]cuda:0" = torch.nn.functional.embedding(relative_buckets_2, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_relative_attention_bias_parameters_weight_, None, None, 2.0, False, False); relative_buckets_2 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_relative_attention_bias_parameters_weight_ = None 2025-03-14T07:25:53.1191304Z 2025-03-14T07:25:53.1192169Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:451 in compute_bias, code: values = values.permute([2, 0, 1]).unsqueeze(0) # shape (1, num_heads, query_length, key_length) 2025-03-14T07:25:53.1193227Z permute_1: "bf16[8, 1, 1][1, 8, 8]cuda:0" = values_2.permute([2, 0, 1]); values_2 = None 2025-03-14T07:25:53.1193841Z values_3: "bf16[1, 8, 1, 1][8, 1, 8, 8]cuda:0" = permute_1.unsqueeze(0); permute_1 = None 2025-03-14T07:25:53.1194238Z 2025-03-14T07:25:53.1195065Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:552 in forward, code: position_bias = position_bias + mask # (batch_size, n_heads, seq_length, key_length) 2025-03-14T07:25:53.1196222Z position_bias_1: "bf16[4, 8, 1, 1][8, 1, 8, 8]cuda:0" = values_3 + extended_attention_mask_5; values_3 = extended_attention_mask_5 = None 2025-03-14T07:25:53.1196773Z 2025-03-14T07:25:53.1197427Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1198336Z scores_12 += position_bias_1; scores_13: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_12; scores_12 = None 2025-03-14T07:25:53.1198804Z 2025-03-14T07:25:53.1199609Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1200539Z float_9: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_13.float() 2025-03-14T07:25:53.1201091Z softmax_6: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_9, dim = -1); float_9 = None 2025-03-14T07:25:53.1201773Z attn_weights_12: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_6.type_as(scores_13); softmax_6 = scores_13 = None 2025-03-14T07:25:53.1202260Z 2025-03-14T07:25:53.1202957Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1204028Z attn_weights_13: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_12, p = 0.1, training = False); attn_weights_12 = None 2025-03-14T07:25:53.1204636Z 2025-03-14T07:25:53.1205515Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1206747Z matmul_13: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_13, hidden_states_81); attn_weights_13 = hidden_states_81 = None 2025-03-14T07:25:53.1207324Z 2025-03-14T07:25:53.1208141Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1209231Z transpose_34: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_13.transpose(1, 2); matmul_13 = None 2025-03-14T07:25:53.1209848Z contiguous_6: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_34.contiguous(); transpose_34 = None 2025-03-14T07:25:53.1210476Z attn_output_12: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_6.view(4, -1, 512); contiguous_6 = None 2025-03-14T07:25:53.1210932Z 2025-03-14T07:25:53.1211598Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1213367Z attn_output_13: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_12, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_12 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1214759Z 2025-03-14T07:25:53.1215619Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1216748Z dropout_28: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_13, 0.1, False, False); attn_output_13 = None 2025-03-14T07:25:53.1217553Z hidden_states_82: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_77 + dropout_28; hidden_states_77 = dropout_28 = None 2025-03-14T07:25:53.1218068Z 2025-03-14T07:25:53.1218866Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1219822Z to_36: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_82.to(torch.float32) 2025-03-14T07:25:53.1220298Z pow_15: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_36.pow(2); to_36 = None 2025-03-14T07:25:53.1220812Z variance_14: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_15.mean(-1, keepdim = True); pow_15 = None 2025-03-14T07:25:53.1221235Z 2025-03-14T07:25:53.1222049Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1223019Z add_33: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_14 + 1e-06; variance_14 = None 2025-03-14T07:25:53.1223496Z rsqrt_14: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_33); add_33 = None 2025-03-14T07:25:53.1224045Z hidden_states_83: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_82 * rsqrt_14; rsqrt_14 = None 2025-03-14T07:25:53.1224490Z 2025-03-14T07:25:53.1225219Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1226522Z hidden_states_84: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_83.to(torch.bfloat16); hidden_states_83 = None 2025-03-14T07:25:53.1227032Z 2025-03-14T07:25:53.1227698Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1229338Z normed_hidden_states_7: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_84; l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_84 = None 2025-03-14T07:25:53.1230521Z 2025-03-14T07:25:53.1231375Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1233396Z linear_40: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_7, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_7 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1234767Z 2025-03-14T07:25:53.1235612Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1236818Z view_30: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_40.view(4, -1, 8, 64); linear_40 = None 2025-03-14T07:25:53.1237406Z query_states_7: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_30.transpose(1, 2); view_30 = None 2025-03-14T07:25:53.1237846Z 2025-03-14T07:25:53.1238746Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1240560Z linear_41: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1241861Z 2025-03-14T07:25:53.1242704Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1243778Z view_31: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_41.view(4, -1, 8, 64); linear_41 = None 2025-03-14T07:25:53.1244427Z hidden_states_85: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_31.transpose(1, 2); view_31 = None 2025-03-14T07:25:53.1244888Z 2025-03-14T07:25:53.1245621Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1247435Z linear_42: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1248730Z 2025-03-14T07:25:53.1249577Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1250650Z view_32: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_42.view(4, -1, 8, 64); linear_42 = None 2025-03-14T07:25:53.1251292Z hidden_states_86: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_32.transpose(1, 2); view_32 = None 2025-03-14T07:25:53.1251757Z 2025-03-14T07:25:53.1252446Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1253438Z transpose_38: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_85.transpose(3, 2); hidden_states_85 = None 2025-03-14T07:25:53.1253949Z 2025-03-14T07:25:53.1254572Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1255568Z scores_14: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_7, transpose_38); query_states_7 = transpose_38 = None 2025-03-14T07:25:53.1256142Z 2025-03-14T07:25:53.1256791Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:538 in forward, code: position_bias = torch.zeros( 2025-03-14T07:25:53.1257829Z position_bias_2: "bf16[1, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.zeros((1, 8, 1, 2048), device = device(type='cuda', index=0), dtype = torch.bfloat16) 2025-03-14T07:25:53.1258547Z 2025-03-14T07:25:53.1259372Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:552 in forward, code: position_bias = position_bias + mask # (batch_size, n_heads, seq_length, key_length) 2025-03-14T07:25:53.1260681Z position_bias_3: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = position_bias_2 + encoder_extended_attention_mask_2; position_bias_2 = encoder_extended_attention_mask_2 = None 2025-03-14T07:25:53.1261383Z 2025-03-14T07:25:53.1262115Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1263066Z scores_14 += position_bias_3; scores_15: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_14; scores_14 = None 2025-03-14T07:25:53.1263572Z 2025-03-14T07:25:53.1264360Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1265317Z float_10: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_15.float() 2025-03-14T07:25:53.1265958Z softmax_7: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_10, dim = -1); float_10 = None 2025-03-14T07:25:53.1266846Z attn_weights_14: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_7.type_as(scores_15); softmax_7 = scores_15 = None 2025-03-14T07:25:53.1267373Z 2025-03-14T07:25:53.1268052Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1269169Z attn_weights_15: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_14, p = 0.1, training = False); attn_weights_14 = None 2025-03-14T07:25:53.1269818Z 2025-03-14T07:25:53.1270698Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1271932Z matmul_15: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_15, hidden_states_86); attn_weights_15 = hidden_states_86 = None 2025-03-14T07:25:53.1272496Z 2025-03-14T07:25:53.1273320Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1274359Z transpose_39: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_15.transpose(1, 2); matmul_15 = None 2025-03-14T07:25:53.1274971Z contiguous_7: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_39.contiguous(); transpose_39 = None 2025-03-14T07:25:53.1275605Z attn_output_14: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_7.view(4, -1, 512); contiguous_7 = None 2025-03-14T07:25:53.1276057Z 2025-03-14T07:25:53.1276722Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1278504Z attn_output_15: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_14, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_14 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1279826Z 2025-03-14T07:25:53.1280596Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1281806Z dropout_30: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_15, 0.1, False, False); attn_output_15 = None 2025-03-14T07:25:53.1282608Z layer_output: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_82 + dropout_30; hidden_states_82 = dropout_30 = None 2025-03-14T07:25:53.1283117Z 2025-03-14T07:25:53.1283987Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1284941Z to_38: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output.to(torch.float32) 2025-03-14T07:25:53.1285410Z pow_16: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_38.pow(2); to_38 = None 2025-03-14T07:25:53.1285923Z variance_15: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_16.mean(-1, keepdim = True); pow_16 = None 2025-03-14T07:25:53.1286347Z 2025-03-14T07:25:53.1287158Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1288120Z add_36: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_15 + 1e-06; variance_15 = None 2025-03-14T07:25:53.1288593Z rsqrt_15: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_36); add_36 = None 2025-03-14T07:25:53.1289136Z hidden_states_87: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output * rsqrt_15; rsqrt_15 = None 2025-03-14T07:25:53.1289568Z 2025-03-14T07:25:53.1290297Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1291307Z hidden_states_88: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_87.to(torch.bfloat16); hidden_states_87 = None 2025-03-14T07:25:53.1291810Z 2025-03-14T07:25:53.1292476Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1294115Z forwarded_states_6: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_88; l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_88 = None 2025-03-14T07:25:53.1295290Z 2025-03-14T07:25:53.1295970Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1297823Z hidden_states_89: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_6, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_6 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1299212Z 2025-03-14T07:25:53.1299894Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1300959Z hidden_states_90: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_89, inplace = False); hidden_states_89 = None 2025-03-14T07:25:53.1301558Z 2025-03-14T07:25:53.1302253Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1303349Z hidden_states_91: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_90, 0.1, False, False); hidden_states_90 = None 2025-03-14T07:25:53.1304057Z 2025-03-14T07:25:53.1304727Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1306677Z hidden_states_92: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_91, l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_91 = l_self_modules_decoder_modules_block_modules_0_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1308019Z 2025-03-14T07:25:53.1308785Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1309915Z dropout_32: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_92, 0.1, False, False); hidden_states_92 = None 2025-03-14T07:25:53.1310721Z hidden_states_93: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output + dropout_32; layer_output = dropout_32 = None 2025-03-14T07:25:53.1311216Z 2025-03-14T07:25:53.1312007Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1312969Z to_40: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_93.to(torch.float32) 2025-03-14T07:25:53.1313449Z pow_17: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_40.pow(2); to_40 = None 2025-03-14T07:25:53.1313959Z variance_16: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_17.mean(-1, keepdim = True); pow_17 = None 2025-03-14T07:25:53.1314381Z 2025-03-14T07:25:53.1315197Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1316164Z add_38: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_16 + 1e-06; variance_16 = None 2025-03-14T07:25:53.1316641Z rsqrt_16: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_38); add_38 = None 2025-03-14T07:25:53.1317188Z hidden_states_94: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_93 * rsqrt_16; rsqrt_16 = None 2025-03-14T07:25:53.1317633Z 2025-03-14T07:25:53.1318387Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1319429Z hidden_states_95: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_94.to(torch.bfloat16); hidden_states_94 = None 2025-03-14T07:25:53.1319931Z 2025-03-14T07:25:53.1320592Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1322231Z normed_hidden_states_8: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_95; l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_95 = None 2025-03-14T07:25:53.1323410Z 2025-03-14T07:25:53.1324264Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1326482Z linear_46: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_8, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1327963Z 2025-03-14T07:25:53.1328800Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1329839Z view_34: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_46.view(4, -1, 8, 64); linear_46 = None 2025-03-14T07:25:53.1330567Z query_states_8: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_34.transpose(1, 2); view_34 = None 2025-03-14T07:25:53.1331001Z 2025-03-14T07:25:53.1331715Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1333500Z linear_47: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_8, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1334769Z 2025-03-14T07:25:53.1335616Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1336653Z view_35: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_47.view(4, -1, 8, 64); linear_47 = None 2025-03-14T07:25:53.1337238Z hidden_states_96: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_35.transpose(1, 2); view_35 = None 2025-03-14T07:25:53.1337675Z 2025-03-14T07:25:53.1338437Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1340311Z linear_48: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_8, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_8 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1341659Z 2025-03-14T07:25:53.1342499Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1343539Z view_36: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_48.view(4, -1, 8, 64); linear_48 = None 2025-03-14T07:25:53.1344122Z hidden_states_97: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_36.transpose(1, 2); view_36 = None 2025-03-14T07:25:53.1344564Z 2025-03-14T07:25:53.1345263Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1346221Z transpose_43: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_96.transpose(3, 2); hidden_states_96 = None 2025-03-14T07:25:53.1353879Z 2025-03-14T07:25:53.1354577Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1355527Z scores_16: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_8, transpose_43); query_states_8 = transpose_43 = None 2025-03-14T07:25:53.1356053Z 2025-03-14T07:25:53.1356706Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1357742Z scores_16 += position_bias_1; scores_17: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_16; scores_16 = None 2025-03-14T07:25:53.1358210Z 2025-03-14T07:25:53.1358998Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1359926Z float_11: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_17.float() 2025-03-14T07:25:53.1360573Z softmax_8: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_11, dim = -1); float_11 = None 2025-03-14T07:25:53.1361257Z attn_weights_16: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_8.type_as(scores_17); softmax_8 = scores_17 = None 2025-03-14T07:25:53.1361735Z 2025-03-14T07:25:53.1362422Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1363493Z attn_weights_17: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_16, p = 0.1, training = False); attn_weights_16 = None 2025-03-14T07:25:53.1364084Z 2025-03-14T07:25:53.1364962Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1366201Z matmul_17: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_17, hidden_states_97); attn_weights_17 = hidden_states_97 = None 2025-03-14T07:25:53.1366769Z 2025-03-14T07:25:53.1367592Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1368697Z transpose_44: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_17.transpose(1, 2); matmul_17 = None 2025-03-14T07:25:53.1369314Z contiguous_8: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_44.contiguous(); transpose_44 = None 2025-03-14T07:25:53.1369946Z attn_output_16: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_8.view(4, -1, 512); contiguous_8 = None 2025-03-14T07:25:53.1370405Z 2025-03-14T07:25:53.1371081Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1372847Z attn_output_17: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_16, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_16 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1374161Z 2025-03-14T07:25:53.1374936Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1376060Z dropout_34: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_17, 0.1, False, False); attn_output_17 = None 2025-03-14T07:25:53.1376870Z hidden_states_98: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_93 + dropout_34; hidden_states_93 = dropout_34 = None 2025-03-14T07:25:53.1377383Z 2025-03-14T07:25:53.1378175Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1379136Z to_42: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_98.to(torch.float32) 2025-03-14T07:25:53.1379702Z pow_18: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_42.pow(2); to_42 = None 2025-03-14T07:25:53.1380210Z variance_17: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_18.mean(-1, keepdim = True); pow_18 = None 2025-03-14T07:25:53.1380629Z 2025-03-14T07:25:53.1381441Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1382482Z add_40: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_17 + 1e-06; variance_17 = None 2025-03-14T07:25:53.1382963Z rsqrt_17: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_40); add_40 = None 2025-03-14T07:25:53.1383511Z hidden_states_99: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_98 * rsqrt_17; rsqrt_17 = None 2025-03-14T07:25:53.1383953Z 2025-03-14T07:25:53.1384682Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1385706Z hidden_states_100: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_99.to(torch.bfloat16); hidden_states_99 = None 2025-03-14T07:25:53.1386212Z 2025-03-14T07:25:53.1386969Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1388671Z normed_hidden_states_9: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_100; l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_100 = None 2025-03-14T07:25:53.1389856Z 2025-03-14T07:25:53.1390708Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1392718Z linear_50: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_9, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_9 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1394086Z 2025-03-14T07:25:53.1394923Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1395964Z view_38: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_50.view(4, -1, 8, 64); linear_50 = None 2025-03-14T07:25:53.1396545Z query_states_9: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_38.transpose(1, 2); view_38 = None 2025-03-14T07:25:53.1396974Z 2025-03-14T07:25:53.1397693Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1399500Z linear_51: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1400792Z 2025-03-14T07:25:53.1401626Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1402782Z view_39: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_51.view(4, -1, 8, 64); linear_51 = None 2025-03-14T07:25:53.1403433Z hidden_states_101: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_39.transpose(1, 2); view_39 = None 2025-03-14T07:25:53.1403898Z 2025-03-14T07:25:53.1404621Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1406498Z linear_52: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1407791Z 2025-03-14T07:25:53.1408627Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1409694Z view_40: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_52.view(4, -1, 8, 64); linear_52 = None 2025-03-14T07:25:53.1410332Z hidden_states_102: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_40.transpose(1, 2); view_40 = None 2025-03-14T07:25:53.1410796Z 2025-03-14T07:25:53.1411497Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1412488Z transpose_48: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_101.transpose(3, 2); hidden_states_101 = None 2025-03-14T07:25:53.1413005Z 2025-03-14T07:25:53.1413633Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1414621Z scores_18: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_9, transpose_48); query_states_9 = transpose_48 = None 2025-03-14T07:25:53.1415197Z 2025-03-14T07:25:53.1415854Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1416807Z scores_18 += position_bias_3; scores_19: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_18; scores_18 = None 2025-03-14T07:25:53.1417313Z 2025-03-14T07:25:53.1418108Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1419068Z float_12: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_19.float() 2025-03-14T07:25:53.1419706Z softmax_9: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_12, dim = -1); float_12 = None 2025-03-14T07:25:53.1420485Z attn_weights_18: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_9.type_as(scores_19); softmax_9 = scores_19 = None 2025-03-14T07:25:53.1421007Z 2025-03-14T07:25:53.1421694Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1422811Z attn_weights_19: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_18, p = 0.1, training = False); attn_weights_18 = None 2025-03-14T07:25:53.1423459Z 2025-03-14T07:25:53.1424334Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1425649Z matmul_19: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_19, hidden_states_102); attn_weights_19 = hidden_states_102 = None 2025-03-14T07:25:53.1426578Z 2025-03-14T07:25:53.1427400Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1428638Z transpose_49: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_19.transpose(1, 2); matmul_19 = None 2025-03-14T07:25:53.1429345Z contiguous_9: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_49.contiguous(); transpose_49 = None 2025-03-14T07:25:53.1430077Z attn_output_18: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_9.view(4, -1, 512); contiguous_9 = None 2025-03-14T07:25:53.1430611Z 2025-03-14T07:25:53.1431385Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1433592Z attn_output_19: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_18, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_18 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1435224Z 2025-03-14T07:25:53.1436126Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1437468Z dropout_36: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_19, 0.1, False, False); attn_output_19 = None 2025-03-14T07:25:53.1438425Z layer_output_1: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_98 + dropout_36; hidden_states_98 = dropout_36 = None 2025-03-14T07:25:53.1439038Z 2025-03-14T07:25:53.1439972Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1441096Z to_44: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_1.to(torch.float32) 2025-03-14T07:25:53.1441629Z pow_19: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_44.pow(2); to_44 = None 2025-03-14T07:25:53.1442204Z variance_18: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_19.mean(-1, keepdim = True); pow_19 = None 2025-03-14T07:25:53.1442676Z 2025-03-14T07:25:53.1443638Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1444791Z add_42: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_18 + 1e-06; variance_18 = None 2025-03-14T07:25:53.1445327Z rsqrt_18: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_42); add_42 = None 2025-03-14T07:25:53.1445951Z hidden_states_103: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_1 * rsqrt_18; rsqrt_18 = None 2025-03-14T07:25:53.1446461Z 2025-03-14T07:25:53.1447314Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1448538Z hidden_states_104: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_103.to(torch.bfloat16); hidden_states_103 = None 2025-03-14T07:25:53.1449131Z 2025-03-14T07:25:53.1449905Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1451723Z forwarded_states_7: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_104; l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_104 = None 2025-03-14T07:25:53.1452899Z 2025-03-14T07:25:53.1453658Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1455527Z hidden_states_105: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_7, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_7 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1456925Z 2025-03-14T07:25:53.1457610Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1458689Z hidden_states_106: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_105, inplace = False); hidden_states_105 = None 2025-03-14T07:25:53.1459307Z 2025-03-14T07:25:53.1460012Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1461131Z hidden_states_107: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_106, 0.1, False, False); hidden_states_106 = None 2025-03-14T07:25:53.1461760Z 2025-03-14T07:25:53.1462433Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1464262Z hidden_states_108: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_107, l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_107 = l_self_modules_decoder_modules_block_modules_1_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1465616Z 2025-03-14T07:25:53.1466387Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1467591Z dropout_38: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_108, 0.1, False, False); hidden_states_108 = None 2025-03-14T07:25:53.1468415Z hidden_states_109: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output_1 + dropout_38; layer_output_1 = dropout_38 = None 2025-03-14T07:25:53.1468922Z 2025-03-14T07:25:53.1469723Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1470680Z to_46: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_109.to(torch.float32) 2025-03-14T07:25:53.1471162Z pow_20: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_46.pow(2); to_46 = None 2025-03-14T07:25:53.1471672Z variance_19: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_20.mean(-1, keepdim = True); pow_20 = None 2025-03-14T07:25:53.1472082Z 2025-03-14T07:25:53.1472895Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1473956Z add_44: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_19 + 1e-06; variance_19 = None 2025-03-14T07:25:53.1474428Z rsqrt_19: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_44); add_44 = None 2025-03-14T07:25:53.1474983Z hidden_states_110: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_109 * rsqrt_19; rsqrt_19 = None 2025-03-14T07:25:53.1475433Z 2025-03-14T07:25:53.1476241Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1477265Z hidden_states_111: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_110.to(torch.bfloat16); hidden_states_110 = None 2025-03-14T07:25:53.1477775Z 2025-03-14T07:25:53.1478439Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1480090Z normed_hidden_states_10: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_111; l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_111 = None 2025-03-14T07:25:53.1481278Z 2025-03-14T07:25:53.1482127Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1484044Z linear_56: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_10, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1485321Z 2025-03-14T07:25:53.1486154Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1487192Z view_42: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_56.view(4, -1, 8, 64); linear_56 = None 2025-03-14T07:25:53.1487772Z query_states_10: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_42.transpose(1, 2); view_42 = None 2025-03-14T07:25:53.1488210Z 2025-03-14T07:25:53.1488924Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1490700Z linear_57: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_10, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1491972Z 2025-03-14T07:25:53.1492809Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1493861Z view_43: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_57.view(4, -1, 8, 64); linear_57 = None 2025-03-14T07:25:53.1494451Z hidden_states_112: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_43.transpose(1, 2); view_43 = None 2025-03-14T07:25:53.1494894Z 2025-03-14T07:25:53.1495604Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1497546Z linear_58: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_10, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_10 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1498900Z 2025-03-14T07:25:53.1499913Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1500954Z view_44: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_58.view(4, -1, 8, 64); linear_58 = None 2025-03-14T07:25:53.1501554Z hidden_states_113: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_44.transpose(1, 2); view_44 = None 2025-03-14T07:25:53.1502002Z 2025-03-14T07:25:53.1502685Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1503657Z transpose_53: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_112.transpose(3, 2); hidden_states_112 = None 2025-03-14T07:25:53.1504138Z 2025-03-14T07:25:53.1504770Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1505719Z scores_20: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_10, transpose_53); query_states_10 = transpose_53 = None 2025-03-14T07:25:53.1506248Z 2025-03-14T07:25:53.1506958Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1507862Z scores_20 += position_bias_1; scores_21: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_20; scores_20 = None 2025-03-14T07:25:53.1508337Z 2025-03-14T07:25:53.1509131Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1510056Z float_13: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_21.float() 2025-03-14T07:25:53.1510633Z softmax_10: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_13, dim = -1); float_13 = None 2025-03-14T07:25:53.1511318Z attn_weights_20: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_10.type_as(scores_21); softmax_10 = scores_21 = None 2025-03-14T07:25:53.1511796Z 2025-03-14T07:25:53.1512474Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1513541Z attn_weights_21: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_20, p = 0.1, training = False); attn_weights_20 = None 2025-03-14T07:25:53.1514129Z 2025-03-14T07:25:53.1515009Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1516245Z matmul_21: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_21, hidden_states_113); attn_weights_21 = hidden_states_113 = None 2025-03-14T07:25:53.1516813Z 2025-03-14T07:25:53.1517636Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1518677Z transpose_54: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_21.transpose(1, 2); matmul_21 = None 2025-03-14T07:25:53.1519409Z contiguous_10: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_54.contiguous(); transpose_54 = None 2025-03-14T07:25:53.1520042Z attn_output_20: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_10.view(4, -1, 512); contiguous_10 = None 2025-03-14T07:25:53.1520507Z 2025-03-14T07:25:53.1521168Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1523010Z attn_output_21: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_20, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_20 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1524323Z 2025-03-14T07:25:53.1525103Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1526568Z dropout_40: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_21, 0.1, False, False); attn_output_21 = None 2025-03-14T07:25:53.1527383Z hidden_states_114: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_109 + dropout_40; hidden_states_109 = dropout_40 = None 2025-03-14T07:25:53.1527901Z 2025-03-14T07:25:53.1528718Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1529705Z to_48: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_114.to(torch.float32) 2025-03-14T07:25:53.1530177Z pow_21: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_48.pow(2); to_48 = None 2025-03-14T07:25:53.1530693Z variance_20: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_21.mean(-1, keepdim = True); pow_21 = None 2025-03-14T07:25:53.1531107Z 2025-03-14T07:25:53.1531924Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1532886Z add_46: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_20 + 1e-06; variance_20 = None 2025-03-14T07:25:53.1533367Z rsqrt_20: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_46); add_46 = None 2025-03-14T07:25:53.1533918Z hidden_states_115: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_114 * rsqrt_20; rsqrt_20 = None 2025-03-14T07:25:53.1534372Z 2025-03-14T07:25:53.1535104Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1536133Z hidden_states_116: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_115.to(torch.bfloat16); hidden_states_115 = None 2025-03-14T07:25:53.1536642Z 2025-03-14T07:25:53.1537309Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1538956Z normed_hidden_states_11: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_116; l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_116 = None 2025-03-14T07:25:53.1540144Z 2025-03-14T07:25:53.1540995Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1543141Z linear_60: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_11, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_11 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1544515Z 2025-03-14T07:25:53.1545473Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1546571Z view_46: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_60.view(4, -1, 8, 64); linear_60 = None 2025-03-14T07:25:53.1547155Z query_states_11: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_46.transpose(1, 2); view_46 = None 2025-03-14T07:25:53.1547600Z 2025-03-14T07:25:53.1548331Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1550154Z linear_61: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1551442Z 2025-03-14T07:25:53.1552274Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1553343Z view_47: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_61.view(4, -1, 8, 64); linear_61 = None 2025-03-14T07:25:53.1553991Z hidden_states_117: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_47.transpose(1, 2); view_47 = None 2025-03-14T07:25:53.1554457Z 2025-03-14T07:25:53.1555178Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1556988Z linear_62: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1558276Z 2025-03-14T07:25:53.1559158Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1560225Z view_48: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_62.view(4, -1, 8, 64); linear_62 = None 2025-03-14T07:25:53.1560866Z hidden_states_118: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_48.transpose(1, 2); view_48 = None 2025-03-14T07:25:53.1561329Z 2025-03-14T07:25:53.1562023Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1563012Z transpose_58: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_117.transpose(3, 2); hidden_states_117 = None 2025-03-14T07:25:53.1563522Z 2025-03-14T07:25:53.1564148Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1565231Z scores_22: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_11, transpose_58); query_states_11 = transpose_58 = None 2025-03-14T07:25:53.1565805Z 2025-03-14T07:25:53.1566457Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1567478Z scores_22 += position_bias_3; scores_23: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_22; scores_22 = None 2025-03-14T07:25:53.1567983Z 2025-03-14T07:25:53.1568769Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1569719Z float_14: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_23.float() 2025-03-14T07:25:53.1570365Z softmax_11: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_14, dim = -1); float_14 = None 2025-03-14T07:25:53.1571143Z attn_weights_22: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_11.type_as(scores_23); softmax_11 = scores_23 = None 2025-03-14T07:25:53.1571668Z 2025-03-14T07:25:53.1572353Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1573476Z attn_weights_23: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_22, p = 0.1, training = False); attn_weights_22 = None 2025-03-14T07:25:53.1574128Z 2025-03-14T07:25:53.1575008Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1576246Z matmul_23: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_23, hidden_states_118); attn_weights_23 = hidden_states_118 = None 2025-03-14T07:25:53.1576817Z 2025-03-14T07:25:53.1577636Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1578678Z transpose_59: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_23.transpose(1, 2); matmul_23 = None 2025-03-14T07:25:53.1579294Z contiguous_11: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_59.contiguous(); transpose_59 = None 2025-03-14T07:25:53.1579934Z attn_output_22: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_11.view(4, -1, 512); contiguous_11 = None 2025-03-14T07:25:53.1580398Z 2025-03-14T07:25:53.1581061Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1582846Z attn_output_23: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_22, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_22 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1584166Z 2025-03-14T07:25:53.1584939Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1586062Z dropout_42: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_23, 0.1, False, False); attn_output_23 = None 2025-03-14T07:25:53.1586927Z layer_output_2: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_114 + dropout_42; hidden_states_114 = dropout_42 = None 2025-03-14T07:25:53.1587531Z 2025-03-14T07:25:53.1588322Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1589273Z to_50: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_2.to(torch.float32) 2025-03-14T07:25:53.1589741Z pow_22: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_50.pow(2); to_50 = None 2025-03-14T07:25:53.1590338Z variance_21: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_22.mean(-1, keepdim = True); pow_22 = None 2025-03-14T07:25:53.1590754Z 2025-03-14T07:25:53.1591569Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1592537Z add_48: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_21 + 1e-06; variance_21 = None 2025-03-14T07:25:53.1593017Z rsqrt_21: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_48); add_48 = None 2025-03-14T07:25:53.1593561Z hidden_states_119: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_2 * rsqrt_21; rsqrt_21 = None 2025-03-14T07:25:53.1594004Z 2025-03-14T07:25:53.1594738Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1595764Z hidden_states_120: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_119.to(torch.bfloat16); hidden_states_119 = None 2025-03-14T07:25:53.1596274Z 2025-03-14T07:25:53.1596945Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1598591Z forwarded_states_8: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_120; l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_120 = None 2025-03-14T07:25:53.1599768Z 2025-03-14T07:25:53.1600442Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1602305Z hidden_states_121: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_8, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_8 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1603698Z 2025-03-14T07:25:53.1604384Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1605461Z hidden_states_122: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_121, inplace = False); hidden_states_121 = None 2025-03-14T07:25:53.1606073Z 2025-03-14T07:25:53.1606782Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1607894Z hidden_states_123: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_122, 0.1, False, False); hidden_states_122 = None 2025-03-14T07:25:53.1608517Z 2025-03-14T07:25:53.1609195Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1611139Z hidden_states_124: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_123, l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_123 = l_self_modules_decoder_modules_block_modules_2_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1612493Z 2025-03-14T07:25:53.1613331Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1614466Z dropout_44: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_124, 0.1, False, False); hidden_states_124 = None 2025-03-14T07:25:53.1615292Z hidden_states_125: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output_2 + dropout_44; layer_output_2 = dropout_44 = None 2025-03-14T07:25:53.1615809Z 2025-03-14T07:25:53.1616605Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1617560Z to_52: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_125.to(torch.float32) 2025-03-14T07:25:53.1618035Z pow_23: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_52.pow(2); to_52 = None 2025-03-14T07:25:53.1618561Z variance_22: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_23.mean(-1, keepdim = True); pow_23 = None 2025-03-14T07:25:53.1618984Z 2025-03-14T07:25:53.1619803Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1620764Z add_50: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_22 + 1e-06; variance_22 = None 2025-03-14T07:25:53.1621242Z rsqrt_22: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_50); add_50 = None 2025-03-14T07:25:53.1621801Z hidden_states_126: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_125 * rsqrt_22; rsqrt_22 = None 2025-03-14T07:25:53.1622258Z 2025-03-14T07:25:53.1622999Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1624024Z hidden_states_127: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_126.to(torch.bfloat16); hidden_states_126 = None 2025-03-14T07:25:53.1624533Z 2025-03-14T07:25:53.1625199Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1627166Z normed_hidden_states_12: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_127; l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_127 = None 2025-03-14T07:25:53.1628359Z 2025-03-14T07:25:53.1629262Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1631175Z linear_66: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_12, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1632444Z 2025-03-14T07:25:53.1633410Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1634449Z view_50: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_66.view(4, -1, 8, 64); linear_66 = None 2025-03-14T07:25:53.1635033Z query_states_12: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_50.transpose(1, 2); view_50 = None 2025-03-14T07:25:53.1635467Z 2025-03-14T07:25:53.1636294Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1638079Z linear_67: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_12, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1639354Z 2025-03-14T07:25:53.1640187Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1641225Z view_51: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_67.view(4, -1, 8, 64); linear_67 = None 2025-03-14T07:25:53.1641815Z hidden_states_128: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_51.transpose(1, 2); view_51 = None 2025-03-14T07:25:53.1642254Z 2025-03-14T07:25:53.1642965Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1644816Z linear_68: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_12, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_12 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1646171Z 2025-03-14T07:25:53.1647014Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1648055Z view_52: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_68.view(4, -1, 8, 64); linear_68 = None 2025-03-14T07:25:53.1648644Z hidden_states_129: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_52.transpose(1, 2); view_52 = None 2025-03-14T07:25:53.1649084Z 2025-03-14T07:25:53.1649779Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1650750Z transpose_63: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_128.transpose(3, 2); hidden_states_128 = None 2025-03-14T07:25:53.1658712Z 2025-03-14T07:25:53.1659378Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1660348Z scores_24: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_12, transpose_63); query_states_12 = transpose_63 = None 2025-03-14T07:25:53.1660880Z 2025-03-14T07:25:53.1661542Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1662455Z scores_24 += position_bias_1; scores_25: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_24; scores_24 = None 2025-03-14T07:25:53.1663039Z 2025-03-14T07:25:53.1663838Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1664766Z float_15: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_25.float() 2025-03-14T07:25:53.1665330Z softmax_12: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_15, dim = -1); float_15 = None 2025-03-14T07:25:53.1666093Z attn_weights_24: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_12.type_as(scores_25); softmax_12 = scores_25 = None 2025-03-14T07:25:53.1666651Z 2025-03-14T07:25:53.1667335Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1668403Z attn_weights_25: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_24, p = 0.1, training = False); attn_weights_24 = None 2025-03-14T07:25:53.1669007Z 2025-03-14T07:25:53.1669888Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1671133Z matmul_25: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_25, hidden_states_129); attn_weights_25 = hidden_states_129 = None 2025-03-14T07:25:53.1671711Z 2025-03-14T07:25:53.1672545Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1673595Z transpose_64: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_25.transpose(1, 2); matmul_25 = None 2025-03-14T07:25:53.1674215Z contiguous_12: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_64.contiguous(); transpose_64 = None 2025-03-14T07:25:53.1674858Z attn_output_24: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_12.view(4, -1, 512); contiguous_12 = None 2025-03-14T07:25:53.1675333Z 2025-03-14T07:25:53.1676005Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1677771Z attn_output_25: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_24, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_24 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1679134Z 2025-03-14T07:25:53.1679793Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1680182Z dropout_46: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_25, 0.1, False, False); attn_output_25 = None 2025-03-14T07:25:53.1680524Z hidden_states_130: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_125 + dropout_46; hidden_states_125 = dropout_46 = None 2025-03-14T07:25:53.1680613Z 2025-03-14T07:25:53.1681292Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1681491Z to_54: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_130.to(torch.float32) 2025-03-14T07:25:53.1681675Z pow_24: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_54.pow(2); to_54 = None 2025-03-14T07:25:53.1681905Z variance_23: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_24.mean(-1, keepdim = True); pow_24 = None 2025-03-14T07:25:53.1682087Z 2025-03-14T07:25:53.1682772Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1682962Z add_52: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_23 + 1e-06; variance_23 = None 2025-03-14T07:25:53.1683147Z rsqrt_23: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_52); add_52 = None 2025-03-14T07:25:53.1683491Z hidden_states_131: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_130 * rsqrt_23; rsqrt_23 = None 2025-03-14T07:25:53.1683580Z 2025-03-14T07:25:53.1684190Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1684515Z hidden_states_132: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_131.to(torch.bfloat16); hidden_states_131 = None 2025-03-14T07:25:53.1684621Z 2025-03-14T07:25:53.1685158Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1686175Z normed_hidden_states_13: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_132; l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_132 = None 2025-03-14T07:25:53.1686270Z 2025-03-14T07:25:53.1686996Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1688198Z linear_70: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_13, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_13 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1688289Z 2025-03-14T07:25:53.1689014Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1689250Z view_54: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_70.view(4, -1, 8, 64); linear_70 = None 2025-03-14T07:25:53.1689499Z query_states_13: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_54.transpose(1, 2); view_54 = None 2025-03-14T07:25:53.1689588Z 2025-03-14T07:25:53.1690201Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1691546Z linear_71: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1691654Z 2025-03-14T07:25:53.1692376Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1692638Z view_55: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_71.view(4, -1, 8, 64); linear_71 = None 2025-03-14T07:25:53.1693012Z hidden_states_133: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_55.transpose(1, 2); view_55 = None 2025-03-14T07:25:53.1693101Z 2025-03-14T07:25:53.1693708Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1694893Z linear_72: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1694991Z 2025-03-14T07:25:53.1695710Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1695986Z view_56: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_72.view(4, -1, 8, 64); linear_72 = None 2025-03-14T07:25:53.1696267Z hidden_states_134: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_56.transpose(1, 2); view_56 = None 2025-03-14T07:25:53.1696366Z 2025-03-14T07:25:53.1696933Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1697271Z transpose_68: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_133.transpose(3, 2); hidden_states_133 = None 2025-03-14T07:25:53.1697362Z 2025-03-14T07:25:53.1697871Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1698263Z scores_26: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_13, transpose_68); query_states_13 = transpose_68 = None 2025-03-14T07:25:53.1698366Z 2025-03-14T07:25:53.1698890Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1699213Z scores_26 += position_bias_3; scores_27: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_26; scores_26 = None 2025-03-14T07:25:53.1699307Z 2025-03-14T07:25:53.1699978Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1700180Z float_16: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_27.float() 2025-03-14T07:25:53.1700521Z softmax_13: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_16, dim = -1); float_16 = None 2025-03-14T07:25:53.1700862Z attn_weights_26: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_13.type_as(scores_27); softmax_13 = scores_27 = None 2025-03-14T07:25:53.1700952Z 2025-03-14T07:25:53.1701515Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1701982Z attn_weights_27: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_26, p = 0.1, training = False); attn_weights_26 = None 2025-03-14T07:25:53.1702075Z 2025-03-14T07:25:53.1702823Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1703212Z matmul_27: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_27, hidden_states_134); attn_weights_27 = hidden_states_134 = None 2025-03-14T07:25:53.1703380Z 2025-03-14T07:25:53.1704083Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1704337Z transpose_69: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_27.transpose(1, 2); matmul_27 = None 2025-03-14T07:25:53.1704706Z contiguous_13: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_69.contiguous(); transpose_69 = None 2025-03-14T07:25:53.1704981Z attn_output_26: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_13.view(4, -1, 512); contiguous_13 = None 2025-03-14T07:25:53.1705075Z 2025-03-14T07:25:53.1705614Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1706822Z attn_output_27: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_26, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_26 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1706917Z 2025-03-14T07:25:53.1707564Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1707949Z dropout_48: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_27, 0.1, False, False); attn_output_27 = None 2025-03-14T07:25:53.1708272Z layer_output_3: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_130 + dropout_48; hidden_states_130 = dropout_48 = None 2025-03-14T07:25:53.1708373Z 2025-03-14T07:25:53.1709038Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1709234Z to_56: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_3.to(torch.float32) 2025-03-14T07:25:53.1709410Z pow_25: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_56.pow(2); to_56 = None 2025-03-14T07:25:53.1709652Z variance_24: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_25.mean(-1, keepdim = True); pow_25 = None 2025-03-14T07:25:53.1709739Z 2025-03-14T07:25:53.1710430Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1710612Z add_54: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_24 + 1e-06; variance_24 = None 2025-03-14T07:25:53.1710809Z rsqrt_24: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_54); add_54 = None 2025-03-14T07:25:53.1711066Z hidden_states_135: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_3 * rsqrt_24; rsqrt_24 = None 2025-03-14T07:25:53.1711166Z 2025-03-14T07:25:53.1711773Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1712103Z hidden_states_136: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_135.to(torch.bfloat16); hidden_states_135 = None 2025-03-14T07:25:53.1712194Z 2025-03-14T07:25:53.1712736Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1713736Z forwarded_states_9: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_136; l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_136 = None 2025-03-14T07:25:53.1713918Z 2025-03-14T07:25:53.1714475Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1715769Z hidden_states_137: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_9, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_9 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1715868Z 2025-03-14T07:25:53.1716422Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1716860Z hidden_states_138: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_137, inplace = False); hidden_states_137 = None 2025-03-14T07:25:53.1716950Z 2025-03-14T07:25:53.1717523Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1717967Z hidden_states_139: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_138, 0.1, False, False); hidden_states_138 = None 2025-03-14T07:25:53.1718065Z 2025-03-14T07:25:53.1718609Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1719789Z hidden_states_140: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_139, l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_139 = l_self_modules_decoder_modules_block_modules_3_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1719895Z 2025-03-14T07:25:53.1720541Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1720952Z dropout_50: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_140, 0.1, False, False); hidden_states_140 = None 2025-03-14T07:25:53.1721275Z hidden_states_141: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output_3 + dropout_50; layer_output_3 = dropout_50 = None 2025-03-14T07:25:53.1721369Z 2025-03-14T07:25:53.1722040Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1722250Z to_58: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_141.to(torch.float32) 2025-03-14T07:25:53.1722425Z pow_26: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_58.pow(2); to_58 = None 2025-03-14T07:25:53.1722667Z variance_25: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_26.mean(-1, keepdim = True); pow_26 = None 2025-03-14T07:25:53.1722758Z 2025-03-14T07:25:53.1723459Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1723641Z add_56: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_25 + 1e-06; variance_25 = None 2025-03-14T07:25:53.1723836Z rsqrt_25: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_56); add_56 = None 2025-03-14T07:25:53.1724179Z hidden_states_142: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_141 * rsqrt_25; rsqrt_25 = None 2025-03-14T07:25:53.1724276Z 2025-03-14T07:25:53.1724881Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1725206Z hidden_states_143: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_142.to(torch.bfloat16); hidden_states_142 = None 2025-03-14T07:25:53.1725368Z 2025-03-14T07:25:53.1725912Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1727075Z normed_hidden_states_14: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_143; l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_143 = None 2025-03-14T07:25:53.1727177Z 2025-03-14T07:25:53.1727902Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1729017Z linear_76: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_14, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1729110Z 2025-03-14T07:25:53.1729826Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1730075Z view_58: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_76.view(4, -1, 8, 64); linear_76 = None 2025-03-14T07:25:53.1730318Z query_states_14: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_58.transpose(1, 2); view_58 = None 2025-03-14T07:25:53.1730409Z 2025-03-14T07:25:53.1731007Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1732109Z linear_77: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_14, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1732205Z 2025-03-14T07:25:53.1732927Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1733163Z view_59: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_77.view(4, -1, 8, 64); linear_77 = None 2025-03-14T07:25:53.1733425Z hidden_states_144: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_59.transpose(1, 2); view_59 = None 2025-03-14T07:25:53.1733525Z 2025-03-14T07:25:53.1734117Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1735298Z linear_78: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_14, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_14 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1735527Z 2025-03-14T07:25:53.1736248Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1736760Z view_60: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_78.view(4, -1, 8, 64); linear_78 = None 2025-03-14T07:25:53.1737024Z hidden_states_145: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_60.transpose(1, 2); view_60 = None 2025-03-14T07:25:53.1737114Z 2025-03-14T07:25:53.1737685Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1737996Z transpose_73: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_144.transpose(3, 2); hidden_states_144 = None 2025-03-14T07:25:53.1738089Z 2025-03-14T07:25:53.1738594Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1738944Z scores_28: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_14, transpose_73); query_states_14 = transpose_73 = None 2025-03-14T07:25:53.1739032Z 2025-03-14T07:25:53.1739573Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1739857Z scores_28 += position_bias_1; scores_29: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_28; scores_28 = None 2025-03-14T07:25:53.1739954Z 2025-03-14T07:25:53.1740620Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1740797Z float_17: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_29.float() 2025-03-14T07:25:53.1741092Z softmax_14: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_17, dim = -1); float_17 = None 2025-03-14T07:25:53.1741384Z attn_weights_28: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_14.type_as(scores_29); softmax_14 = scores_29 = None 2025-03-14T07:25:53.1741483Z 2025-03-14T07:25:53.1742042Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1742455Z attn_weights_29: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_28, p = 0.1, training = False); attn_weights_28 = None 2025-03-14T07:25:53.1742548Z 2025-03-14T07:25:53.1743303Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1743686Z matmul_29: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_29, hidden_states_145); attn_weights_29 = hidden_states_145 = None 2025-03-14T07:25:53.1743780Z 2025-03-14T07:25:53.1744481Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1744742Z transpose_74: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_29.transpose(1, 2); matmul_29 = None 2025-03-14T07:25:53.1745006Z contiguous_14: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_74.contiguous(); transpose_74 = None 2025-03-14T07:25:53.1745437Z attn_output_28: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_14.view(4, -1, 512); contiguous_14 = None 2025-03-14T07:25:53.1745526Z 2025-03-14T07:25:53.1746071Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1747326Z attn_output_29: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_28, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_28 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1747423Z 2025-03-14T07:25:53.1748073Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1748517Z dropout_52: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_29, 0.1, False, False); attn_output_29 = None 2025-03-14T07:25:53.1748857Z hidden_states_146: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_141 + dropout_52; hidden_states_141 = dropout_52 = None 2025-03-14T07:25:53.1748949Z 2025-03-14T07:25:53.1749629Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1749825Z to_60: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_146.to(torch.float32) 2025-03-14T07:25:53.1750007Z pow_27: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_60.pow(2); to_60 = None 2025-03-14T07:25:53.1750238Z variance_26: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_27.mean(-1, keepdim = True); pow_27 = None 2025-03-14T07:25:53.1750336Z 2025-03-14T07:25:53.1751023Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1751212Z add_58: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_26 + 1e-06; variance_26 = None 2025-03-14T07:25:53.1751396Z rsqrt_26: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_58); add_58 = None 2025-03-14T07:25:53.1751673Z hidden_states_147: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_146 * rsqrt_26; rsqrt_26 = None 2025-03-14T07:25:53.1751765Z 2025-03-14T07:25:53.1752375Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1752697Z hidden_states_148: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_147.to(torch.bfloat16); hidden_states_147 = None 2025-03-14T07:25:53.1752798Z 2025-03-14T07:25:53.1753335Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1754346Z normed_hidden_states_15: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_148; l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_148 = None 2025-03-14T07:25:53.1754439Z 2025-03-14T07:25:53.1755171Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1756373Z linear_80: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_15, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_15 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1756543Z 2025-03-14T07:25:53.1757267Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1757576Z view_62: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_80.view(4, -1, 8, 64); linear_80 = None 2025-03-14T07:25:53.1757831Z query_states_15: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_62.transpose(1, 2); view_62 = None 2025-03-14T07:25:53.1757919Z 2025-03-14T07:25:53.1758527Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1759648Z linear_81: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1759743Z 2025-03-14T07:25:53.1760462Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1760734Z view_63: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_81.view(4, -1, 8, 64); linear_81 = None 2025-03-14T07:25:53.1761019Z hidden_states_149: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_63.transpose(1, 2); view_63 = None 2025-03-14T07:25:53.1761116Z 2025-03-14T07:25:53.1761719Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1762845Z linear_82: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1762938Z 2025-03-14T07:25:53.1763651Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1763915Z view_64: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_82.view(4, -1, 8, 64); linear_82 = None 2025-03-14T07:25:53.1764196Z hidden_states_150: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_64.transpose(1, 2); view_64 = None 2025-03-14T07:25:53.1764295Z 2025-03-14T07:25:53.1764857Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1765197Z transpose_78: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_149.transpose(3, 2); hidden_states_149 = None 2025-03-14T07:25:53.1765286Z 2025-03-14T07:25:53.1765798Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1766194Z scores_30: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_15, transpose_78); query_states_15 = transpose_78 = None 2025-03-14T07:25:53.1766368Z 2025-03-14T07:25:53.1766894Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1767227Z scores_30 += position_bias_3; scores_31: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_30; scores_30 = None 2025-03-14T07:25:53.1767318Z 2025-03-14T07:25:53.1768054Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1768249Z float_18: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_31.float() 2025-03-14T07:25:53.1768594Z softmax_15: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_18, dim = -1); float_18 = None 2025-03-14T07:25:53.1768926Z attn_weights_30: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_15.type_as(scores_31); softmax_15 = scores_31 = None 2025-03-14T07:25:53.1769024Z 2025-03-14T07:25:53.1769580Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1770045Z attn_weights_31: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_30, p = 0.1, training = False); attn_weights_30 = None 2025-03-14T07:25:53.1770136Z 2025-03-14T07:25:53.1770892Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1771280Z matmul_31: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_31, hidden_states_150); attn_weights_31 = hidden_states_150 = None 2025-03-14T07:25:53.1771371Z 2025-03-14T07:25:53.1772070Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1772317Z transpose_79: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_31.transpose(1, 2); matmul_31 = None 2025-03-14T07:25:53.1772587Z contiguous_15: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_79.contiguous(); transpose_79 = None 2025-03-14T07:25:53.1772871Z attn_output_30: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_15.view(4, -1, 512); contiguous_15 = None 2025-03-14T07:25:53.1772963Z 2025-03-14T07:25:53.1773504Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1774653Z attn_output_31: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_30, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_30 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1774746Z 2025-03-14T07:25:53.1775391Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1775777Z dropout_54: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_31, 0.1, False, False); attn_output_31 = None 2025-03-14T07:25:53.1776111Z layer_output_4: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_146 + dropout_54; hidden_states_146 = dropout_54 = None 2025-03-14T07:25:53.1776202Z 2025-03-14T07:25:53.1776867Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1777145Z to_62: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_4.to(torch.float32) 2025-03-14T07:25:53.1777321Z pow_28: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_62.pow(2); to_62 = None 2025-03-14T07:25:53.1777553Z variance_27: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_28.mean(-1, keepdim = True); pow_28 = None 2025-03-14T07:25:53.1777640Z 2025-03-14T07:25:53.1778433Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1778645Z add_60: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_27 + 1e-06; variance_27 = None 2025-03-14T07:25:53.1778837Z rsqrt_27: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_60); add_60 = None 2025-03-14T07:25:53.1779098Z hidden_states_151: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_4 * rsqrt_27; rsqrt_27 = None 2025-03-14T07:25:53.1779191Z 2025-03-14T07:25:53.1779798Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1780124Z hidden_states_152: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_151.to(torch.bfloat16); hidden_states_151 = None 2025-03-14T07:25:53.1780218Z 2025-03-14T07:25:53.1780773Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1781773Z forwarded_states_10: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_152; l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_152 = None 2025-03-14T07:25:53.1781872Z 2025-03-14T07:25:53.1782422Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1783650Z hidden_states_153: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_10, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_10 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1783750Z 2025-03-14T07:25:53.1784307Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1784737Z hidden_states_154: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_153, inplace = False); hidden_states_153 = None 2025-03-14T07:25:53.1784823Z 2025-03-14T07:25:53.1785398Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1785841Z hidden_states_155: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_154, 0.1, False, False); hidden_states_154 = None 2025-03-14T07:25:53.1785941Z 2025-03-14T07:25:53.1786536Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1787708Z hidden_states_156: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_155, l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_155 = l_self_modules_decoder_modules_block_modules_4_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1787903Z 2025-03-14T07:25:53.1788548Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1789021Z dropout_56: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_156, 0.1, False, False); hidden_states_156 = None 2025-03-14T07:25:53.1789353Z hidden_states_157: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output_4 + dropout_56; layer_output_4 = dropout_56 = None 2025-03-14T07:25:53.1789439Z 2025-03-14T07:25:53.1790110Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1790309Z to_64: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_157.to(torch.float32) 2025-03-14T07:25:53.1790480Z pow_29: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_64.pow(2); to_64 = None 2025-03-14T07:25:53.1790708Z variance_28: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_29.mean(-1, keepdim = True); pow_29 = None 2025-03-14T07:25:53.1790796Z 2025-03-14T07:25:53.1791483Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1791675Z add_62: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_28 + 1e-06; variance_28 = None 2025-03-14T07:25:53.1791864Z rsqrt_28: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_62); add_62 = None 2025-03-14T07:25:53.1792137Z hidden_states_158: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_157 * rsqrt_28; rsqrt_28 = None 2025-03-14T07:25:53.1792233Z 2025-03-14T07:25:53.1792846Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1793170Z hidden_states_159: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_158.to(torch.bfloat16); hidden_states_158 = None 2025-03-14T07:25:53.1793271Z 2025-03-14T07:25:53.1793816Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1794834Z normed_hidden_states_16: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_layer_norm_parameters_weight_ * hidden_states_159; l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_layer_norm_parameters_weight_ = hidden_states_159 = None 2025-03-14T07:25:53.1794933Z 2025-03-14T07:25:53.1795669Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1796772Z linear_86: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_16, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1796861Z 2025-03-14T07:25:53.1797582Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1797896Z view_66: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_86.view(4, -1, 8, 64); linear_86 = None 2025-03-14T07:25:53.1798151Z query_states_16: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_66.transpose(1, 2); view_66 = None 2025-03-14T07:25:53.1798240Z 2025-03-14T07:25:53.1798847Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1800016Z linear_87: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_16, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1800112Z 2025-03-14T07:25:53.1800827Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1801074Z view_67: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_87.view(4, -1, 8, 64); linear_87 = None 2025-03-14T07:25:53.1801326Z hidden_states_160: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_67.transpose(1, 2); view_67 = None 2025-03-14T07:25:53.1801424Z 2025-03-14T07:25:53.1802017Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:498 in project, code: hidden_states = shape(proj_layer(hidden_states)) 2025-03-14T07:25:53.1803199Z linear_88: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_16, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_, None); normed_hidden_states_16 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1803299Z 2025-03-14T07:25:53.1804011Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1804249Z view_68: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_88.view(4, -1, 8, 64); linear_88 = None 2025-03-14T07:25:53.1804503Z hidden_states_161: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_68.transpose(1, 2); view_68 = None 2025-03-14T07:25:53.1804598Z 2025-03-14T07:25:53.1805162Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1805476Z transpose_83: "bf16[4, 8, 64, 1][512, 64, 1, 512]cuda:0" = hidden_states_160.transpose(3, 2); hidden_states_160 = None 2025-03-14T07:25:53.1805569Z 2025-03-14T07:25:53.1806076Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1806418Z scores_32: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.matmul(query_states_16, transpose_83); query_states_16 = transpose_83 = None 2025-03-14T07:25:53.1806515Z 2025-03-14T07:25:53.1807045Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1807370Z scores_32 += position_bias_1; scores_33: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_32; scores_32 = position_bias_1 = None 2025-03-14T07:25:53.1807459Z 2025-03-14T07:25:53.1808124Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1808414Z float_19: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = scores_33.float() 2025-03-14T07:25:53.1808723Z softmax_16: "f32[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.softmax(float_19, dim = -1); float_19 = None 2025-03-14T07:25:53.1809008Z attn_weights_32: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = softmax_16.type_as(scores_33); softmax_16 = scores_33 = None 2025-03-14T07:25:53.1809102Z 2025-03-14T07:25:53.1809745Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1810159Z attn_weights_33: "bf16[4, 8, 1, 1][8, 1, 1, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_32, p = 0.1, training = False); attn_weights_32 = None 2025-03-14T07:25:53.1810253Z 2025-03-14T07:25:53.1811006Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1811398Z matmul_33: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_33, hidden_states_161); attn_weights_33 = hidden_states_161 = None 2025-03-14T07:25:53.1811485Z 2025-03-14T07:25:53.1812195Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1812445Z transpose_84: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_33.transpose(1, 2); matmul_33 = None 2025-03-14T07:25:53.1812714Z contiguous_16: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_84.contiguous(); transpose_84 = None 2025-03-14T07:25:53.1812990Z attn_output_32: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_16.view(4, -1, 512); contiguous_16 = None 2025-03-14T07:25:53.1813087Z 2025-03-14T07:25:53.1813624Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1814757Z attn_output_33: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_32, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_, None); attn_output_32 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_0_modules_self_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1814846Z 2025-03-14T07:25:53.1815499Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:611 in forward, code: hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1815881Z dropout_58: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_33, 0.1, False, False); attn_output_33 = None 2025-03-14T07:25:53.1816221Z hidden_states_162: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_157 + dropout_58; hidden_states_157 = dropout_58 = None 2025-03-14T07:25:53.1816310Z 2025-03-14T07:25:53.1816981Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1817191Z to_66: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_162.to(torch.float32) 2025-03-14T07:25:53.1817368Z pow_30: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_66.pow(2); to_66 = None 2025-03-14T07:25:53.1817602Z variance_29: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_30.mean(-1, keepdim = True); pow_30 = None 2025-03-14T07:25:53.1817690Z 2025-03-14T07:25:53.1818384Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1818649Z add_64: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_29 + 1e-06; variance_29 = None 2025-03-14T07:25:53.1818840Z rsqrt_29: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_64); add_64 = None 2025-03-14T07:25:53.1819100Z hidden_states_163: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_162 * rsqrt_29; rsqrt_29 = None 2025-03-14T07:25:53.1819193Z 2025-03-14T07:25:53.1819896Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1820224Z hidden_states_164: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_163.to(torch.bfloat16); hidden_states_163 = None 2025-03-14T07:25:53.1820312Z 2025-03-14T07:25:53.1820861Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1821868Z normed_hidden_states_17: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_layer_norm_parameters_weight_ * hidden_states_164; l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_layer_norm_parameters_weight_ = hidden_states_164 = None 2025-03-14T07:25:53.1821966Z 2025-03-14T07:25:53.1822694Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:521 in forward, code: query_states = shape(self.q(hidden_states)) # (batch_size, n_heads, seq_length, dim_per_head) 2025-03-14T07:25:53.1823886Z linear_90: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(normed_hidden_states_17, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_, None); normed_hidden_states_17 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_q_parameters_weight_ = None 2025-03-14T07:25:53.1823991Z 2025-03-14T07:25:53.1824702Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1824955Z view_70: "bf16[4, 1, 8, 64][512, 512, 64, 1]cuda:0" = linear_90.view(4, -1, 8, 64); linear_90 = None 2025-03-14T07:25:53.1825201Z query_states_17: "bf16[4, 8, 1, 64][512, 64, 512, 1]cuda:0" = view_70.transpose(1, 2); view_70 = None 2025-03-14T07:25:53.1825297Z 2025-03-14T07:25:53.1825896Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1827192Z linear_91: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_, None); l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_k_parameters_weight_ = None 2025-03-14T07:25:53.1827283Z 2025-03-14T07:25:53.1828003Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1828265Z view_71: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_91.view(4, -1, 8, 64); linear_91 = None 2025-03-14T07:25:53.1828554Z hidden_states_165: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_71.transpose(1, 2); view_71 = None 2025-03-14T07:25:53.1828641Z 2025-03-14T07:25:53.1829363Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:502 in project, code: hidden_states = shape(proj_layer(key_value_states)) 2025-03-14T07:25:53.1830638Z linear_92: "bf16[4, 2048, 512][1048576, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_76, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_, None); hidden_states_76 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_v_parameters_weight_ = None 2025-03-14T07:25:53.1830730Z 2025-03-14T07:25:53.1831449Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:487 in shape, code: return states.view(batch_size, -1, self.n_heads, self.key_value_proj_dim).transpose(1, 2) 2025-03-14T07:25:53.1831711Z view_72: "bf16[4, 2048, 8, 64][1048576, 512, 64, 1]cuda:0" = linear_92.view(4, -1, 8, 64); linear_92 = None 2025-03-14T07:25:53.1832002Z hidden_states_166: "bf16[4, 8, 2048, 64][1048576, 64, 512, 1]cuda:0" = view_72.transpose(1, 2); view_72 = None 2025-03-14T07:25:53.1832091Z 2025-03-14T07:25:53.1832662Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:533 in forward, code: query_states, key_states.transpose(3, 2) 2025-03-14T07:25:53.1832995Z transpose_88: "bf16[4, 8, 64, 2048][1048576, 64, 1, 512]cuda:0" = hidden_states_165.transpose(3, 2); hidden_states_165 = None 2025-03-14T07:25:53.1833100Z 2025-03-14T07:25:53.1833601Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:532 in forward, code: scores = torch.matmul( 2025-03-14T07:25:53.1834007Z scores_34: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.matmul(query_states_17, transpose_88); query_states_17 = transpose_88 = None 2025-03-14T07:25:53.1834102Z 2025-03-14T07:25:53.1834635Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:561 in forward, code: scores += position_bias_masked 2025-03-14T07:25:53.1834996Z scores_34 += position_bias_3; scores_35: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_34; scores_34 = position_bias_3 = None 2025-03-14T07:25:53.1835091Z 2025-03-14T07:25:53.1835754Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:562 in forward, code: attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:25:53.1835958Z float_20: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = scores_35.float() 2025-03-14T07:25:53.1836300Z softmax_17: "f32[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.softmax(float_20, dim = -1); float_20 = None 2025-03-14T07:25:53.1836638Z attn_weights_34: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = softmax_17.type_as(scores_35); softmax_17 = scores_35 = None 2025-03-14T07:25:53.1836731Z 2025-03-14T07:25:53.1837294Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:565 in forward, code: attn_weights = nn.functional.dropout( 2025-03-14T07:25:53.1837751Z attn_weights_35: "bf16[4, 8, 1, 2048][16384, 2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(attn_weights_34, p = 0.1, training = False); attn_weights_34 = None 2025-03-14T07:25:53.1837845Z 2025-03-14T07:25:53.1838657Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:573 in forward, code: attn_output = unshape(torch.matmul(attn_weights, value_states)) # (batch_size, seq_length, dim) 2025-03-14T07:25:53.1839038Z matmul_35: "bf16[4, 8, 1, 64][512, 64, 64, 1]cuda:0" = torch.matmul(attn_weights_35, hidden_states_166); attn_weights_35 = hidden_states_166 = None 2025-03-14T07:25:53.1839130Z 2025-03-14T07:25:53.1839903Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:491 in unshape, code: return states.transpose(1, 2).contiguous().view(batch_size, -1, self.inner_dim) 2025-03-14T07:25:53.1840162Z transpose_89: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = matmul_35.transpose(1, 2); matmul_35 = None 2025-03-14T07:25:53.1840424Z contiguous_17: "bf16[4, 1, 8, 64][512, 64, 64, 1]cuda:0" = transpose_89.contiguous(); transpose_89 = None 2025-03-14T07:25:53.1840779Z attn_output_34: "bf16[4, 1, 512][512, 64, 1]cuda:0" = contiguous_17.view(4, -1, 512); contiguous_17 = None 2025-03-14T07:25:53.1840868Z 2025-03-14T07:25:53.1841414Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:574 in forward, code: attn_output = self.o(attn_output) 2025-03-14T07:25:53.1842547Z attn_output_35: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(attn_output_34, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_, None); attn_output_34 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_1_modules_enc_dec_attention_modules_o_parameters_weight_ = None 2025-03-14T07:25:53.1842649Z 2025-03-14T07:25:53.1843291Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:647 in forward, code: layer_output = hidden_states + self.dropout(attention_output[0]) 2025-03-14T07:25:53.1843683Z dropout_60: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(attn_output_35, 0.1, False, False); attn_output_35 = None 2025-03-14T07:25:53.1844007Z layer_output_5: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_162 + dropout_60; hidden_states_162 = dropout_60 = None 2025-03-14T07:25:53.1844103Z 2025-03-14T07:25:53.1844769Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1844971Z to_68: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_5.to(torch.float32) 2025-03-14T07:25:53.1845154Z pow_31: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_68.pow(2); to_68 = None 2025-03-14T07:25:53.1845384Z variance_30: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_31.mean(-1, keepdim = True); pow_31 = None 2025-03-14T07:25:53.1845479Z 2025-03-14T07:25:53.1846167Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1846359Z add_66: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_30 + 1e-06; variance_30 = None 2025-03-14T07:25:53.1846546Z rsqrt_30: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_66); add_66 = None 2025-03-14T07:25:53.1846808Z hidden_states_167: "f32[4, 1, 512][512, 512, 1]cuda:0" = layer_output_5 * rsqrt_30; rsqrt_30 = None 2025-03-14T07:25:53.1846895Z 2025-03-14T07:25:53.1847509Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1847831Z hidden_states_168: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_167.to(torch.bfloat16); hidden_states_167 = None 2025-03-14T07:25:53.1847924Z 2025-03-14T07:25:53.1848468Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1849474Z forwarded_states_11: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_layer_norm_parameters_weight_ * hidden_states_168; l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_layer_norm_parameters_weight_ = hidden_states_168 = None 2025-03-14T07:25:53.1849646Z 2025-03-14T07:25:53.1850204Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:289 in forward, code: hidden_states = self.wi(hidden_states) 2025-03-14T07:25:53.1851485Z hidden_states_169: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch._C._nn.linear(forwarded_states_11, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_, None); forwarded_states_11 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_dense_relu_dense_modules_wi_parameters_weight_ = None 2025-03-14T07:25:53.1851583Z 2025-03-14T07:25:53.1852146Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:290 in forward, code: hidden_states = self.act(hidden_states) 2025-03-14T07:25:53.1852570Z hidden_states_170: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.relu(hidden_states_169, inplace = False); hidden_states_169 = None 2025-03-14T07:25:53.1852663Z 2025-03-14T07:25:53.1853230Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:291 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1853677Z hidden_states_171: "bf16[4, 1, 2048][2048, 2048, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_170, 0.1, False, False); hidden_states_170 = None 2025-03-14T07:25:53.1853772Z 2025-03-14T07:25:53.1854322Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:298 in forward, code: hidden_states = self.wo(hidden_states) 2025-03-14T07:25:53.1855489Z hidden_states_172: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch._C._nn.linear(hidden_states_171, l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_, None); hidden_states_171 = l_self_modules_decoder_modules_block_modules_5_modules_layer_modules_2_modules_dense_relu_dense_modules_wo_parameters_weight_ = None 2025-03-14T07:25:53.1855591Z 2025-03-14T07:25:53.1856233Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:345 in forward, code: hidden_states = hidden_states + self.dropout(forwarded_states) 2025-03-14T07:25:53.1856646Z dropout_62: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_172, 0.1, False, False); hidden_states_172 = None 2025-03-14T07:25:53.1856970Z hidden_states_173: "bf16[4, 1, 512][512, 512, 1]cuda:0" = layer_output_5 + dropout_62; layer_output_5 = dropout_62 = None 2025-03-14T07:25:53.1857065Z 2025-03-14T07:25:53.1857736Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:254 in forward, code: variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) 2025-03-14T07:25:53.1857941Z to_70: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_173.to(torch.float32) 2025-03-14T07:25:53.1858115Z pow_32: "f32[4, 1, 512][512, 512, 1]cuda:0" = to_70.pow(2); to_70 = None 2025-03-14T07:25:53.1858353Z variance_31: "f32[4, 1, 1][1, 1, 1]cuda:0" = pow_32.mean(-1, keepdim = True); pow_32 = None 2025-03-14T07:25:53.1858442Z 2025-03-14T07:25:53.1859145Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:255 in forward, code: hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) 2025-03-14T07:25:53.1859339Z add_68: "f32[4, 1, 1][1, 1, 1]cuda:0" = variance_31 + 1e-06; variance_31 = None 2025-03-14T07:25:53.1859523Z rsqrt_31: "f32[4, 1, 1][1, 1, 1]cuda:0" = torch.rsqrt(add_68); add_68 = None 2025-03-14T07:25:53.1859851Z hidden_states_174: "f32[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_173 * rsqrt_31; hidden_states_173 = rsqrt_31 = None 2025-03-14T07:25:53.1860025Z 2025-03-14T07:25:53.1860637Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:259 in forward, code: hidden_states = hidden_states.to(self.weight.dtype) 2025-03-14T07:25:53.1860958Z hidden_states_175: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_174.to(torch.bfloat16); hidden_states_174 = None 2025-03-14T07:25:53.1861052Z 2025-03-14T07:25:53.1861685Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:261 in forward, code: return self.weight * hidden_states 2025-03-14T07:25:53.1862399Z hidden_states_176: "bf16[4, 1, 512][512, 512, 1]cuda:0" = l_self_modules_decoder_modules_final_layer_norm_parameters_weight_ * hidden_states_175; l_self_modules_decoder_modules_final_layer_norm_parameters_weight_ = hidden_states_175 = None 2025-03-14T07:25:53.1862488Z 2025-03-14T07:25:53.1863076Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1158 in forward, code: hidden_states = self.dropout(hidden_states) 2025-03-14T07:25:53.1863497Z hidden_states_177: "bf16[4, 1, 512][512, 512, 1]cuda:0" = torch.nn.functional.dropout(hidden_states_176, 0.1, False, False); hidden_states_176 = None 2025-03-14T07:25:53.1863593Z 2025-03-14T07:25:53.1864226Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1774 in forward, code: sequence_output = sequence_output * (self.model_dim**-0.5) 2025-03-14T07:25:53.1864556Z sequence_output: "bf16[4, 1, 512][512, 512, 1]cuda:0" = hidden_states_177 * 0.04419417382415922; hidden_states_177 = None 2025-03-14T07:25:53.1864645Z 2025-03-14T07:25:53.1865216Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py:1776 in forward, code: lm_logits = self.lm_head(sequence_output) 2025-03-14T07:25:53.1865990Z lm_logits: "bf16[4, 1, 32128][32128, 32128, 1]cuda:0" = torch._C._nn.linear(sequence_output, l_self_modules_encoder_modules_embed_tokens_parameters_weight_, None); sequence_output = l_self_modules_encoder_modules_embed_tokens_parameters_weight_ = None 2025-03-14T07:25:53.1866094Z 2025-03-14T07:25:53.1866751Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2414 in greedy_search, code: next_token_logits = outputs.logits[:, -1, :] 2025-03-14T07:25:53.1867151Z next_token_logits: "bf16[4, 32128][32128, 1]cuda:0" = lm_logits[(slice(None, None, None), -1, slice(None, None, None))]; lm_logits = None 2025-03-14T07:25:53.1867246Z 2025-03-14T07:25:53.1867864Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2440 in greedy_search, code: next_tokens = torch.argmax(next_tokens_scores, dim=-1) 2025-03-14T07:25:53.1868168Z next_tokens: "i64[4][1]cuda:0" = torch.argmax(next_token_logits, dim = -1); next_token_logits = None 2025-03-14T07:25:53.1868256Z 2025-03-14T07:25:53.1869000Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2446 in greedy_search, code: next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences) 2025-03-14T07:25:53.1869219Z mul_73: "i64[4][1]cuda:0" = next_tokens * unfinished_sequences; next_tokens = None 2025-03-14T07:25:53.1869373Z sub_5: "i64[4][1]cuda:0" = 1 - unfinished_sequences 2025-03-14T07:25:53.1869519Z mul_74: "i64[4][1]cuda:0" = 0 * sub_5; sub_5 = None 2025-03-14T07:25:53.1869719Z next_tokens_1: "i64[4][1]cuda:0" = mul_73 + mul_74; mul_73 = mul_74 = None 2025-03-14T07:25:53.1869807Z 2025-03-14T07:25:53.1870464Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2449 in greedy_search, code: input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1) 2025-03-14T07:25:53.1870764Z getitem_11: "i64[4, 1][1, 1]cuda:0" = next_tokens_1[(slice(None, None, None), None)] 2025-03-14T07:25:53.1871196Z input_ids_2: "i64[4, 2][2, 1]cuda:0" = torch.cat([decoder_input_ids_start, getitem_11], dim = -1); decoder_input_ids_start = getitem_11 = input_ids_2 = None 2025-03-14T07:25:53.1871288Z 2025-03-14T07:25:53.1872128Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2459 in greedy_search, code: next_tokens.tile(eos_token_id_tensor.shape[0], 1).ne(eos_token_id_tensor.unsqueeze(1)).prod(dim=0) 2025-03-14T07:25:53.1872329Z tile: "i64[1, 4][4, 1]cuda:0" = next_tokens_1.tile(1, 1); next_tokens_1 = None 2025-03-14T07:25:53.1872610Z unsqueeze_2: "i64[1, 1][1, 1]cuda:0" = eos_token_id_tensor.unsqueeze(1); eos_token_id_tensor = None 2025-03-14T07:25:53.1872801Z ne: "b8[1, 4][4, 1]cuda:0" = tile.ne(unsqueeze_2); tile = unsqueeze_2 = None 2025-03-14T07:25:53.1872969Z prod: "i64[4][1]cuda:0" = ne.prod(dim = 0); ne = None 2025-03-14T07:25:53.1873057Z 2025-03-14T07:25:53.1873675Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2458 in greedy_search, code: unfinished_sequences = unfinished_sequences.mul( 2025-03-14T07:25:53.1874021Z unfinished_sequences_1: "i64[4][1]cuda:0" = unfinished_sequences.mul(prod); unfinished_sequences = prod = None 2025-03-14T07:25:53.1874116Z 2025-03-14T07:25:53.1874673Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py:2463 in greedy_search, code: if unfinished_sequences.max() == 0: 2025-03-14T07:25:53.1874922Z max_1: "i64[][]cuda:0" = unfinished_sequences_1.max(); unfinished_sequences_1 = None 2025-03-14T07:25:53.1875061Z eq: "b8[][]cuda:0" = max_1 == 0; max_1 = eq = None 2025-03-14T07:25:53.1875157Z 2025-03-14T07:26:34.6331864Z CUDAGraph supports dynamic shapes by recording a new graph for each distinct input size. Recording too many CUDAGraphs may lead to extra overhead. We have observed 51 distinct sizes. Please consider the following options for better performance: a) padding inputs to a few fixed number of shapes; or b) set torch._inductor.config.triton.cudagraph_skip_dynamic_graphs=True. Set torch._inductor.config.triton.cudagraph_dynamic_shape_warn_limit=None to silence this warning. 2025-03-14T07:30:30.2523670Z Compilation time (from dynamo_timed): 44.533768579 2025-03-14T07:30:30.2524049Z pass 2025-03-14T07:30:30.2557897Z TIMING: entire_frame_compile:44.53377 gc:0.00543 _recursive_pre_grad_passes:0.02371 pad_mm_benchmark:0.57052 _recursive_joint_graph_passes:2.1247 _recursive_post_grad_passes:0.9916 async_compile.wait:1.9636 code_gen:7.41431 inductor_compile:15.26755 backend_compile:29.84858 cudagraphify.get_container:0.24444 CachingAutotuner.benchmark_all_configs:0.0971 CUDAGraphNode.record:193.61538 total_wall_time:44.53377 2025-03-14T07:30:30.2560342Z STATS: call_* op count: 1815 | FakeTensorMode.__torch_dispatch__:51709 | ProxyTorchDispatchMode.__torch_dispatch__:18086 | FakeTensor.__torch_dispatch__:4557 | attempt fast:115 | fast is_contiguous:85 | slow both tensors nontrivially broadcast:12 | slow no contiguity match:18 2025-03-14T07:30:30.2561563Z Dynamo produced 4 graphs covering 1815 ops with 5 graph breaks (3 unique) 2025-03-14T07:30:37.8060177Z 2025-03-14T07:30:47.9360533Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:30:47.9360890Z loading model: 0it [00:10, ?it/s] 2025-03-14T07:30:47.9697575Z cuda eval hf_T5_large 2025-03-14T07:30:47.9932991Z Compilation time (from dynamo_timed): 0 2025-03-14T07:30:47.9933861Z pass_due_to_skip 2025-03-14T07:30:48.2783754Z TIMING: total_wall_time:0 2025-03-14T07:30:48.2784168Z STATS: call_* op count: 0 2025-03-14T07:30:48.2784569Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:30:52.5999181Z 2025-03-14T07:30:54.0298633Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:30:54.0299105Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:30:54.0316409Z cuda eval hf_Whisper 2025-03-14T07:31:02.4881945Z Compilation time (from dynamo_timed): 5.952938419 2025-03-14T07:31:02.4882331Z pass 2025-03-14T07:31:02.4892914Z TIMING: _recursive_pre_grad_passes:0.00487 pad_mm_benchmark:0.07874 _recursive_joint_graph_passes:0.30956 _recursive_post_grad_passes:0.15759 async_compile.wait:1.1752 code_gen:2.16368 inductor_compile:3.066 backend_compile:4.93844 entire_frame_compile:5.95294 gc:0.00056 cudagraphify.get_container:0.20226 CachingAutotuner.benchmark_all_configs:0.12948 CUDAGraphNode.record:0.19907 total_wall_time:5.95294 2025-03-14T07:31:02.4894849Z STATS: call_* op count: 115 | FakeTensorMode.__torch_dispatch__:5176 | FakeTensor.__torch_dispatch__:674 | ProxyTorchDispatchMode.__torch_dispatch__:2054 2025-03-14T07:31:02.4895657Z Dynamo produced 1 graphs covering 115 ops with 0 graph breaks (0 unique) 2025-03-14T07:31:07.8100700Z 2025-03-14T07:31:12.7159275Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:31:12.7159652Z loading model: 0it [00:04, ?it/s] 2025-03-14T07:31:12.7255497Z cuda eval hf_distil_whisper 2025-03-14T07:31:58.0347159Z Compilation time (from dynamo_timed): 20.054314253 2025-03-14T07:31:58.0347681Z pass 2025-03-14T07:31:58.0512880Z TIMING: _recursive_pre_grad_passes:0.01082 pad_mm_benchmark:0.0847 _recursive_joint_graph_passes:0.77693 _recursive_post_grad_passes:1.0759 async_compile.wait:1.11149 code_gen:4.32149 inductor_compile:8.82988 backend_compile:15.55097 entire_frame_compile:20.05431 gc:0.00014 cudagraphify.get_container:0.34333 CachingAutotuner.benchmark_all_configs:0.30333 CUDAGraphNode.record:0.33742 total_wall_time:20.05431 2025-03-14T07:31:58.0514853Z STATS: call_* op count: 635 | FakeTensorMode.__torch_dispatch__:25994 | FakeTensor.__torch_dispatch__:3554 | ProxyTorchDispatchMode.__torch_dispatch__:11254 2025-03-14T07:31:58.0515683Z Dynamo produced 1 graphs covering 635 ops with 0 graph breaks (0 unique) 2025-03-14T07:32:03.9648111Z 2025-03-14T07:32:04.4924628Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:32:04.4925018Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:32:04.4929884Z cuda eval lennard_jones 2025-03-14T07:32:07.5716207Z Compilation time (from dynamo_timed): 1.650986015 2025-03-14T07:32:07.5717060Z pass 2025-03-14T07:32:07.5718508Z TIMING: _recursive_pre_grad_passes:0.00334 pad_mm_benchmark:0.22624 _recursive_joint_graph_passes:0.36329 _recursive_post_grad_passes:0.03963 async_compile.precompile:0.09385 async_compile.wait:0.00018 code_gen:0.15688 inductor_compile:0.28894 backend_compile:1.47389 entire_frame_compile:1.65099 gc:0.00022 cudagraphify.get_container:0.15418 CUDAGraphNode.record:0.15307 total_wall_time:1.65099 2025-03-14T07:32:07.5720385Z STATS: call_* op count: 9 | FakeTensorMode.__torch_dispatch__:824 | ProxyTorchDispatchMode.__torch_dispatch__:223 | FakeTensor.__torch_dispatch__:49 2025-03-14T07:32:07.5721180Z Dynamo produced 1 graphs covering 9 ops with 0 graph breaks (0 unique) 2025-03-14T07:32:09.4535078Z accuracy pass_rate=66.67% 2025-03-14T07:32:09.4538827Z calls_captured gmean=0.00x mean=291.104x 2025-03-14T07:32:09.4542261Z unique_graphs gmean=0.00x mean=1.729x 2025-03-14T07:32:09.4545861Z graph_breaks gmean=0.00x mean=1.000x 2025-03-14T07:32:09.4549589Z unique_graph_breaks gmean=0.00x mean=0.312x 2025-03-14T07:32:09.4552845Z autograd_captures gmean=0.00x mean=0.000x 2025-03-14T07:32:09.4556171Z autograd_compiles gmean=0.00x mean=0.000x 2025-03-14T07:32:09.4559431Z cudagraph_skips gmean=0.00x mean=0.292x 2025-03-14T07:32:09.4560370Z compilation_latency mean=9.341 seconds 2025-03-14T07:32:10.5374113Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv 2025-03-14T07:32:10.8732241Z torchrec_dlrm XFAIL 2025-03-14T07:32:10.8736485Z BERT_pytorch PASS 2025-03-14T07:32:10.8740932Z Background_Matting XFAIL 2025-03-14T07:32:10.8745904Z LearningToPaint PASS 2025-03-14T07:32:10.8750863Z Super_SloMo PASS 2025-03-14T07:32:10.8755421Z alexnet PASS 2025-03-14T07:32:10.8760139Z basic_gnn_edgecnn PASS 2025-03-14T07:32:10.8765032Z basic_gnn_gcn PASS 2025-03-14T07:32:10.8769703Z basic_gnn_gin PASS 2025-03-14T07:32:10.8774771Z basic_gnn_sage PASS 2025-03-14T07:32:10.8779120Z dcgan PASS 2025-03-14T07:32:10.8783759Z demucs PASS 2025-03-14T07:32:10.8788432Z densenet121 PASS 2025-03-14T07:32:10.8793297Z detectron2_fasterrcnn_r_101_c4 XFAIL 2025-03-14T07:32:10.8797976Z detectron2_fasterrcnn_r_101_dc5 XFAIL 2025-03-14T07:32:10.8802537Z detectron2_fasterrcnn_r_101_fpn XFAIL 2025-03-14T07:32:10.8807317Z detectron2_fasterrcnn_r_50_c4 XFAIL 2025-03-14T07:32:10.8811909Z detectron2_fasterrcnn_r_50_dc5 XFAIL 2025-03-14T07:32:10.8816640Z detectron2_fasterrcnn_r_50_fpn XFAIL 2025-03-14T07:32:10.8821245Z detectron2_fcos_r_50_fpn PASS 2025-03-14T07:32:10.8826031Z detectron2_maskrcnn_r_101_c4 XFAIL 2025-03-14T07:32:10.8831367Z detectron2_maskrcnn_r_101_fpn XFAIL 2025-03-14T07:32:10.8836050Z detectron2_maskrcnn_r_50_c4 XFAIL 2025-03-14T07:32:10.8840530Z detectron2_maskrcnn_r_50_fpn XFAIL 2025-03-14T07:32:10.8845225Z dlrm PASS 2025-03-14T07:32:10.8849766Z doctr_det_predictor PASS 2025-03-14T07:32:10.8854471Z doctr_reco_predictor PASS 2025-03-14T07:32:10.8859109Z drq PASS 2025-03-14T07:32:10.8863784Z fastNLP_Bert PASS 2025-03-14T07:32:10.8868691Z functorch_dp_cifar10 PASS 2025-03-14T07:32:10.8873377Z functorch_maml_omniglot PASS 2025-03-14T07:32:10.8877956Z hf_Albert PASS 2025-03-14T07:32:10.8882605Z hf_Bart PASS 2025-03-14T07:32:10.8887205Z hf_Bert PASS 2025-03-14T07:32:10.8891913Z hf_Bert_large PASS 2025-03-14T07:32:10.8896517Z hf_BigBird XFAIL 2025-03-14T07:32:10.8901276Z hf_DistilBert PASS 2025-03-14T07:32:10.8905852Z hf_GPT2 PASS 2025-03-14T07:32:10.8910853Z hf_GPT2_large XFAIL 2025-03-14T07:32:10.8915370Z hf_Reformer PASS 2025-03-14T07:32:10.8920082Z hf_Roberta_base PASS 2025-03-14T07:32:10.8924680Z hf_T5 PASS 2025-03-14T07:32:10.8929798Z hf_T5_base XFAIL 2025-03-14T07:32:10.8934349Z hf_T5_generate PASS 2025-03-14T07:32:10.8939030Z hf_T5_large XFAIL 2025-03-14T07:32:10.8943611Z hf_Whisper PASS 2025-03-14T07:32:10.8948585Z hf_distil_whisper PASS 2025-03-14T07:32:10.8953322Z lennard_jones PASS 2025-03-14T07:32:10.9482288Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv 2025-03-14T07:32:11.2773221Z torchrec_dlrm PASS 2025-03-14T07:32:11.2775976Z BERT_pytorch PASS 2025-03-14T07:32:11.2780529Z Background_Matting PASS 2025-03-14T07:32:11.2786777Z LearningToPaint PASS 2025-03-14T07:32:11.2792859Z Super_SloMo PASS 2025-03-14T07:32:11.2798499Z alexnet PASS 2025-03-14T07:32:11.2804261Z basic_gnn_edgecnn PASS 2025-03-14T07:32:11.2810144Z basic_gnn_gcn PASS 2025-03-14T07:32:11.2815825Z basic_gnn_gin PASS 2025-03-14T07:32:11.2821619Z basic_gnn_sage PASS 2025-03-14T07:32:11.2827295Z dcgan PASS 2025-03-14T07:32:11.2833041Z demucs PASS 2025-03-14T07:32:11.2837676Z densenet121 PASS 2025-03-14T07:32:11.2842756Z detectron2_fasterrcnn_r_101_c4 PASS 2025-03-14T07:32:11.2847502Z detectron2_fasterrcnn_r_101_dc5 PASS 2025-03-14T07:32:11.2852586Z detectron2_fasterrcnn_r_101_fpn PASS 2025-03-14T07:32:11.2857648Z detectron2_fasterrcnn_r_50_c4 PASS 2025-03-14T07:32:11.2862389Z detectron2_fasterrcnn_r_50_dc5 PASS 2025-03-14T07:32:11.2867272Z detectron2_fasterrcnn_r_50_fpn PASS 2025-03-14T07:32:11.2872437Z detectron2_fcos_r_50_fpn PASS 2025-03-14T07:32:11.2877038Z detectron2_maskrcnn_r_101_c4 PASS_BUT_FLAKY 2025-03-14T07:32:11.2882213Z detectron2_maskrcnn_r_101_fpn PASS 2025-03-14T07:32:11.2886957Z detectron2_maskrcnn_r_50_c4 PASS 2025-03-14T07:32:11.2891827Z detectron2_maskrcnn_r_50_fpn PASS 2025-03-14T07:32:11.2896577Z dlrm PASS 2025-03-14T07:32:11.2901504Z doctr_det_predictor PASS 2025-03-14T07:32:11.2906257Z doctr_reco_predictor PASS 2025-03-14T07:32:11.2911318Z drq PASS 2025-03-14T07:32:11.2916134Z fastNLP_Bert PASS 2025-03-14T07:32:11.2920886Z functorch_dp_cifar10 PASS 2025-03-14T07:32:11.2925742Z functorch_maml_omniglot PASS 2025-03-14T07:32:11.2931894Z hf_Albert PASS 2025-03-14T07:32:11.2936743Z hf_Bart PASS 2025-03-14T07:32:11.2941619Z hf_Bert PASS 2025-03-14T07:32:11.2946433Z hf_Bert_large PASS 2025-03-14T07:32:11.2951653Z hf_BigBird PASS 2025-03-14T07:32:11.2956322Z hf_DistilBert PASS 2025-03-14T07:32:11.2961114Z hf_GPT2 PASS 2025-03-14T07:32:11.2965949Z hf_GPT2_large PASS 2025-03-14T07:32:11.2970773Z hf_Reformer PASS 2025-03-14T07:32:11.2975616Z hf_Roberta_base PASS 2025-03-14T07:32:11.2980427Z hf_T5 PASS 2025-03-14T07:32:11.2985260Z hf_T5_base PASS 2025-03-14T07:32:11.2990307Z hf_T5_generate PASS 2025-03-14T07:32:11.2994967Z hf_T5_large PASS 2025-03-14T07:32:11.2999886Z hf_Whisper PASS 2025-03-14T07:32:11.3004595Z hf_distil_whisper PASS 2025-03-14T07:32:11.3009451Z lennard_jones PASS 2025-03-14T07:32:11.3549513Z + test_single_dynamo_benchmark training torchbench 0 --training --amp 2025-03-14T07:32:11.3553363Z ++ pwd 2025-03-14T07:32:11.3556768Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-03-14T07:32:11.3557240Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-03-14T07:32:11.3582918Z + local name=training 2025-03-14T07:32:11.3583275Z + shift 2025-03-14T07:32:11.3583572Z + local suite=torchbench 2025-03-14T07:32:11.3583897Z + shift 2025-03-14T07:32:11.3584110Z + local shard_id=0 2025-03-14T07:32:11.3584348Z + shift 2025-03-14T07:32:11.3584573Z + partition_flags=() 2025-03-14T07:32:11.3584839Z + local partition_flags 2025-03-14T07:32:11.3585101Z + [[ -n 2 ]] 2025-03-14T07:32:11.3585325Z + [[ -n 0 ]] 2025-03-14T07:32:11.3585732Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-03-14T07:32:11.3586270Z + [[ inductor_torchbench == *perf_compare* ]] 2025-03-14T07:32:11.3586728Z + [[ inductor_torchbench == *perf* ]] 2025-03-14T07:32:11.3587059Z + [[ inductor_torchbench == *_avx2* ]] 2025-03-14T07:32:11.3587392Z + [[ inductor_torchbench == *_avx512* ]] 2025-03-14T07:32:11.3588518Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cuda --training --amp --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2025-03-14T07:32:18.1059537Z 2025-03-14T07:32:19.2136415Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:32:19.2136787Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:32:19.2137121Z cuda train torchrec_dlrm 2025-03-14T07:32:33.0464302Z Compilation time (from dynamo_timed): 11.521633976 2025-03-14T07:32:33.0468877Z pass 2025-03-14T07:32:33.1579410Z TIMING: entire_frame_compile:9.34908 gc:0.00312 _recursive_pre_grad_passes:0.00528 pad_mm_benchmark:1.48425 _recursive_joint_graph_passes:1.69633 _recursive_post_grad_passes:0.10706 async_compile.wait:1.87386 code_gen:3.53632 inductor_compile:5.06747 backend_compile:6.88687 cudagraphify.get_container:0.24287 entire_backward_compile:2.17256 CUDAGraphNode.record:0.48008 total_wall_time:11.52163 2025-03-14T07:32:33.1581304Z STATS: call_* op count: 123 | FakeTensorMode.__torch_dispatch__:8241 | FakeTensor.__torch_dispatch__:1431 | ProxyTorchDispatchMode.__torch_dispatch__:3086 2025-03-14T07:32:33.1582133Z Dynamo produced 2 graphs covering 123 ops with 6 graph breaks (5 unique) 2025-03-14T07:32:34.8238711Z [rank0]:[W314 07:32:34.800397285 ProcessGroupNCCL.cpp:1497] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-03-14T07:32:38.7029672Z 2025-03-14T07:32:41.0000115Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:32:41.0000501Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:32:41.0000839Z cuda train BERT_pytorch 2025-03-14T07:33:17.0824970Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:33:17.0827045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:33:17.0828873Z pred = mod(*cloned_inputs) 2025-03-14T07:33:17.0829894Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/language_model.py", line 24, in forward 2025-03-14T07:33:17.0830641Z x = self.bert(x, segment_label) 2025-03-14T07:33:17.0831328Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/bert.py", line 47, in forward 2025-03-14T07:33:17.0832027Z x = self.embedding(x, segment_info) 2025-03-14T07:33:17.0832778Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch/bert_pytorch/model/embedding/bert.py", line 32, in forward 2025-03-14T07:33:17.0833649Z x = self.token(sequence) + self.position(sequence) + self.segment(segment_label) 2025-03-14T07:33:17.0834009Z 2025-03-14T07:33:17.5120608Z Compilation time (from dynamo_timed): 32.640146211 2025-03-14T07:33:17.5145855Z pass 2025-03-14T07:33:17.5914601Z TIMING: entire_frame_compile:22.58049 gc:0.00378 _recursive_pre_grad_passes:0.00911 pad_mm_benchmark:0.58027 _recursive_joint_graph_passes:1.76174 _recursive_post_grad_passes:1.2353 async_compile.wait:3.0153 code_gen:10.87687 inductor_compile:17.70491 backend_compile:17.57051 cudagraphify.get_container:0.34476 entire_backward_compile:10.05966 total_wall_time:32.64015 2025-03-14T07:33:17.5917504Z STATS: call_* op count: 547 | FakeTensorMode.__torch_dispatch__:42971 | ProxyTorchDispatchMode.__torch_dispatch__:21212 | FakeTensor.__torch_dispatch__:5520 2025-03-14T07:33:17.5918350Z Dynamo produced 2 graphs covering 547 ops with 6 graph breaks (5 unique) 2025-03-14T07:33:23.9413057Z 2025-03-14T07:33:28.7440728Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:33:28.7441105Z loading model: 0it [00:04, ?it/s] 2025-03-14T07:33:28.7441449Z cuda train Background_Matting 2025-03-14T07:33:28.7677114Z Compilation time (from dynamo_timed): 0 2025-03-14T07:33:28.7677521Z pass_due_to_skip 2025-03-14T07:33:28.7927168Z TIMING: total_wall_time:0 2025-03-14T07:33:28.7927781Z STATS: call_* op count: 0 2025-03-14T07:33:28.7929138Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:33:33.1295444Z 2025-03-14T07:33:36.4292661Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:33:36.4293177Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:33:36.4293621Z cuda train LearningToPaint 2025-03-14T07:33:52.6662203Z Compilation time (from dynamo_timed): 13.083214714 2025-03-14T07:33:52.6670677Z pass 2025-03-14T07:33:52.6746163Z TIMING: entire_frame_compile:7.97168 gc:0.00269 _recursive_pre_grad_passes:0.00618 pad_mm_benchmark:0.21013 _recursive_joint_graph_passes:0.40987 _recursive_post_grad_passes:0.15675 async_compile.wait:2.16358 code_gen:5.34611 inductor_compile:7.58032 backend_compile:5.89647 cudagraphify.get_container:0.24046 entire_backward_compile:5.11153 CUDAGraphNode.record:0.54825 total_wall_time:13.08321 2025-03-14T07:33:52.6748141Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:9321 | ProxyTorchDispatchMode.__torch_dispatch__:4410 | FakeTensor.__torch_dispatch__:1936 2025-03-14T07:33:52.6748970Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2025-03-14T07:33:58.4762403Z 2025-03-14T07:34:01.1797674Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:34:01.1798149Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:34:01.1798522Z cuda train Super_SloMo 2025-03-14T07:35:05.0958133Z W0314 07:35:05.094000 58168 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:35:27.3248033Z Compilation time (from dynamo_timed): 60.104234169 2025-03-14T07:35:27.3261757Z pass 2025-03-14T07:35:27.3496534Z TIMING: entire_frame_compile:45.58108 gc:0.00453 _recursive_pre_grad_passes:0.02207 _recursive_joint_graph_passes:0.69427 _recursive_post_grad_passes:0.50144 async_compile.wait:5.59614 code_gen:29.32625 inductor_compile:39.16671 backend_compile:36.58781 cudagraphify.get_container:0.36821 CachingAutotuner.benchmark_all_configs:3.71108 entire_backward_compile:14.52315 CUDAGraphNode.record:1.83048 total_wall_time:60.10423 2025-03-14T07:35:27.3500021Z STATS: call_* op count: 857 | FakeTensorMode.__torch_dispatch__:39809 | ProxyTorchDispatchMode.__torch_dispatch__:17943 | FakeTensor.__torch_dispatch__:8745 2025-03-14T07:35:27.3500860Z Dynamo produced 3 graphs covering 857 ops with 7 graph breaks (5 unique) 2025-03-14T07:35:34.5844256Z 2025-03-14T07:35:36.2323695Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:35:36.2324078Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:35:36.2324445Z cuda train alexnet 2025-03-14T07:35:51.8256063Z Compilation time (from dynamo_timed): 11.756133115 2025-03-14T07:35:51.8256727Z pass 2025-03-14T07:35:51.8454647Z TIMING: entire_frame_compile:4.2853 gc:0.00169 _recursive_pre_grad_passes:0.00415 pad_mm_benchmark:0.53937 _recursive_joint_graph_passes:0.6969 _recursive_post_grad_passes:0.0512 async_compile.wait:1.64166 code_gen:8.61419 inductor_compile:9.25831 backend_compile:3.72128 cudagraphify.get_container:0.18163 CachingAutotuner.benchmark_all_configs:0.7428 entire_backward_compile:7.47083 CUDAGraphNode.record:0.38082 total_wall_time:11.75613 2025-03-14T07:35:51.8456775Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:2645 | ProxyTorchDispatchMode.__torch_dispatch__:954 | FakeTensor.__torch_dispatch__:270 2025-03-14T07:35:51.8457599Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2025-03-14T07:35:57.2723230Z 2025-03-14T07:35:59.3779158Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:35:59.3779680Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:35:59.3780099Z cuda train basic_gnn_edgecnn 2025-03-14T07:36:02.8597281Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_reduce.two Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 98, in torch_dynamo_resume_in_scatter_at_93 2025-03-14T07:36:02.8598979Z return src.new_zeros(size).scatter_reduce_( 2025-03-14T07:36:02.8599564Z 2025-03-14T07:36:02.8599568Z 2025-03-14T07:36:04.8739392Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:36:04.8739940Z File "/tmp/jenkins_pyg/tmp4ngdy7fy.py", line 190, in propagate 2025-03-14T07:36:04.8740403Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2025-03-14T07:36:04.8740857Z File "/tmp/jenkins_pyg/tmp4ngdy7fy.py", line 128, in _collect 2025-03-14T07:36:04.8741256Z x_i = self._lift(x_i, edge_def, i) 2025-03-14T07:36:04.8741987Z File "/tmp/jenkins_pyg/tmp4ngdy7fy.py", line 88, in _lift 2025-03-14T07:36:04.8742408Z return src.index_select(self.node_dim, index) 2025-03-14T07:36:04.8742655Z 2025-03-14T07:36:05.0394268Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:36:05.0394794Z File "/tmp/jenkins_pyg/tmpra4t_wpw.py", line 190, in propagate 2025-03-14T07:36:05.0395252Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2025-03-14T07:36:05.0395720Z File "/tmp/jenkins_pyg/tmpra4t_wpw.py", line 128, in _collect 2025-03-14T07:36:05.0396116Z x_i = self._lift(x_i, edge_def, i) 2025-03-14T07:36:05.0396498Z File "/tmp/jenkins_pyg/tmpra4t_wpw.py", line 88, in _lift 2025-03-14T07:36:05.0396918Z return src.index_select(self.node_dim, index) 2025-03-14T07:36:05.0397157Z 2025-03-14T07:36:05.2743185Z W0314 07:36:05.273000 61200 site-packages/torch/_logging/_internal.py:1130] [24/0] Profiler function will be ignored 2025-03-14T07:36:09.6993949Z Compilation time (from dynamo_timed): 8.027627113 2025-03-14T07:36:09.6994719Z pass 2025-03-14T07:36:09.7006185Z TIMING: entire_frame_compile:7.2255 gc:0.01032 _recursive_pre_grad_passes:0.01153 _recursive_joint_graph_passes:0.19772 inductor_compile:4.14694 backend_compile:5.34815 pad_mm_benchmark:0.01025 _recursive_post_grad_passes:0.05934 async_compile.wait:2.45448 code_gen:3.13317 cudagraphify.get_container:0.17065 entire_backward_compile:0.80213 CUDAGraphNode.record:1.05292 total_wall_time:8.02763 2025-03-14T07:36:09.7008086Z STATS: call_* op count: 128 | FakeTensorMode.__torch_dispatch__:4847 | ProxyTorchDispatchMode.__torch_dispatch__:1882 | FakeTensor.__torch_dispatch__:905 2025-03-14T07:36:09.7008964Z Dynamo produced 8 graphs covering 128 ops with 20 graph breaks (6 unique) 2025-03-14T07:36:15.1948543Z 2025-03-14T07:36:17.3339437Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:36:17.3339884Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:36:17.3340223Z cuda train basic_gnn_gcn 2025-03-14T07:36:20.2011754Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 100, in torch_dynamo_resume_in_gcn_norm_at_91 2025-03-14T07:36:20.2013014Z deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum') 2025-03-14T07:36:20.2013729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-03-14T07:36:20.2014404Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-03-14T07:36:20.2014681Z 2025-03-14T07:36:20.2014686Z 2025-03-14T07:36:20.6676574Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmpxm84vu9v.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-03-14T07:36:20.6677581Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-03-14T07:36:20.6678102Z File "/tmp/jenkins_pyg/tmpxm84vu9v.py", line 188, in propagate 2025-03-14T07:36:20.6678673Z out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index) 2025-03-14T07:36:20.6679494Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-03-14T07:36:20.6680239Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-03-14T07:36:20.6681270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-03-14T07:36:20.6681936Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-03-14T07:36:20.6682593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-03-14T07:36:20.6683254Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-03-14T07:36:20.6684050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-03-14T07:36:20.6684669Z return scatter(x, index, dim, dim_size, reduce) 2025-03-14T07:36:20.6685282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-03-14T07:36:20.6685926Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-03-14T07:36:20.6686198Z 2025-03-14T07:36:20.6686202Z 2025-03-14T07:36:21.1243443Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmptu47i6us.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-03-14T07:36:21.1244439Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-03-14T07:36:21.1244931Z File "/tmp/jenkins_pyg/tmptu47i6us.py", line 188, in propagate 2025-03-14T07:36:21.1245493Z out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index) 2025-03-14T07:36:21.1246336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-03-14T07:36:21.1247083Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-03-14T07:36:21.1247772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-03-14T07:36:21.1248442Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-03-14T07:36:21.1249124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-03-14T07:36:21.1249806Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-03-14T07:36:21.1250474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-03-14T07:36:21.1251144Z return scatter(x, index, dim, dim_size, reduce) 2025-03-14T07:36:21.1251767Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-03-14T07:36:21.1252432Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-03-14T07:36:21.1252704Z 2025-03-14T07:36:21.1252716Z 2025-03-14T07:36:21.3909868Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmptu47i6us.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-03-14T07:36:21.3910850Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-03-14T07:36:21.3911369Z File "/tmp/jenkins_pyg/tmptu47i6us.py", line 188, in propagate 2025-03-14T07:36:21.3911982Z out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index) 2025-03-14T07:36:21.3912799Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-03-14T07:36:21.3913542Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-03-14T07:36:21.3914241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-03-14T07:36:21.3914914Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-03-14T07:36:21.3915574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-03-14T07:36:21.3916239Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-03-14T07:36:21.3917267Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-03-14T07:36:21.3917878Z return scatter(x, index, dim, dim_size, reduce) 2025-03-14T07:36:21.3918493Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-03-14T07:36:21.3919139Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-03-14T07:36:21.3919409Z 2025-03-14T07:36:21.3919414Z 2025-03-14T07:36:21.9983038Z W0314 07:36:21.997000 61555 site-packages/torch/_logging/_internal.py:1130] [18/0] Profiler function will be ignored 2025-03-14T07:36:25.3301455Z Compilation time (from dynamo_timed): 5.772798531 2025-03-14T07:36:25.3301967Z pass 2025-03-14T07:36:25.3307890Z TIMING: entire_frame_compile:5.30471 gc:0.00929 _recursive_pre_grad_passes:0.01079 _recursive_joint_graph_passes:0.17506 inductor_compile:3.16179 backend_compile:4.33513 _recursive_post_grad_passes:0.03762 async_compile.wait:2.05991 code_gen:2.58343 cudagraphify.get_container:0.16299 pad_mm_benchmark:0.00487 entire_backward_compile:0.46809 CUDAGraphNode.record:1.34361 total_wall_time:5.7728 2025-03-14T07:36:25.3309825Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:2926 | FakeTensor.__torch_dispatch__:463 | ProxyTorchDispatchMode.__torch_dispatch__:1055 2025-03-14T07:36:25.3310657Z Dynamo produced 10 graphs covering 112 ops with 13 graph breaks (6 unique) 2025-03-14T07:36:30.7058795Z 2025-03-14T07:36:32.8056569Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:36:32.8057303Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:36:32.8057955Z cuda train basic_gnn_gin 2025-03-14T07:36:35.6732959Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:36:35.6734957Z pred = mod(*cloned_inputs) 2025-03-14T07:36:35.6735575Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward 2025-03-14T07:36:35.6736208Z x = self.convs[i](x, edge_index) 2025-03-14T07:36:35.6736606Z File "/tmp/jenkins_pyg/tmpvpzggykk.py", line 225, in forward 2025-03-14T07:36:35.6737046Z out = self.propagate(edge_index, x=x, size=size) 2025-03-14T07:36:35.6737484Z File "/tmp/jenkins_pyg/tmpvpzggykk.py", line 182, in propagate 2025-03-14T07:36:35.6738057Z out = self.aggregate(out, index=kwargs.index, dim_size=kwargs.dim_size, ptr=kwargs.ptr) 2025-03-14T07:36:35.6738870Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-03-14T07:36:35.6739604Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-03-14T07:36:35.6740287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-03-14T07:36:35.6740956Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-03-14T07:36:35.6741615Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-03-14T07:36:35.6742279Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-03-14T07:36:35.6742940Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-03-14T07:36:35.6743550Z return scatter(x, index, dim, dim_size, reduce) 2025-03-14T07:36:35.6744169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-03-14T07:36:35.6744815Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-03-14T07:36:35.6745091Z 2025-03-14T07:36:35.6745096Z 2025-03-14T07:36:36.3891544Z W0314 07:36:36.388000 61887 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:36:39.9164003Z Compilation time (from dynamo_timed): 6.044359741 2025-03-14T07:36:39.9164382Z pass 2025-03-14T07:36:39.9192646Z TIMING: entire_frame_compile:5.51892 gc:0.00277 _recursive_pre_grad_passes:0.00869 pad_mm_benchmark:0.01019 _recursive_joint_graph_passes:0.18943 _recursive_post_grad_passes:0.07733 async_compile.wait:1.24843 code_gen:2.10576 inductor_compile:2.87318 backend_compile:4.19722 entire_backward_compile:0.52544 cudagraphify.get_container:0.20788 CUDAGraphNode.record:0.20044 total_wall_time:6.04436 2025-03-14T07:36:39.9194843Z STATS: call_* op count: 135 | FakeTensorMode.__torch_dispatch__:4239 | ProxyTorchDispatchMode.__torch_dispatch__:1846 | FakeTensor.__torch_dispatch__:774 2025-03-14T07:36:39.9195681Z Dynamo produced 3 graphs covering 135 ops with 7 graph breaks (5 unique) 2025-03-14T07:36:45.3193812Z 2025-03-14T07:36:47.4056677Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:36:47.4057041Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:36:47.4057408Z cuda train basic_gnn_sage 2025-03-14T07:36:50.2227637Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:36:50.2229102Z pred = mod(*cloned_inputs) 2025-03-14T07:36:50.2229908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward 2025-03-14T07:36:50.2230798Z x = self.convs[i](x, edge_index) 2025-03-14T07:36:50.2231222Z File "/tmp/jenkins_pyg/tmpep955g9_.py", line 228, in forward 2025-03-14T07:36:50.2231654Z out = self.propagate(edge_index, x=x, size=size) 2025-03-14T07:36:50.2232092Z File "/tmp/jenkins_pyg/tmpep955g9_.py", line 182, in propagate 2025-03-14T07:36:50.2232658Z out = self.aggregate(out, ptr=kwargs.ptr, dim_size=kwargs.dim_size, index=kwargs.index) 2025-03-14T07:36:50.2233484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-03-14T07:36:50.2234231Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-03-14T07:36:50.2234913Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-03-14T07:36:50.2235586Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-03-14T07:36:50.2236255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 34, in forward 2025-03-14T07:36:50.2236927Z return self.reduce(x, index, ptr, dim_size, dim, reduce='mean') 2025-03-14T07:36:50.2237594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-03-14T07:36:50.2238206Z return scatter(x, index, dim, dim_size, reduce) 2025-03-14T07:36:50.2238823Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 78, in scatter 2025-03-14T07:36:50.2239485Z count.scatter_add_(0, index, src.new_ones(src.size(dim))) 2025-03-14T07:36:50.2239763Z 2025-03-14T07:36:50.2239768Z 2025-03-14T07:36:50.8067588Z W0314 07:36:50.805000 62134 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:36:54.0325698Z Compilation time (from dynamo_timed): 5.589447238 2025-03-14T07:36:54.0326084Z pass 2025-03-14T07:36:54.0356480Z TIMING: entire_frame_compile:5.17403 gc:0.00358 _recursive_pre_grad_passes:0.00817 pad_mm_benchmark:0.0097 _recursive_joint_graph_passes:0.18386 _recursive_post_grad_passes:0.06064 async_compile.wait:1.44112 code_gen:2.12808 inductor_compile:2.75987 backend_compile:4.2228 entire_backward_compile:0.41541 cudagraphify.get_container:0.1996 CUDAGraphNode.record:0.19334 total_wall_time:5.58945 2025-03-14T07:36:54.0358364Z STATS: call_* op count: 126 | FakeTensorMode.__torch_dispatch__:3576 | ProxyTorchDispatchMode.__torch_dispatch__:1471 | FakeTensor.__torch_dispatch__:591 2025-03-14T07:36:54.0359533Z Dynamo produced 3 graphs covering 126 ops with 7 graph breaks (5 unique) 2025-03-14T07:36:59.4605558Z 2025-03-14T07:37:00.0578475Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:37:00.0578838Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:37:00.0579176Z cuda train dcgan 2025-03-14T07:37:05.4643269Z Compilation time (from dynamo_timed): 3.60564644 2025-03-14T07:37:05.4643643Z pass 2025-03-14T07:37:05.4664125Z TIMING: entire_frame_compile:2.70953 gc:0.00192 _recursive_pre_grad_passes:0.00386 _recursive_joint_graph_passes:0.14297 _recursive_post_grad_passes:0.19992 async_compile.wait:0.51144 code_gen:1.09398 inductor_compile:1.71936 backend_compile:2.15216 cudagraphify.get_container:0.1648 entire_backward_compile:0.89612 CUDAGraphNode.record:0.33958 total_wall_time:3.60565 2025-03-14T07:37:05.4665920Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:2050 | FakeTensor.__torch_dispatch__:382 | ProxyTorchDispatchMode.__torch_dispatch__:834 2025-03-14T07:37:05.4666850Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2025-03-14T07:37:10.7699355Z 2025-03-14T07:37:14.4766867Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:37:14.4767321Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:37:14.4767715Z cuda train demucs 2025-03-14T07:38:18.8474189Z Compilation time (from dynamo_timed): 11.56135177 2025-03-14T07:38:18.8481515Z pass 2025-03-14T07:38:18.8744436Z TIMING: entire_frame_compile:7.24733 gc:0.00399 _recursive_pre_grad_passes:0.008 _recursive_joint_graph_passes:0.17469 inductor_compile:7.43392 backend_compile:5.24562 _recursive_post_grad_passes:0.12281 async_compile.precompile:0.45129 async_compile.wait:0.64607 code_gen:5.64479 cudagraphify.get_container:0.16871 entire_backward_compile:4.31402 CUDAGraphNode.record:1.1742 total_wall_time:11.56135 2025-03-14T07:38:18.8746363Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:6255 | ProxyTorchDispatchMode.__torch_dispatch__:2367 | FakeTensor.__torch_dispatch__:1045 2025-03-14T07:38:18.8747265Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2025-03-14T07:38:24.5419824Z 2025-03-14T07:38:26.7728906Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:38:26.7729381Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:38:26.7729717Z cuda train densenet121 2025-03-14T07:40:11.0176889Z Compilation time (from dynamo_timed): 95.019250086 2025-03-14T07:40:11.0227770Z pass 2025-03-14T07:40:11.0309650Z TIMING: entire_frame_compile:49.5903 gc:0.00365 _recursive_pre_grad_passes:0.01508 pad_mm_benchmark:0.21651 _recursive_joint_graph_passes:0.75822 _recursive_post_grad_passes:0.92963 async_compile.wait:15.03948 code_gen:48.9581 inductor_compile:62.92209 backend_compile:36.24074 cudagraphify.get_container:0.63061 entire_backward_compile:45.42895 CUDAGraphNode.record:2.08907 total_wall_time:95.01925 2025-03-14T07:40:11.0311686Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:59628 | ProxyTorchDispatchMode.__torch_dispatch__:30355 | FakeTensor.__torch_dispatch__:13117 2025-03-14T07:40:11.0312524Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2025-03-14T07:40:19.3638684Z 2025-03-14T07:40:21.8920847Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:40:21.8921201Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:40:21.8921544Z cuda train detectron2_maskrcnn_r_50_c4 2025-03-14T07:40:21.8936166Z Traceback (most recent call last): 2025-03-14T07:40:21.8936732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1913, in validate_model 2025-03-14T07:40:21.8937295Z self.model_iter_fn(model, example_inputs) 2025-03-14T07:40:21.8937921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in forward_and_backward_pass 2025-03-14T07:40:21.8938528Z pred = mod(*cloned_inputs) 2025-03-14T07:40:21.8939508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:40:21.8940147Z return self._call_impl(*args, **kwargs) 2025-03-14T07:40:21.8940745Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:40:21.8941336Z return forward_call(*args, **kwargs) 2025-03-14T07:40:21.8941965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2025-03-14T07:40:21.8942757Z return self.inference(batched_inputs) 2025-03-14T07:40:21.8943402Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2025-03-14T07:40:21.8944111Z results, _ = self.roi_heads(images, features, proposals, None) 2025-03-14T07:40:21.8944813Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:40:21.8945453Z return self._call_impl(*args, **kwargs) 2025-03-14T07:40:21.8946045Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:40:21.8946721Z return forward_call(*args, **kwargs) 2025-03-14T07:40:21.8947362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2025-03-14T07:40:21.8948018Z box_features = self._shared_roi_transform( 2025-03-14T07:40:21.8948733Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2025-03-14T07:40:21.8949427Z x = self.pooler(features, boxes) 2025-03-14T07:40:21.8950053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:40:21.8950692Z return self._call_impl(*args, **kwargs) 2025-03-14T07:40:21.8951287Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:40:21.8951876Z return forward_call(*args, **kwargs) 2025-03-14T07:40:21.8952466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2025-03-14T07:40:21.8953108Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2025-03-14T07:40:21.8953777Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:40:21.8954416Z return self._call_impl(*args, **kwargs) 2025-03-14T07:40:21.8955010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:40:21.8955603Z return forward_call(*args, **kwargs) 2025-03-14T07:40:21.8956180Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2025-03-14T07:40:21.8956753Z return roi_align( 2025-03-14T07:40:21.8957300Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2025-03-14T07:40:21.8958100Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2025-03-14T07:40:21.8958895Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2025-03-14T07:40:21.8959673Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2025-03-14T07:40:21.8960495Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2025-03-14T07:40:21.8961137Z v1 = masked_index(y_low, x_low) 2025-03-14T07:40:21.8961726Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2025-03-14T07:40:21.8962311Z return input[ 2025-03-14T07:40:21.8964298Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21908.58 GiB. GPU 0 has a total capacity of 21.98 GiB of which 20.22 GiB is free. Process 897216 has 1.75 GiB memory in use. Of the allocated memory 1.38 GiB is allocated by PyTorch, and 77.04 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2025-03-14T07:40:21.8966205Z 2025-03-14T07:40:21.8966542Z The above exception was the direct cause of the following exception: 2025-03-14T07:40:21.8966860Z 2025-03-14T07:40:21.8966989Z Traceback (most recent call last): 2025-03-14T07:40:21.8967468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3991, in run 2025-03-14T07:40:21.8967946Z ) = runner.load_model( 2025-03-14T07:40:21.8968438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2025-03-14T07:40:21.8968999Z self.validate_model(model, example_inputs) 2025-03-14T07:40:21.8969549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1915, in validate_model 2025-03-14T07:40:21.8970112Z raise RuntimeError("Eager run failed") from e 2025-03-14T07:40:21.8970467Z RuntimeError: Eager run failed 2025-03-14T07:40:21.8970668Z 2025-03-14T07:40:21.8970769Z eager_fail_to_run 2025-03-14T07:40:26.2901937Z 2025-03-14T07:40:35.2611040Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:40:35.2611417Z loading model: 0it [00:08, ?it/s] 2025-03-14T07:40:35.2611760Z cuda train dlrm 2025-03-14T07:40:41.4995527Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2025-03-14T07:40:41.4996657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:40:41.4997364Z pred = mod(*cloned_inputs) 2025-03-14T07:40:41.4998014Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward 2025-03-14T07:40:41.4998699Z return self.sequential_forward(dense_x, lS_o, lS_i) 2025-03-14T07:40:41.4999428Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward 2025-03-14T07:40:41.5000132Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2025-03-14T07:40:41.5000806Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb 2025-03-14T07:40:41.5001509Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2025-03-14T07:40:41.5001780Z 2025-03-14T07:40:42.0379148Z Compilation time (from dynamo_timed): 4.782366977000001 2025-03-14T07:40:42.0379974Z pass 2025-03-14T07:40:42.0784422Z TIMING: entire_frame_compile:4.08258 gc:0.0015 _recursive_pre_grad_passes:0.00463 pad_mm_benchmark:0.90818 _recursive_joint_graph_passes:1.08552 _recursive_post_grad_passes:0.07654 async_compile.wait:0.26042 code_gen:0.83993 inductor_compile:1.6365 backend_compile:3.34893 cudagraphify.get_container:0.18761 entire_backward_compile:0.69979 CUDAGraphNode.record:0.18266 total_wall_time:4.78237 2025-03-14T07:40:42.0788291Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:3526 | ProxyTorchDispatchMode.__torch_dispatch__:1383 | FakeTensor.__torch_dispatch__:316 2025-03-14T07:40:42.0789901Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2025-03-14T07:40:47.3899695Z 2025-03-14T07:40:49.2006924Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:40:49.2007308Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:40:49.2007633Z cuda train drq 2025-03-14T07:40:55.2463829Z W0314 07:40:55.245000 67351 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:41:00.9394089Z Compilation time (from dynamo_timed): 10.389013579999999 2025-03-14T07:41:00.9395196Z pass 2025-03-14T07:41:00.9441152Z TIMING: entire_frame_compile:9.39679 gc:0.00441 _recursive_pre_grad_passes:0.01083 pad_mm_benchmark:0.56655 _recursive_joint_graph_passes:0.75204 _recursive_post_grad_passes:0.07855 async_compile.wait:3.28153 code_gen:4.4627 inductor_compile:5.59092 backend_compile:7.58102 cudagraphify.get_container:0.18614 entire_backward_compile:0.99222 total_wall_time:10.38901 2025-03-14T07:41:00.9443143Z STATS: call_* op count: 174 | FakeTensorMode.__torch_dispatch__:5637 | FakeTensor.__torch_dispatch__:1277 | ProxyTorchDispatchMode.__torch_dispatch__:2245 2025-03-14T07:41:00.9443964Z Dynamo produced 5 graphs covering 174 ops with 7 graph breaks (5 unique) 2025-03-14T07:41:06.5525469Z 2025-03-14T07:41:08.4705590Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:41:08.4705963Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:41:08.4706299Z cuda train fastNLP_Bert 2025-03-14T07:41:15.2190843Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] Graph break from `Tensor.item()`, consider setting: 2025-03-14T07:41:15.2191895Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] torch._dynamo.config.capture_scalar_outputs = True 2025-03-14T07:41:15.2192702Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] or: 2025-03-14T07:41:15.2193480Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2025-03-14T07:41:15.2194412Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] to include these operations in the captured graph. 2025-03-14T07:41:15.2195188Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] 2025-03-14T07:41:15.2195908Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] Graph break: from user code at: 2025-03-14T07:41:15.2197147Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 468, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:41:15.2198361Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] pred = mod(*cloned_inputs) 2025-03-14T07:41:15.2199445Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward 2025-03-14T07:41:15.2200539Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] sequence_output = self.bert(words) 2025-03-14T07:41:15.2201701Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward 2025-03-14T07:41:15.2202843Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] outputs = self.model(words) 2025-03-14T07:41:15.2203981Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward 2025-03-14T07:41:15.2208216Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item() # 表示word piece的长度(包括padding) 2025-03-14T07:41:15.2209204Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] 2025-03-14T07:41:15.2209812Z W0314 07:41:15.217000 67825 site-packages/torch/_dynamo/variables/tensor.py:913] [2/0] 2025-03-14T07:41:44.4120850Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:41:44.4121605Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2025-03-14T07:41:44.4122830Z embedding_output = self.embeddings(input_ids, token_type_ids) 2025-03-14T07:41:44.4123524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2025-03-14T07:41:44.4124172Z words_embeddings = self.word_embeddings(input_ids) 2025-03-14T07:41:44.4124434Z 2025-03-14T07:41:47.1873650Z Compilation time (from dynamo_timed): 28.981338114000003 2025-03-14T07:41:47.1896719Z pass 2025-03-14T07:41:47.2584205Z TIMING: entire_frame_compile:20.35767 gc:0.00568 _recursive_pre_grad_passes:0.01088 _recursive_joint_graph_passes:2.03278 inductor_compile:15.90683 backend_compile:17.05999 _recursive_post_grad_passes:0.86378 async_compile.precompile:0.22124 async_compile.wait:1.59713 code_gen:9.93855 cudagraphify.get_container:0.17608 pad_mm_benchmark:0.90924 CachingAutotuner.benchmark_all_configs:0.15328 entire_backward_compile:8.62367 CUDAGraphNode.record:2.3873 total_wall_time:28.98134 2025-03-14T07:41:47.2586849Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:36811 | ProxyTorchDispatchMode.__torch_dispatch__:19075 | FakeTensor.__torch_dispatch__:4829 2025-03-14T07:41:47.2587684Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2025-03-14T07:41:53.5894641Z 2025-03-14T07:41:54.6822674Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:41:54.6823029Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:41:54.6823366Z cuda train functorch_dp_cifar10 2025-03-14T07:42:16.1417907Z W0314 07:42:16.140000 68374 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:42:30.3097955Z Compilation time (from dynamo_timed): 32.485785958 2025-03-14T07:42:30.3105514Z pass 2025-03-14T07:42:30.3167080Z TIMING: entire_frame_compile:22.10351 gc:0.00417 _recursive_pre_grad_passes:0.01426 pad_mm_benchmark:0.22697 _recursive_joint_graph_passes:0.70995 _recursive_post_grad_passes:0.27141 async_compile.wait:4.94991 code_gen:14.94586 inductor_compile:20.65057 backend_compile:17.82373 cudagraphify.get_container:0.23196 entire_backward_compile:10.38228 CUDAGraphNode.record:1.16779 total_wall_time:32.48579 2025-03-14T07:42:30.3170878Z STATS: call_* op count: 402 | FakeTensorMode.__torch_dispatch__:19033 | ProxyTorchDispatchMode.__torch_dispatch__:8437 | FakeTensor.__torch_dispatch__:5816 2025-03-14T07:42:30.3172538Z Dynamo produced 3 graphs covering 402 ops with 7 graph breaks (5 unique) 2025-03-14T07:42:36.6549672Z 2025-03-14T07:42:37.4058627Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:42:37.4058997Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:42:37.4059347Z cuda train functorch_maml_omniglot 2025-03-14T07:42:45.0540576Z W0314 07:42:45.053000 69421 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:42:49.5592059Z Compilation time (from dynamo_timed): 10.105713479 2025-03-14T07:42:49.5592643Z pass 2025-03-14T07:42:49.5594363Z TIMING: entire_frame_compile:7.10273 gc:0.0028 _recursive_pre_grad_passes:0.00863 pad_mm_benchmark:0.21792 _recursive_joint_graph_passes:0.3821 _recursive_post_grad_passes:0.05401 async_compile.wait:2.44437 code_gen:5.68947 inductor_compile:6.83609 backend_compile:6.04466 cudagraphify.get_container:0.16767 entire_backward_compile:3.00299 CUDAGraphNode.record:0.59262 total_wall_time:10.10571 2025-03-14T07:42:49.5596511Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:4027 | ProxyTorchDispatchMode.__torch_dispatch__:1548 | FakeTensor.__torch_dispatch__:1035 2025-03-14T07:42:49.5597428Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2025-03-14T07:42:54.9929655Z 2025-03-14T07:42:56.7117333Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:42:56.7117682Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:42:56.7118023Z cuda train hf_Albert 2025-03-14T07:43:28.7274229Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:43:28.7276987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:43:28.7277727Z pred = mod(**cloned_inputs) 2025-03-14T07:43:28.7278390Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2025-03-14T07:43:28.7279058Z outputs = self.albert( 2025-03-14T07:43:28.7279933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2025-03-14T07:43:28.7280619Z embedding_output = self.embeddings( 2025-03-14T07:43:28.7281293Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2025-03-14T07:43:28.7281991Z inputs_embeds = self.word_embeddings(input_ids) 2025-03-14T07:43:28.7282244Z 2025-03-14T07:43:28.9547547Z W0314 07:43:28.953000 69894 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:43:35.8339969Z Compilation time (from dynamo_timed): 30.14940422 2025-03-14T07:43:35.8349192Z pass 2025-03-14T07:43:35.8555971Z TIMING: entire_frame_compile:22.01352 gc:0.00363 _recursive_pre_grad_passes:0.01409 pad_mm_benchmark:0.16725 _recursive_joint_graph_passes:2.10894 _recursive_post_grad_passes:0.57833 async_compile.wait:2.86172 code_gen:9.94847 inductor_compile:16.01405 backend_compile:17.70413 cudagraphify.get_container:0.31134 entire_backward_compile:8.13589 CachingAutotuner.benchmark_all_configs:0.06504 CUDAGraphNode.record:0.88835 total_wall_time:30.1494 2025-03-14T07:43:35.8558126Z STATS: call_* op count: 602 | FakeTensorMode.__torch_dispatch__:43033 | FakeTensor.__torch_dispatch__:5995 | ProxyTorchDispatchMode.__torch_dispatch__:21902 2025-03-14T07:43:35.8558940Z Dynamo produced 3 graphs covering 602 ops with 6 graph breaks (5 unique) 2025-03-14T07:43:42.3066089Z 2025-03-14T07:43:46.5788532Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:43:46.5789016Z loading model: 0it [00:04, ?it/s] 2025-03-14T07:43:46.5789459Z cuda train hf_Bart 2025-03-14T07:44:28.5222486Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:44:28.5223592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:44:28.5224305Z pred = mod(**cloned_inputs) 2025-03-14T07:44:28.5224986Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2025-03-14T07:44:28.5225640Z outputs = self.model( 2025-03-14T07:44:28.5226670Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2025-03-14T07:44:28.5227334Z encoder_outputs = self.encoder( 2025-03-14T07:44:28.5227983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2025-03-14T07:44:28.5228710Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2025-03-14T07:44:28.5229007Z 2025-03-14T07:44:28.6875974Z W0314 07:44:28.686000 70496 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:45:15.8889183Z Compilation time (from dynamo_timed): 75.447345921 2025-03-14T07:45:15.8918566Z pass 2025-03-14T07:45:15.9401551Z TIMING: entire_frame_compile:66.3589 gc:0.00517 _recursive_pre_grad_passes:0.0404 pad_mm_benchmark:0.51082 _recursive_joint_graph_passes:1.85139 _recursive_post_grad_passes:1.17304 async_compile.wait:3.12763 code_gen:22.73323 inductor_compile:37.86315 backend_compile:51.54797 cudagraphify.get_container:0.3773 entire_backward_compile:9.08845 CUDAGraphNode.record:2.0799 total_wall_time:75.44735 2025-03-14T07:45:15.9405062Z STATS: call_* op count: 1780 | FakeTensorMode.__torch_dispatch__:74139 | FakeTensor.__torch_dispatch__:17204 | ProxyTorchDispatchMode.__torch_dispatch__:34392 2025-03-14T07:45:15.9406396Z Dynamo produced 3 graphs covering 1780 ops with 6 graph breaks (5 unique) 2025-03-14T07:45:24.5406555Z 2025-03-14T07:45:27.5796623Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:45:27.5797103Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:45:27.5797555Z cuda train hf_Bert 2025-03-14T07:46:02.8258680Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:46:02.8259867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:46:02.8260580Z pred = mod(**cloned_inputs) 2025-03-14T07:46:02.8261231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2025-03-14T07:46:02.8261879Z outputs = self.bert( 2025-03-14T07:46:02.8262500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2025-03-14T07:46:02.8263158Z embedding_output = self.embeddings( 2025-03-14T07:46:02.8263808Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2025-03-14T07:46:02.8264492Z inputs_embeds = self.word_embeddings(input_ids) 2025-03-14T07:46:02.8264742Z 2025-03-14T07:46:03.0004272Z W0314 07:46:02.999000 71040 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:46:40.3420884Z Compilation time (from dynamo_timed): 61.653753868 2025-03-14T07:46:40.3445525Z pass 2025-03-14T07:46:40.3875640Z TIMING: entire_frame_compile:55.0895 gc:0.00504 _recursive_pre_grad_passes:0.03351 pad_mm_benchmark:0.43806 _recursive_joint_graph_passes:2.2881 _recursive_post_grad_passes:0.66192 async_compile.wait:2.68799 code_gen:17.46067 inductor_compile:28.87745 backend_compile:41.75884 cudagraphify.get_container:0.35017 entire_backward_compile:6.56425 CUDAGraphNode.record:1.66248 total_wall_time:61.65375 2025-03-14T07:46:40.3877650Z STATS: call_* op count: 1403 | FakeTensorMode.__torch_dispatch__:63486 | FakeTensor.__torch_dispatch__:13415 | ProxyTorchDispatchMode.__torch_dispatch__:29552 2025-03-14T07:46:40.3878479Z Dynamo produced 3 graphs covering 1403 ops with 6 graph breaks (5 unique) 2025-03-14T07:46:48.2465347Z 2025-03-14T07:46:53.3352690Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:46:53.3353270Z loading model: 0it [00:05, ?it/s] 2025-03-14T07:46:53.3353799Z cuda train hf_Bert_large 2025-03-14T07:48:04.0583732Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:48:04.0586271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:48:04.0587323Z pred = mod(**cloned_inputs) 2025-03-14T07:48:04.0588219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2025-03-14T07:48:04.0589056Z outputs = self.bert( 2025-03-14T07:48:04.0589669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2025-03-14T07:48:04.0590358Z embedding_output = self.embeddings( 2025-03-14T07:48:04.0591052Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2025-03-14T07:48:04.0591742Z inputs_embeds = self.word_embeddings(input_ids) 2025-03-14T07:48:04.0591994Z 2025-03-14T07:48:04.2954233Z W0314 07:48:04.294000 71511 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:49:20.3098494Z Compilation time (from dynamo_timed): 121.243715091 2025-03-14T07:49:20.3142777Z pass 2025-03-14T07:49:20.4290861Z TIMING: entire_frame_compile:108.33982 gc:0.00358 _recursive_pre_grad_passes:0.06656 pad_mm_benchmark:0.60691 _recursive_joint_graph_passes:4.46516 _recursive_post_grad_passes:1.28651 async_compile.wait:4.60059 code_gen:36.15573 inductor_compile:58.3539 backend_compile:82.43883 cudagraphify.get_container:0.57418 entire_backward_compile:12.90389 CUDAGraphNode.record:3.32585 total_wall_time:121.24372 2025-03-14T07:49:20.4296110Z STATS: call_* op count: 2711 | FakeTensorMode.__torch_dispatch__:123474 | FakeTensor.__torch_dispatch__:26015 | ProxyTorchDispatchMode.__torch_dispatch__:57728 2025-03-14T07:49:20.4297848Z Dynamo produced 3 graphs covering 2711 ops with 6 graph breaks (5 unique) 2025-03-14T07:49:30.9900224Z 2025-03-14T07:49:35.1736203Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:49:35.1736571Z loading model: 0it [00:04, ?it/s] 2025-03-14T07:49:35.1736904Z cuda train hf_BigBird 2025-03-14T07:49:37.9527553Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2025-03-14T07:50:33.6668697Z W0314 07:50:33.665000 72003 site-packages/torch/_inductor/utils.py:1780] [2/0_1] DeviceCopy in input program 2025-03-14T07:50:58.2327995Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_1). Found from : 2025-03-14T07:50:58.2330240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:50:58.2330978Z pred = mod(**cloned_inputs) 2025-03-14T07:50:58.2331688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2025-03-14T07:50:58.2332367Z outputs = self.bert( 2025-03-14T07:50:58.2333002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2025-03-14T07:50:58.2333677Z encoder_outputs = self.encoder( 2025-03-14T07:50:58.2334354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2025-03-14T07:50:58.2335022Z layer_outputs = layer_module( 2025-03-14T07:50:58.2335676Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2025-03-14T07:50:58.2336366Z self_attention_outputs = self.attention( 2025-03-14T07:50:58.2337058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2025-03-14T07:50:58.2337721Z self_outputs = self.self( 2025-03-14T07:50:58.2338362Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2025-03-14T07:50:58.2339136Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2025-03-14T07:50:58.2340050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention 2025-03-14T07:50:58.2340820Z rand_attn = np.stack(rand_attn, axis=0) 2025-03-14T07:50:58.2341044Z 2025-03-14T07:51:25.2530316Z W0314 07:51:25.251000 72003 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:52:03.4787318Z Compilation time (from dynamo_timed): 137.32423194199998 2025-03-14T07:52:03.4822695Z pass 2025-03-14T07:52:03.5200626Z TIMING: entire_frame_compile:111.52947 gc:0.00558 _recursive_pre_grad_passes:0.06582 pad_mm_benchmark:1.44994 _recursive_joint_graph_passes:5.81471 _recursive_post_grad_passes:3.60082 async_compile.wait:4.26141 code_gen:46.57432 inductor_compile:72.09615 backend_compile:89.27168 entire_backward_compile:25.79477 cudagraphify.get_container:1.49177 CUDAGraphNode.record:1.43726 total_wall_time:137.32423 2025-03-14T07:52:03.5202559Z STATS: call_* op count: 4383 | FakeTensorMode.__torch_dispatch__:190521 | FakeTensor.__torch_dispatch__:22501 | ProxyTorchDispatchMode.__torch_dispatch__:86243 2025-03-14T07:52:03.5203894Z Dynamo produced 3 graphs covering 4383 ops with 6 graph breaks (5 unique) 2025-03-14T07:52:14.1894954Z 2025-03-14T07:52:17.6156174Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:52:17.6156539Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:52:17.6156866Z cuda train hf_DistilBert 2025-03-14T07:52:38.9203687Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:52:38.9206515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:52:38.9207261Z pred = mod(**cloned_inputs) 2025-03-14T07:52:38.9207958Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2025-03-14T07:52:38.9208657Z dlbrt_output = self.distilbert( 2025-03-14T07:52:38.9209366Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-03-14T07:52:38.9210199Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2025-03-14T07:52:38.9211038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2025-03-14T07:52:38.9211844Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2025-03-14T07:52:38.9212180Z 2025-03-14T07:52:39.0704134Z W0314 07:52:39.069000 72839 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:52:58.8740240Z Compilation time (from dynamo_timed): 34.23863063 2025-03-14T07:52:58.8751928Z pass 2025-03-14T07:52:58.9005122Z TIMING: entire_frame_compile:29.87719 gc:0.00439 _recursive_pre_grad_passes:0.02084 pad_mm_benchmark:0.12035 _recursive_joint_graph_passes:0.86026 _recursive_post_grad_passes:0.60501 async_compile.wait:2.73744 code_gen:10.38083 inductor_compile:17.25099 backend_compile:23.02656 cudagraphify.get_container:0.26671 entire_backward_compile:4.36144 CUDAGraphNode.record:0.96608 total_wall_time:34.23863 2025-03-14T07:52:58.9007036Z STATS: call_* op count: 754 | FakeTensorMode.__torch_dispatch__:33024 | FakeTensor.__torch_dispatch__:7395 | ProxyTorchDispatchMode.__torch_dispatch__:15200 2025-03-14T07:52:58.9007855Z Dynamo produced 3 graphs covering 754 ops with 6 graph breaks (5 unique) 2025-03-14T07:53:05.5298832Z 2025-03-14T07:53:09.1256001Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:53:09.1256924Z loading model: 0it [00:03, ?it/s] 2025-03-14T07:53:09.1257753Z cuda train hf_GPT2 2025-03-14T07:53:42.0270306Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:53:42.0271305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:53:42.0272122Z pred = mod(**cloned_inputs) 2025-03-14T07:53:42.0272849Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2025-03-14T07:53:42.0273599Z transformer_outputs = self.transformer( 2025-03-14T07:53:42.0274342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2025-03-14T07:53:42.0275059Z inputs_embeds = self.wte(input_ids) 2025-03-14T07:53:42.0275296Z 2025-03-14T07:53:42.1721579Z W0314 07:53:42.171000 73303 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:54:09.4171223Z Compilation time (from dynamo_timed): 52.502175625999996 2025-03-14T07:54:09.4193123Z pass 2025-03-14T07:54:09.4975224Z TIMING: entire_frame_compile:44.15232 gc:0.00467 _recursive_pre_grad_passes:0.03035 pad_mm_benchmark:0.41642 _recursive_joint_graph_passes:1.7083 _recursive_post_grad_passes:0.77945 async_compile.wait:2.90637 code_gen:17.35439 inductor_compile:27.29504 backend_compile:34.48934 cudagraphify.get_container:0.3448 entire_backward_compile:8.34985 CUDAGraphNode.record:1.38433 total_wall_time:52.50218 2025-03-14T07:54:09.4977586Z STATS: call_* op count: 1393 | FakeTensorMode.__torch_dispatch__:51882 | FakeTensor.__torch_dispatch__:11405 | ProxyTorchDispatchMode.__torch_dispatch__:23394 2025-03-14T07:54:09.4978419Z Dynamo produced 3 graphs covering 1393 ops with 6 graph breaks (5 unique) 2025-03-14T07:54:16.9523246Z 2025-03-14T07:54:30.4222809Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:54:30.4225174Z loading model: 0it [00:13, ?it/s] 2025-03-14T07:54:30.4225846Z cuda train hf_GPT2_large 2025-03-14T07:54:30.4453867Z Compilation time (from dynamo_timed): 0 2025-03-14T07:54:30.4454208Z pass_due_to_skip 2025-03-14T07:54:30.7112751Z TIMING: total_wall_time:0 2025-03-14T07:54:30.7113687Z STATS: call_* op count: 0 2025-03-14T07:54:30.7114977Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:54:35.0419992Z 2025-03-14T07:54:37.1729471Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:54:37.1729826Z loading model: 0it [00:02, ?it/s] 2025-03-14T07:54:37.1730158Z cuda train hf_Reformer 2025-03-14T07:54:45.2701363Z 2025-03-14T07:54:45.2701787Z class GraphModule(torch.nn.Module): 2025-03-14T07:54:45.2702273Z def forward(self, L_cloned_inputs_input_ids_: "i64[4, 4096][4096, 1]cuda:0"): 2025-03-14T07:54:45.2702814Z l_cloned_inputs_input_ids_ = L_cloned_inputs_input_ids_ 2025-03-14T07:54:45.2703174Z 2025-03-14T07:54:45.2703442Z # No stacktrace found for following nodes 2025-03-14T07:54:45.2704172Z _enter_autocast = torch.amp.autocast_mode._enter_autocast('cuda', None, True, None); _enter_autocast = None 2025-03-14T07:54:45.2704705Z 2025-03-14T07:54:45.2705651Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:91 in _get_least_common_mult_chunk_len, code: return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length) 2025-03-14T07:54:45.2706957Z least_common_mult_chunk_length: "i64[][]cpu" = torch__dynamo_utils_wrapped_lcm(64, 64) 2025-03-14T07:54:45.2707651Z 2025-03-14T07:54:45.2708492Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:2061 in forward, code: input_shape[-1] % least_common_mult_chunk_length != 0 2025-03-14T07:54:45.2709666Z wrapped_mod: "i64[][]cpu" = torch__dynamo_utils_wrapped_mod(4096, least_common_mult_chunk_length); least_common_mult_chunk_length = None 2025-03-14T07:54:45.2710480Z wrapped_ne: "b8[][]cpu" = torch__dynamo_utils_wrapped_ne(wrapped_mod, 0); wrapped_mod = wrapped_ne = None 2025-03-14T07:54:45.2710974Z 2025-03-14T07:54:45.2711114Z 2025-03-14T07:54:45.2711238Z class GraphModule(torch.nn.Module): 2025-03-14T07:54:45.2711677Z def forward(self, L_cloned_inputs_input_ids_: "i64[4, 4096][4096, 1]cuda:0"): 2025-03-14T07:54:45.2712165Z l_cloned_inputs_input_ids_ = L_cloned_inputs_input_ids_ 2025-03-14T07:54:45.2712520Z 2025-03-14T07:54:45.2712777Z # No stacktrace found for following nodes 2025-03-14T07:54:45.2713349Z _enter_autocast = torch.amp.autocast_mode._enter_autocast('cuda', None, True, None); _enter_autocast = None 2025-03-14T07:54:45.2713885Z 2025-03-14T07:54:45.2714813Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:91 in _get_least_common_mult_chunk_len, code: return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length) 2025-03-14T07:54:45.2715965Z least_common_mult_chunk_length: "i64[][]cpu" = torch__dynamo_utils_wrapped_lcm(64, 64) 2025-03-14T07:54:45.2716418Z 2025-03-14T07:54:45.2717203Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:2061 in forward, code: input_shape[-1] % least_common_mult_chunk_length != 0 2025-03-14T07:54:45.2718777Z wrapped_mod: "i64[][]cpu" = torch__dynamo_utils_wrapped_mod(4096, least_common_mult_chunk_length); least_common_mult_chunk_length = None 2025-03-14T07:54:45.2719593Z wrapped_ne: "b8[][]cpu" = torch__dynamo_utils_wrapped_ne(wrapped_mod, 0); wrapped_mod = wrapped_ne = None 2025-03-14T07:54:45.2720091Z 2025-03-14T07:54:46.1709339Z 2025-03-14T07:54:46.1710220Z class GraphModule(torch.nn.Module): 2025-03-14T07:54:46.1710699Z def forward(self, L_input_ids_: "i64[4, 4096][4096, 1]cuda:0"): 2025-03-14T07:54:46.1711098Z l_input_ids_ = L_input_ids_ 2025-03-14T07:54:46.1711389Z 2025-03-14T07:54:46.1712396Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:91 in _get_least_common_mult_chunk_len, code: return np.lcm(config.lsh_attn_chunk_length, config.local_attn_chunk_length) 2025-03-14T07:54:46.1713586Z least_common_mult_chunk_length: "i64[][]cpu" = torch__dynamo_utils_wrapped_lcm(64, 64) 2025-03-14T07:54:46.1714035Z 2025-03-14T07:54:46.1714831Z # File: /opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py:2061 in forward, code: input_shape[-1] % least_common_mult_chunk_length != 0 2025-03-14T07:54:46.1716004Z wrapped_mod: "i64[][]cpu" = torch__dynamo_utils_wrapped_mod(4096, least_common_mult_chunk_length); least_common_mult_chunk_length = None 2025-03-14T07:54:46.1716817Z wrapped_ne: "b8[][]cpu" = torch__dynamo_utils_wrapped_ne(wrapped_mod, 0); wrapped_mod = wrapped_ne = None 2025-03-14T07:54:46.1717312Z 2025-03-14T07:54:55.6260744Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-03-14T07:54:55.6261999Z attn_outputs = self.attention( 2025-03-14T07:54:55.6262682Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-03-14T07:54:55.6263405Z self_attention_outputs = self.self_attention( 2025-03-14T07:54:55.6264112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-03-14T07:54:55.6265044Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-03-14T07:54:55.6266062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-03-14T07:54:55.6267077Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-03-14T07:54:55.6267378Z 2025-03-14T07:54:55.6267388Z 2025-03-14T07:54:58.4307365Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-03-14T07:54:58.4309246Z attn_outputs = self.attention( 2025-03-14T07:54:58.4310262Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-03-14T07:54:58.4311412Z self_attention_outputs = self.self_attention( 2025-03-14T07:54:58.4312482Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-03-14T07:54:58.4313873Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-03-14T07:54:58.4315551Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-03-14T07:54:58.4317590Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-03-14T07:54:58.4318086Z 2025-03-14T07:54:58.4318093Z 2025-03-14T07:54:59.9258361Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-03-14T07:54:59.9259590Z attn_outputs = self.attention( 2025-03-14T07:54:59.9260655Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-03-14T07:54:59.9261399Z self_attention_outputs = self.self_attention( 2025-03-14T07:54:59.9262112Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-03-14T07:54:59.9263000Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-03-14T07:54:59.9264029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-03-14T07:54:59.9264943Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-03-14T07:54:59.9265248Z 2025-03-14T07:54:59.9265253Z 2025-03-14T07:55:03.1091401Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:55:03.1093343Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2025-03-14T07:55:03.1094964Z embedding_output = self.embeddings( 2025-03-14T07:55:03.1096206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2025-03-14T07:55:03.1096935Z inputs_embeds = self.word_embeddings(input_ids) 2025-03-14T07:55:03.1097194Z 2025-03-14T07:55:03.1217908Z W0314 07:55:03.120000 73962 site-packages/torch/_logging/_internal.py:1130] [32/0] Profiler function will be ignored 2025-03-14T07:55:24.0717842Z Compilation time (from dynamo_timed): 31.420383695 2025-03-14T07:55:24.0728007Z pass 2025-03-14T07:55:24.0879001Z TIMING: entire_frame_compile:30.1017 gc:0.01541 _recursive_pre_grad_passes:0.02736 _recursive_joint_graph_passes:0.48983 inductor_compile:17.215 backend_compile:23.27599 _recursive_post_grad_passes:0.15422 async_compile.wait:5.89328 code_gen:12.80233 cudagraphify.get_container:0.18866 pad_mm_benchmark:0.02672 entire_backward_compile:1.31868 CUDAGraphNode.record:6.27725 total_wall_time:31.42038 2025-03-14T07:55:24.0881067Z STATS: call_* op count: 844 | FakeTensorMode.__torch_dispatch__:23650 | ProxyTorchDispatchMode.__torch_dispatch__:9117 | FakeTensor.__torch_dispatch__:4375 | attempt fast:21 | fast is_contiguous:21 2025-03-14T07:55:24.0882041Z Dynamo produced 17 graphs covering 844 ops with 23 graph breaks (8 unique) 2025-03-14T07:55:30.4098757Z 2025-03-14T07:55:37.0888894Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:55:37.0889370Z loading model: 0it [00:06, ?it/s] 2025-03-14T07:55:37.0889817Z cuda train hf_Roberta_base 2025-03-14T07:55:41.5980558Z WARNING:common:fp64 golden ref were not generated for hf_Roberta_base. Setting accuracy check to cosine 2025-03-14T07:56:11.6983615Z skipping cudagraphs due to deterministic index put. Found from : 2025-03-14T07:56:11.6986384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in torch_dynamo_resume_in_forward_and_backward_pass_at_463 2025-03-14T07:56:11.6987208Z pred = mod(**cloned_inputs) 2025-03-14T07:56:11.6987908Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1092, in forward 2025-03-14T07:56:11.6988949Z outputs = self.roberta( 2025-03-14T07:56:11.6989612Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 830, in forward 2025-03-14T07:56:11.6990309Z embedding_output = self.embeddings( 2025-03-14T07:56:11.6991007Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 126, in forward 2025-03-14T07:56:11.6991777Z inputs_embeds = self.word_embeddings(input_ids) 2025-03-14T07:56:11.6992029Z 2025-03-14T07:56:11.9040957Z W0314 07:56:11.903000 74642 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:56:48.6181516Z Compilation time (from dynamo_timed): 61.963511996 2025-03-14T07:56:48.6193191Z fail_accuracy 2025-03-14T07:56:48.6362419Z TIMING: entire_frame_compile:55.2816 gc:0.00517 _recursive_pre_grad_passes:0.03364 pad_mm_benchmark:1.23026 _recursive_joint_graph_passes:3.08168 _recursive_post_grad_passes:0.67166 async_compile.wait:2.4344 code_gen:16.98075 inductor_compile:28.39027 backend_compile:41.96378 cudagraphify.get_container:0.36662 entire_backward_compile:6.68191 CUDAGraphNode.record:1.67091 total_wall_time:61.96351 2025-03-14T07:56:48.6364320Z STATS: call_* op count: 1411 | FakeTensorMode.__torch_dispatch__:63591 | FakeTensor.__torch_dispatch__:13431 | ProxyTorchDispatchMode.__torch_dispatch__:29586 2025-03-14T07:56:48.6365146Z Dynamo produced 3 graphs covering 1411 ops with 6 graph breaks (5 unique) 2025-03-14T07:56:56.5557709Z 2025-03-14T07:57:01.2290773Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:57:01.2291221Z loading model: 0it [00:04, ?it/s] 2025-03-14T07:57:01.2291607Z cuda train hf_T5_base 2025-03-14T07:57:04.3694415Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2025-03-14T07:57:06.3291572Z ERROR:common: 2025-03-14T07:57:06.3292083Z Traceback (most recent call last): 2025-03-14T07:57:06.3293006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2175, in check_accuracy 2025-03-14T07:57:06.3293938Z correct_rerun_result = self.run_n_iterations( 2025-03-14T07:57:06.3294776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1951, in run_n_iterations 2025-03-14T07:57:06.3295677Z model_iter_fn(mod, inputs, collect_outputs=False) 2025-03-14T07:57:06.3296699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 466, in forward_and_backward_pass 2025-03-14T07:57:06.3297664Z pred = mod(**cloned_inputs) 2025-03-14T07:57:06.3298634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:57:06.3299642Z return self._call_impl(*args, **kwargs) 2025-03-14T07:57:06.3300584Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:57:06.3301530Z return forward_call(*args, **kwargs) 2025-03-14T07:57:06.3302483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1748, in forward 2025-03-14T07:57:06.3303522Z decoder_outputs = self.decoder( 2025-03-14T07:57:06.3304544Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:57:06.3305550Z return self._call_impl(*args, **kwargs) 2025-03-14T07:57:06.3306660Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:57:06.3307643Z return forward_call(*args, **kwargs) 2025-03-14T07:57:06.3308721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1115, in forward 2025-03-14T07:57:06.3309825Z layer_outputs = layer_module( 2025-03-14T07:57:06.3310895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:57:06.3312549Z return self._call_impl(*args, **kwargs) 2025-03-14T07:57:06.3313578Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:57:06.3314608Z return forward_call(*args, **kwargs) 2025-03-14T07:57:06.3315681Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 695, in forward 2025-03-14T07:57:06.3316778Z self_attention_outputs = self.layer[0]( 2025-03-14T07:57:06.3318166Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:57:06.3319299Z return self._call_impl(*args, **kwargs) 2025-03-14T07:57:06.3320342Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:57:06.3321376Z return forward_call(*args, **kwargs) 2025-03-14T07:57:06.3322449Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward 2025-03-14T07:57:06.3323561Z attention_output = self.SelfAttention( 2025-03-14T07:57:06.3324666Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-03-14T07:57:06.3325717Z return self._call_impl(*args, **kwargs) 2025-03-14T07:57:06.3327097Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-03-14T07:57:06.3328162Z return forward_call(*args, **kwargs) 2025-03-14T07:57:06.3329231Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 562, in forward 2025-03-14T07:57:06.3330469Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as( 2025-03-14T07:57:06.3331616Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 2140, in softmax 2025-03-14T07:57:06.3332578Z ret = input.softmax(dim) 2025-03-14T07:57:06.3336025Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 30.44 MiB is free. Process 905841 has 21.94 GiB memory in use. Of the allocated memory 21.35 GiB is allocated by PyTorch, and 240.72 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2025-03-14T07:57:06.3542838Z Compilation time (from dynamo_timed): 0 2025-03-14T07:57:06.5287151Z eager_2nd_run_OOM 2025-03-14T07:57:06.5340738Z TIMING: total_wall_time:0 2025-03-14T07:57:06.5341272Z STATS: call_* op count: 0 2025-03-14T07:57:06.5341717Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:57:10.9426957Z 2025-03-14T07:57:21.7552423Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:57:21.7553420Z loading model: 0it [00:10, ?it/s] 2025-03-14T07:57:21.7554251Z cuda train hf_T5_large 2025-03-14T07:57:21.7792321Z Compilation time (from dynamo_timed): 0 2025-03-14T07:57:21.7792768Z pass_due_to_skip 2025-03-14T07:57:22.2323350Z TIMING: total_wall_time:0 2025-03-14T07:57:22.2323776Z STATS: call_* op count: 0 2025-03-14T07:57:22.2324216Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-03-14T07:57:26.6323450Z 2025-03-14T07:57:28.1366547Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:57:28.1367060Z loading model: 0it [00:01, ?it/s] 2025-03-14T07:57:28.1367433Z cuda train hf_Whisper 2025-03-14T07:57:46.7914034Z W0314 07:57:46.790000 75048 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-03-14T07:58:01.4768612Z Compilation time (from dynamo_timed): 27.788504386 2025-03-14T07:58:01.4771931Z pass 2025-03-14T07:58:01.4838381Z TIMING: entire_frame_compile:23.0366 gc:0.00386 _recursive_pre_grad_passes:0.01571 pad_mm_benchmark:0.25905 _recursive_joint_graph_passes:0.89152 _recursive_post_grad_passes:0.28876 async_compile.wait:5.60829 code_gen:12.19185 inductor_compile:16.49577 backend_compile:18.5924 cudagraphify.get_container:0.23825 CachingAutotuner.benchmark_all_configs:0.37453 entire_backward_compile:4.7519 CUDAGraphNode.record:1.12907 total_wall_time:27.7885 2025-03-14T07:58:01.4840690Z STATS: call_* op count: 488 | FakeTensorMode.__torch_dispatch__:20466 | FakeTensor.__torch_dispatch__:5035 | ProxyTorchDispatchMode.__torch_dispatch__:9170 2025-03-14T07:58:01.4841516Z Dynamo produced 3 graphs covering 488 ops with 6 graph breaks (5 unique) 2025-03-14T07:58:07.7487817Z 2025-03-14T07:58:07.9197371Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:58:07.9197732Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:58:07.9198065Z cuda train hf_distil_whisper 2025-03-14T07:58:07.9201762Z Traceback (most recent call last): 2025-03-14T07:58:07.9202299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3991, in run 2025-03-14T07:58:07.9202793Z ) = runner.load_model( 2025-03-14T07:58:07.9203290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2025-03-14T07:58:07.9203823Z benchmark = benchmark_cls( 2025-03-14T07:58:07.9204353Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__ 2025-03-14T07:58:07.9204923Z obj = type.__call__(cls, *args, **kwargs) 2025-03-14T07:58:07.9205603Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 13, in __init__ 2025-03-14T07:58:07.9206335Z raise NotImplementedError("Training is not implemented.") 2025-03-14T07:58:07.9206787Z NotImplementedError: Training is not implemented. 2025-03-14T07:58:07.9207042Z 2025-03-14T07:58:07.9207149Z model_fail_to_load 2025-03-14T07:58:12.0120414Z 2025-03-14T07:58:12.5721428Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:58:12.5721917Z loading model: 0it [00:00, ?it/s] 2025-03-14T07:58:12.5722309Z cuda train lennard_jones 2025-03-14T07:58:16.7460033Z W0314 07:58:16.745000 75792 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-03-14T07:58:19.6895623Z Compilation time (from dynamo_timed): 5.221771782 2025-03-14T07:58:19.6896013Z pass 2025-03-14T07:58:19.6897554Z TIMING: entire_frame_compile:4.86031 gc:0.00371 _recursive_pre_grad_passes:0.00807 pad_mm_benchmark:0.52299 _recursive_joint_graph_passes:0.69038 _recursive_post_grad_passes:0.06551 async_compile.precompile:0.15834 async_compile.wait:1.13285 code_gen:1.79467 inductor_compile:2.38108 backend_compile:4.1437 cudagraphify.get_container:0.15803 entire_backward_compile:0.36146 CUDAGraphNode.record:0.53534 total_wall_time:5.22177 2025-03-14T07:58:19.6899529Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:3052 | ProxyTorchDispatchMode.__torch_dispatch__:1304 | FakeTensor.__torch_dispatch__:671 2025-03-14T07:58:19.6900355Z Dynamo produced 3 graphs covering 82 ops with 7 graph breaks (5 unique) 2025-03-14T07:58:21.6362482Z accuracy pass_rate=79.41% 2025-03-14T07:58:21.6366791Z calls_captured gmean=0.00x mean=579.147x 2025-03-14T07:58:21.6370941Z unique_graphs gmean=0.00x mean=3.294x 2025-03-14T07:58:21.6375409Z graph_breaks gmean=0.00x mean=6.471x 2025-03-14T07:58:21.6379535Z unique_graph_breaks gmean=0.00x mean=4.324x 2025-03-14T07:58:21.6383546Z autograd_captures gmean=0.00x mean=0.000x 2025-03-14T07:58:21.6387994Z autograd_compiles gmean=0.00x mean=0.000x 2025-03-14T07:58:21.6392299Z cudagraph_skips gmean=0.00x mean=0.735x 2025-03-14T07:58:21.6393044Z compilation_latency mean=29.130 seconds 2025-03-14T07:58:23.2690350Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2025-03-14T07:58:23.5990943Z torchrec_dlrm PASS 2025-03-14T07:58:23.5994495Z BERT_pytorch PASS 2025-03-14T07:58:23.5999558Z Background_Matting XFAIL 2025-03-14T07:58:23.6004370Z LearningToPaint PASS 2025-03-14T07:58:23.6009169Z Super_SloMo PASS 2025-03-14T07:58:23.6013837Z alexnet PASS 2025-03-14T07:58:23.6018494Z basic_gnn_edgecnn PASS 2025-03-14T07:58:23.6023207Z basic_gnn_gcn PASS 2025-03-14T07:58:23.6027971Z basic_gnn_gin PASS 2025-03-14T07:58:23.6033280Z basic_gnn_sage PASS 2025-03-14T07:58:23.6037865Z dcgan PASS 2025-03-14T07:58:23.6042497Z demucs PASS 2025-03-14T07:58:23.6047131Z densenet121 PASS 2025-03-14T07:58:23.6051754Z detectron2_maskrcnn_r_50_c4 XFAIL 2025-03-14T07:58:23.6056256Z dlrm PASS 2025-03-14T07:58:23.6061046Z drq PASS 2025-03-14T07:58:23.6065573Z fastNLP_Bert PASS 2025-03-14T07:58:23.6070275Z functorch_dp_cifar10 PASS 2025-03-14T07:58:23.6074843Z functorch_maml_omniglot PASS 2025-03-14T07:58:23.6079556Z hf_Albert PASS 2025-03-14T07:58:23.6084153Z hf_Bart PASS 2025-03-14T07:58:23.6088726Z hf_Bert PASS 2025-03-14T07:58:23.6093397Z hf_Bert_large PASS 2025-03-14T07:58:23.6097946Z hf_BigBird PASS 2025-03-14T07:58:23.6102587Z hf_DistilBert PASS 2025-03-14T07:58:23.6107223Z hf_GPT2 PASS 2025-03-14T07:58:23.6111956Z hf_GPT2_large XFAIL 2025-03-14T07:58:23.6116561Z hf_Reformer PASS 2025-03-14T07:58:23.6121204Z hf_Roberta_base XFAIL 2025-03-14T07:58:23.6125761Z hf_T5_base XFAIL 2025-03-14T07:58:23.6130908Z hf_T5_large XFAIL 2025-03-14T07:58:23.6135510Z hf_Whisper PASS 2025-03-14T07:58:23.6139995Z hf_distil_whisper XFAIL 2025-03-14T07:58:23.6144661Z lennard_jones PASS 2025-03-14T07:58:23.6677459Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2025-03-14T07:58:23.9957751Z torchrec_dlrm PASS 2025-03-14T07:58:23.9962987Z BERT_pytorch PASS 2025-03-14T07:58:23.9967037Z Background_Matting PASS 2025-03-14T07:58:23.9971847Z LearningToPaint PASS 2025-03-14T07:58:23.9976636Z Super_SloMo PASS 2025-03-14T07:58:23.9981426Z alexnet PASS 2025-03-14T07:58:23.9986238Z basic_gnn_edgecnn PASS 2025-03-14T07:58:23.9991300Z basic_gnn_gcn PASS 2025-03-14T07:58:23.9995952Z basic_gnn_gin PASS 2025-03-14T07:58:24.0000739Z basic_gnn_sage PASS 2025-03-14T07:58:24.0005433Z dcgan PASS 2025-03-14T07:58:24.0010137Z demucs PASS 2025-03-14T07:58:24.0014811Z densenet121 PASS 2025-03-14T07:58:24.0019377Z detectron2_maskrcnn_r_50_c4 PASS 2025-03-14T07:58:24.0024156Z dlrm PASS 2025-03-14T07:58:24.0029417Z drq PASS 2025-03-14T07:58:24.0034071Z fastNLP_Bert PASS 2025-03-14T07:58:24.0038919Z functorch_dp_cifar10 PASS 2025-03-14T07:58:24.0043641Z functorch_maml_omniglot PASS 2025-03-14T07:58:24.0048280Z hf_Albert PASS 2025-03-14T07:58:24.0052933Z hf_Bart PASS 2025-03-14T07:58:24.0057534Z hf_Bert PASS 2025-03-14T07:58:24.0062443Z hf_Bert_large PASS 2025-03-14T07:58:24.0067030Z hf_BigBird PASS 2025-03-14T07:58:24.0072161Z hf_DistilBert PASS 2025-03-14T07:58:24.0076418Z hf_GPT2 PASS 2025-03-14T07:58:24.0081137Z hf_GPT2_large PASS 2025-03-14T07:58:24.0086021Z hf_Reformer PASS 2025-03-14T07:58:24.0090718Z hf_Roberta_base PASS 2025-03-14T07:58:24.0095329Z hf_T5_base PASS 2025-03-14T07:58:24.0099887Z hf_T5_large PASS 2025-03-14T07:58:24.0104679Z hf_Whisper PASS 2025-03-14T07:58:24.0109402Z hf_distil_whisper PASS 2025-03-14T07:58:24.0114179Z lennard_jones PASS 2025-03-14T07:58:24.0675981Z + cleanup_workspace 2025-03-14T07:58:24.0676524Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-03-14T07:58:24.0677272Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-03-14T07:58:24.0677885Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-03-14T07:58:24.0678356Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-03-14T07:58:24.0681938Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-03-14T07:58:24.0682543Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-03-14T07:58:24.0683014Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-03-14T07:58:26.1859832Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-03-14T07:58:26.1860306Z with: 2025-03-14T07:58:26.1860575Z benchmark-results-dir: test/test-reports 2025-03-14T07:58:26.1860909Z dry-run: false 2025-03-14T07:58:26.1861169Z schema-version: v3 2025-03-14T07:58:26.1861658Z github-token: *** 2025-03-14T07:58:26.1861919Z env: 2025-03-14T07:58:26.1862152Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:26.1862503Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:26.1863063Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:26.1863553Z ##[endgroup] 2025-03-14T07:58:26.1891848Z ##[group]Run set -eux 2025-03-14T07:58:26.1892133Z set -eux 2025-03-14T07:58:26.1892426Z python3 -mpip install boto3==1.35.33 2025-03-14T07:58:26.1907383Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:26.1907770Z env: 2025-03-14T07:58:26.1908028Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:26.1908379Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:26.1908941Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:26.1909431Z ##[endgroup] 2025-03-14T07:58:26.1945398Z + python3 -mpip install boto3==1.35.33 2025-03-14T07:58:26.4281066Z Defaulting to user installation because normal site-packages is not writeable 2025-03-14T07:58:26.4464528Z Requirement already satisfied: boto3==1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.33) 2025-03-14T07:58:26.4512165Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.99) 2025-03-14T07:58:26.4516131Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4) 2025-03-14T07:58:26.4521061Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-03-14T07:58:26.4569755Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-03-14T07:58:26.4580249Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-03-14T07:58:26.4617813Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-03-14T07:58:26.5866053Z ##[group]Run set -eux 2025-03-14T07:58:26.5866354Z set -eux 2025-03-14T07:58:26.5866739Z  2025-03-14T07:58:26.5867009Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-03-14T07:58:26.5867392Z  echo "Missing github-token input" 2025-03-14T07:58:26.5867732Z  exit 1 2025-03-14T07:58:26.5867980Z fi 2025-03-14T07:58:26.5878358Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:26.5878738Z env: 2025-03-14T07:58:26.5878977Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:26.5879322Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:26.5879878Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:26.5880628Z GITHUB_TOKEN: *** 2025-03-14T07:58:26.5880994Z ##[endgroup] 2025-03-14T07:58:26.5909451Z + [[ -z *** ]] 2025-03-14T07:58:26.5965374Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-03-14T07:58:26.5965871Z with: 2025-03-14T07:58:26.5966251Z github-token: *** 2025-03-14T07:58:26.5966498Z env: 2025-03-14T07:58:26.5966728Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:26.5967076Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:26.5967630Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:26.5968119Z ##[endgroup] 2025-03-14T07:58:26.5993361Z ##[group]Run set -eux 2025-03-14T07:58:26.5993643Z set -eux 2025-03-14T07:58:26.5993881Z  2025-03-14T07:58:26.5994360Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-03-14T07:58:26.6003307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:26.6003684Z env: 2025-03-14T07:58:26.6003906Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:26.6004251Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:26.6004810Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:26.6005624Z GITHUB_TOKEN: *** 2025-03-14T07:58:26.6005877Z ##[endgroup] 2025-03-14T07:58:26.6048334Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 13849515380 i-0166a710cfefd3e7e 2025-03-14T07:58:27.7064551Z setting job-id=38756916747 2025-03-14T07:58:27.7065128Z setting job-name=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T07:58:27.7173527Z ##[group]Run set -eux 2025-03-14T07:58:27.7173815Z set -eux 2025-03-14T07:58:27.7174110Z  2025-03-14T07:58:27.7174517Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-03-14T07:58:27.7175029Z  --schema-version "${SCHEMA_VERSION}" \ 2025-03-14T07:58:27.7175383Z  --repo "${REPO}" \ 2025-03-14T07:58:27.7175700Z  --head-branch "${HEAD_BRANCH}" \ 2025-03-14T07:58:27.7176042Z  --head-sha "${HEAD_SHA}" \ 2025-03-14T07:58:27.7176392Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-03-14T07:58:27.7176763Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-03-14T07:58:27.7177100Z  --job-id "${JOB_ID}" \ 2025-03-14T07:58:27.7177416Z  --job-name "${JOB_NAME}" 2025-03-14T07:58:27.7186218Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:27.7186737Z env: 2025-03-14T07:58:27.7186966Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:27.7187311Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:27.7187857Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:27.7188533Z SCHEMA_VERSION: v3 2025-03-14T07:58:27.7188797Z REPO: pytorch/pytorch 2025-03-14T07:58:27.7189072Z HEAD_BRANCH: refs/heads/main 2025-03-14T07:58:27.7189408Z HEAD_SHA: aed0b7a742a2d7b7901790622829cbd2135049a4 2025-03-14T07:58:27.7189758Z WORKFLOW_RUN_ID: 13849515380 2025-03-14T07:58:27.7190037Z RUN_ATTEMPT: 1 2025-03-14T07:58:27.7190283Z JOB_ID: 38756916747 2025-03-14T07:58:27.7190753Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T07:58:27.7191317Z ##[endgroup] 2025-03-14T07:58:27.7220976Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha aed0b7a742a2d7b7901790622829cbd2135049a4 --workflow-id 13849515380 --run-attempt 1 --job-id 38756916747 --job-name 'cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)' 2025-03-14T07:58:27.7555321Z ##[group]Run set -eux 2025-03-14T07:58:27.7555595Z set -eux 2025-03-14T07:58:27.7555837Z  2025-03-14T07:58:27.7556101Z # TODO (huydhn): Implement this part 2025-03-14T07:58:27.7556472Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2025-03-14T07:58:27.7564437Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:27.7564809Z env: 2025-03-14T07:58:27.7565032Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:27.7565371Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:27.7565923Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:27.7566610Z ##[endgroup] 2025-03-14T07:58:27.7596019Z + echo 'runners=[]' 2025-03-14T07:58:27.7626132Z ##[group]Run set -eux 2025-03-14T07:58:27.7626925Z set -eux 2025-03-14T07:58:27.7627172Z  2025-03-14T07:58:27.7627435Z # TODO (huydhn): Implement this part 2025-03-14T07:58:27.7627841Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-03-14T07:58:27.7635878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:27.7636255Z env: 2025-03-14T07:58:27.7636485Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:27.7636826Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:27.7637380Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:27.7637869Z ##[endgroup] 2025-03-14T07:58:27.7668800Z + echo 'dependencies={}' 2025-03-14T07:58:27.7699334Z ##[group]Run set -eux 2025-03-14T07:58:27.7699629Z set -eux 2025-03-14T07:58:27.7699875Z  2025-03-14T07:58:27.7700172Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-03-14T07:58:27.7700611Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-03-14T07:58:27.7701102Z  # We don't want the job to fail if the directory doesn't exist 2025-03-14T07:58:27.7701505Z  exit 0 2025-03-14T07:58:27.7701745Z fi 2025-03-14T07:58:27.7701971Z  2025-03-14T07:58:27.7702225Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-03-14T07:58:27.7702696Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-03-14T07:58:27.7703244Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-03-14T07:58:27.7703674Z  --metadata "${BENCHMARK_METADATA}" \ 2025-03-14T07:58:27.7704036Z  --runners "${RUNNER_INFO}" \ 2025-03-14T07:58:27.7704393Z  --dependencies "${DEPENDENCIES}" \ 2025-03-14T07:58:27.7704737Z  --dry-run 2025-03-14T07:58:27.7705006Z else 2025-03-14T07:58:27.7705397Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-03-14T07:58:27.7705936Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-03-14T07:58:27.7706363Z  --metadata "${BENCHMARK_METADATA}" \ 2025-03-14T07:58:27.7707026Z  --runners "${RUNNER_INFO}" \ 2025-03-14T07:58:27.7707382Z  --dependencies "${DEPENDENCIES}" 2025-03-14T07:58:27.7707708Z fi 2025-03-14T07:58:27.7715592Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:27.7715960Z env: 2025-03-14T07:58:27.7716186Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:27.7716524Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:27.7717073Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:27.7717595Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-03-14T07:58:27.7717915Z DRY_RUN: false 2025-03-14T07:58:27.7719165Z BENCHMARK_METADATA: {"timestamp": 1741939107, "schema_version": "v3", "name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "aed0b7a742a2d7b7901790622829cbd2135049a4", "workflow_id": 13849515380, "run_attempt": 1, "job_id": 38756916747} 2025-03-14T07:58:27.7720462Z RUNNER_INFO: [] 2025-03-14T07:58:27.7720713Z DEPENDENCIES: {} 2025-03-14T07:58:27.7720958Z ##[endgroup] 2025-03-14T07:58:27.7747993Z + [[ ! -d test/test-reports ]] 2025-03-14T07:58:27.7748451Z + [[ false == \t\r\u\e ]] 2025-03-14T07:58:27.7751556Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1741939107, "schema_version": "v3", "name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "aed0b7a742a2d7b7901790622829cbd2135049a4", "workflow_id": 13849515380, "run_attempt": 1, "job_id": 38756916747}' --runners '[]' --dependencies '{}' 2025-03-14T07:58:27.9244937Z INFO:root:Upload test/test-reports/inference_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/13849515380/38756916747/inference_torchbench.json 2025-03-14T07:58:28.0038124Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-03-14T07:58:28.2767065Z INFO:root:Upload test/test-reports/inference_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/13849515380/38756916747/inference_torchbench_graph_breaks.json 2025-03-14T07:58:28.3758484Z INFO:root:Upload test/test-reports/training_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/13849515380/38756916747/training_torchbench.json 2025-03-14T07:58:28.5249371Z INFO:root:Upload test/test-reports/training_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/13849515380/38756916747/training_torchbench_graph_breaks.json 2025-03-14T07:58:28.7043463Z ##[group]Run cat test/**/*_toprint.log || true 2025-03-14T07:58:28.7043867Z cat test/**/*_toprint.log || true 2025-03-14T07:58:28.7052657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:28.7053047Z env: 2025-03-14T07:58:28.7053277Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:28.7053622Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:28.7054171Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:28.7054665Z ##[endgroup] 2025-03-14T07:58:28.7141187Z cat: 'test/**/*_toprint.log': No such file or directory 2025-03-14T07:58:28.7179007Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-03-14T07:58:28.7179373Z kill "$MONITOR_SCRIPT_PID" 2025-03-14T07:58:28.7187375Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:28.7187755Z env: 2025-03-14T07:58:28.7187992Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:28.7188348Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:28.7188893Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:28.7189402Z MONITOR_SCRIPT_PID: 825437 2025-03-14T07:58:28.7189864Z ##[endgroup] 2025-03-14T07:58:28.7339621Z Prepare all required actions 2025-03-14T07:58:28.7340029Z Getting action download info 2025-03-14T07:58:28.8679688Z Download action repository 'actions/upload-artifact@v4' (SHA:4cec3d8aa04e39d1a68397de0c4cd6fb9dce8ec1) 2025-03-14T07:58:29.3284653Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-03-14T07:58:29.3285017Z with: 2025-03-14T07:58:29.3285434Z file-suffix: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T07:58:29.3285934Z s3-bucket: gha-artifacts 2025-03-14T07:58:29.3286218Z env: 2025-03-14T07:58:29.3286446Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.3286794Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.3287359Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.3287848Z ##[endgroup] 2025-03-14T07:58:29.3333471Z ##[group]Run # Remove any previous test jsons if they exist 2025-03-14T07:58:29.3333958Z # Remove any previous test jsons if they exist 2025-03-14T07:58:29.3334342Z rm -f test-jsons-*.zip 2025-03-14T07:58:29.3334772Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-03-14T07:58:29.3343788Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:29.3344157Z env: 2025-03-14T07:58:29.3344390Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.3344729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.3345273Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.3345939Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T07:58:29.3346414Z ##[endgroup] 2025-03-14T07:58:29.3454179Z adding: test/test-reports/inference_torchbench.json (deflated 99%) 2025-03-14T07:58:29.3483899Z adding: test/test-reports/inference_torchbench_graph_breaks.json (deflated 98%) 2025-03-14T07:58:29.3531460Z adding: test/test-reports/training_torchbench.json (deflated 99%) 2025-03-14T07:58:29.3659954Z adding: test/test-reports/training_torchbench_graph_breaks.json (deflated 98%) 2025-03-14T07:58:29.3692262Z ##[group]Run # Remove any previous test reports if they exist 2025-03-14T07:58:29.3692758Z # Remove any previous test reports if they exist 2025-03-14T07:58:29.3693167Z rm -f test-reports-*.zip 2025-03-14T07:58:29.3693660Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-03-14T07:58:29.3702590Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:29.3702978Z env: 2025-03-14T07:58:29.3703217Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.3703573Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.3704137Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.3704811Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T07:58:29.3705322Z ##[endgroup] 2025-03-14T07:58:29.3759807Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2025-03-14T07:58:29.3763053Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2025-03-14T07:58:29.3764677Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 78%) 2025-03-14T07:58:29.3765511Z adding: test/test-reports/training_torchbench.csv (deflated 59%) 2025-03-14T07:58:29.3775949Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%) 2025-03-14T07:58:29.3777768Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2025-03-14T07:58:29.3808562Z ##[group]Run # Remove any previous usage logs if they exist 2025-03-14T07:58:29.3809027Z # Remove any previous usage logs if they exist 2025-03-14T07:58:29.3809409Z rm -f logs-*.zip 2025-03-14T07:58:29.3809889Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2025-03-14T07:58:29.3810787Z # so check to see if the file exists first 2025-03-14T07:58:29.3811165Z if [ -f 'usage_log.txt' ]; then 2025-03-14T07:58:29.3811552Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2025-03-14T07:58:29.3811915Z fi 2025-03-14T07:58:29.3812313Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2025-03-14T07:58:29.3812898Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2025-03-14T07:58:29.3813299Z fi 2025-03-14T07:58:29.3821166Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:29.3821537Z env: 2025-03-14T07:58:29.3821765Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.3822105Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.3822656Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.3823331Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T07:58:29.3823811Z ##[endgroup] 2025-03-14T07:58:29.3936954Z adding: usage_log.txt (deflated 96%) 2025-03-14T07:58:29.3992938Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-03-14T07:58:29.3993458Z # Remove any previous debugging artifacts if they exist 2025-03-14T07:58:29.3993869Z rm -f debug-*.zip 2025-03-14T07:58:29.3994172Z if [ -d 'test/debug' ]; then 2025-03-14T07:58:29.3994543Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-03-14T07:58:29.3994888Z fi 2025-03-14T07:58:29.4002842Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:29.4003214Z env: 2025-03-14T07:58:29.4003447Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.4003789Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.4004336Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.4005027Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747 2025-03-14T07:58:29.4005507Z ##[endgroup] 2025-03-14T07:58:29.4107104Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:58:29.4107442Z with: 2025-03-14T07:58:29.4107684Z s3-bucket: gha-artifacts 2025-03-14T07:58:29.4108025Z s3-prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:29.4108388Z retention-days: 14 2025-03-14T07:58:29.4108663Z if-no-files-found: warn 2025-03-14T07:58:29.4108951Z path: test-jsons-*.zip 2025-03-14T07:58:29.4109224Z name: artifact 2025-03-14T07:58:29.4109478Z region: us-east-1 2025-03-14T07:58:29.4109724Z env: 2025-03-14T07:58:29.4109957Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.4110312Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.4110878Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.4111367Z ##[endgroup] 2025-03-14T07:58:29.7861548Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:58:29.7862125Z With the provided path, there will be 1 file uploaded 2025-03-14T07:58:29.7862633Z Uploading to s3 prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:29.7917863Z Starting upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:29.9314440Z Finished upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:29.9614842Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:58:29.9615189Z with: 2025-03-14T07:58:29.9615427Z s3-bucket: gha-artifacts 2025-03-14T07:58:29.9615766Z s3-prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:29.9616131Z retention-days: 14 2025-03-14T07:58:29.9616405Z if-no-files-found: error 2025-03-14T07:58:29.9616704Z path: test-reports-*.zip 2025-03-14T07:58:29.9616992Z name: artifact 2025-03-14T07:58:29.9617240Z region: us-east-1 2025-03-14T07:58:29.9617485Z env: 2025-03-14T07:58:29.9617913Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:29.9618452Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:29.9619022Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:29.9619514Z ##[endgroup] 2025-03-14T07:58:30.2906300Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:58:30.2907609Z With the provided path, there will be 1 file uploaded 2025-03-14T07:58:30.2908784Z Uploading to s3 prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:30.2961273Z Starting upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:30.4191834Z Finished upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:30.4488843Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:58:30.4489192Z with: 2025-03-14T07:58:30.4489435Z s3-bucket: gha-artifacts 2025-03-14T07:58:30.4489773Z s3-prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:30.4490159Z retention-days: 14 2025-03-14T07:58:30.4490435Z if-no-files-found: ignore 2025-03-14T07:58:30.4490725Z path: logs-*.zip 2025-03-14T07:58:30.4490977Z name: artifact 2025-03-14T07:58:30.4491222Z region: us-east-1 2025-03-14T07:58:30.4491470Z env: 2025-03-14T07:58:30.4491698Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:30.4492058Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:30.4492623Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:30.4493123Z ##[endgroup] 2025-03-14T07:58:30.7824854Z NOTE: s3-prefix specified, ignoring name parameter 2025-03-14T07:58:30.7825322Z With the provided path, there will be 1 file uploaded 2025-03-14T07:58:30.7825792Z Uploading to s3 prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:30.7881529Z Starting upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:30.9505913Z Finished upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_38756916747.zip 2025-03-14T07:58:30.9820189Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-03-14T07:58:30.9820536Z with: 2025-03-14T07:58:30.9820768Z s3-bucket: gha-artifacts 2025-03-14T07:58:30.9821106Z s3-prefix: pytorch/pytorch/13849515380/1/artifact 2025-03-14T07:58:30.9821461Z retention-days: 14 2025-03-14T07:58:30.9821729Z if-no-files-found: ignore 2025-03-14T07:58:30.9822014Z path: debug-*.zip 2025-03-14T07:58:30.9822258Z name: artifact 2025-03-14T07:58:30.9822503Z region: us-east-1 2025-03-14T07:58:30.9822745Z env: 2025-03-14T07:58:30.9822965Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:30.9823304Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:30.9823849Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:30.9824334Z ##[endgroup] 2025-03-14T07:58:31.3047366Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-03-14T07:58:31.3351638Z ##[group]Run # shellcheck disable=SC2156 2025-03-14T07:58:31.3352022Z # shellcheck disable=SC2156 2025-03-14T07:58:31.3352596Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-03-14T07:58:31.3361761Z shell: /usr/bin/bash -e {0} 2025-03-14T07:58:31.3362041Z env: 2025-03-14T07:58:31.3362271Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:31.3362618Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:31.3363180Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:31.3363684Z ##[endgroup] 2025-03-14T07:58:31.6290752Z Prepare all required actions 2025-03-14T07:58:31.6291152Z Getting action download info 2025-03-14T07:58:31.7453055Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-03-14T07:58:31.7453416Z with: 2025-03-14T07:58:31.7453647Z job_id: 38756916747 2025-03-14T07:58:31.7454303Z job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T07:58:31.7454826Z workflow_name: inductor 2025-03-14T07:58:31.7455109Z workflow_run_id: 13849515380 2025-03-14T07:58:31.7455401Z workflow_attempt: 1 2025-03-14T07:58:31.7455648Z env: 2025-03-14T07:58:31.7455878Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:31.7456220Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:31.7456774Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:31.7457272Z ##[endgroup] 2025-03-14T07:58:31.7496226Z ##[group]Run echo "workflow_id: 13849515380" 2025-03-14T07:58:31.7496624Z echo "workflow_id: 13849515380" 2025-03-14T07:58:31.7496969Z echo "workflow_attempt: 1" 2025-03-14T07:58:31.7497296Z echo "workflow_Name: inductor" 2025-03-14T07:58:31.7497621Z echo "job_id: 38756916747" 2025-03-14T07:58:31.7498190Z echo "job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)" 2025-03-14T07:58:31.7507360Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:31.7507739Z env: 2025-03-14T07:58:31.7507974Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:31.7508319Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:31.7508878Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:31.7509366Z ##[endgroup] 2025-03-14T07:58:31.7537862Z workflow_id: 13849515380 2025-03-14T07:58:31.7538185Z workflow_attempt: 1 2025-03-14T07:58:31.7538464Z workflow_Name: inductor 2025-03-14T07:58:31.7538733Z job_id: 38756916747 2025-03-14T07:58:31.7539218Z job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-03-14T07:58:31.7584524Z ##[group]Run nick-fields/retry@v3.0.0 2025-03-14T07:58:31.7584852Z with: 2025-03-14T07:58:31.7585072Z shell: bash 2025-03-14T07:58:31.7585321Z timeout_minutes: 5 2025-03-14T07:58:31.7585606Z max_attempts: 5 2025-03-14T07:58:31.7585876Z retry_wait_seconds: 30 2025-03-14T07:58:31.7586349Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 2025-03-14T07:58:31.7586968Z polling_interval_seconds: 1 2025-03-14T07:58:31.7587275Z warning_on_retry: true 2025-03-14T07:58:31.7587567Z continue_on_error: false 2025-03-14T07:58:31.7587855Z env: 2025-03-14T07:58:31.7588091Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:31.7588453Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:31.7589017Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:31.7589535Z ##[endgroup] 2025-03-14T07:58:32.9468159Z Requirement already satisfied: python-dateutil==2.8.2 in /home/ec2-user/miniconda/lib/python3.12/site-packages (2.8.2) 2025-03-14T07:58:32.9469051Z Requirement already satisfied: boto3==1.35.42 in /home/ec2-user/miniconda/lib/python3.12/site-packages (1.35.42) 2025-03-14T07:58:32.9469893Z Requirement already satisfied: pandas==2.1.3 in /home/ec2-user/miniconda/lib/python3.12/site-packages (2.1.3) 2025-03-14T07:58:32.9479675Z Requirement already satisfied: six>=1.5 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from python-dateutil==2.8.2) (1.17.0) 2025-03-14T07:58:32.9494734Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from boto3==1.35.42) (1.35.99) 2025-03-14T07:58:32.9498408Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from boto3==1.35.42) (1.0.1) 2025-03-14T07:58:32.9502187Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from boto3==1.35.42) (0.10.4) 2025-03-14T07:58:32.9590312Z Requirement already satisfied: numpy<2,>=1.26.0 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from pandas==2.1.3) (1.26.4) 2025-03-14T07:58:32.9594621Z Requirement already satisfied: pytz>=2020.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from pandas==2.1.3) (2025.1) 2025-03-14T07:58:32.9597977Z Requirement already satisfied: tzdata>=2022.1 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from pandas==2.1.3) (2025.1) 2025-03-14T07:58:32.9619117Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /home/ec2-user/miniconda/lib/python3.12/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.26.20) 2025-03-14T07:58:33.8387856Z Command completed after 1 attempt(s). 2025-03-14T07:58:33.8466102Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-03-14T07:58:33.8466923Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-03-14T07:58:33.8467440Z  --workflow-run-id "13849515380" \ 2025-03-14T07:58:33.8467803Z  --workflow-name "inductor" \ 2025-03-14T07:58:33.8468161Z  --workflow-run-attempt "1" \ 2025-03-14T07:58:33.8468517Z  --job-id "38756916747" \ 2025-03-14T07:58:33.8469070Z  --job-name "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)" 2025-03-14T07:58:33.8478493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:33.8478871Z env: 2025-03-14T07:58:33.8479098Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:33.8479442Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:33.8479998Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:33.8480499Z ##[endgroup] 2025-03-14T07:58:33.8881441Z Traceback (most recent call last): 2025-03-14T07:58:33.8881919Z File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2025-03-14T07:58:33.8882387Z return _run_code(code, main_globals, None, 2025-03-14T07:58:33.8882801Z File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2025-03-14T07:58:33.8883220Z exec(code, run_globals) 2025-03-14T07:58:33.8884703Z File "/home/ec2-user/actions-runner/_work/pytorch/pytorch/tools/stats/upload_utilization_stats/upload_utilization_stats.py", line 20, in 2025-03-14T07:58:33.8886252Z import pandas as pd # type: ignore[import] 2025-03-14T07:58:33.8887021Z ModuleNotFoundError: No module named 'pandas' 2025-03-14T07:58:33.8939804Z ##[error]Process completed with exit code 1. 2025-03-14T07:58:33.8998679Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-03-14T07:58:33.8999139Z with: 2025-03-14T07:58:33.8999374Z env: 2025-03-14T07:58:33.8999618Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:33.8999988Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:33.9000563Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:33.9001079Z ##[endgroup] 2025-03-14T07:58:33.9043130Z ##[group]Run set -eou pipefail 2025-03-14T07:58:33.9043476Z set -eou pipefail 2025-03-14T07:58:33.9043771Z  2025-03-14T07:58:33.9044173Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-03-14T07:58:33.9044647Z for _ in $(seq 1440); do 2025-03-14T07:58:33.9045005Z  # Break if no ssh session exists anymore 2025-03-14T07:58:33.9045373Z  if [ "$(who)" = "" ]; then 2025-03-14T07:58:33.9045692Z  break 2025-03-14T07:58:33.9045985Z  fi 2025-03-14T07:58:33.9046231Z  echo "." 2025-03-14T07:58:33.9046493Z  sleep 5 2025-03-14T07:58:33.9046758Z done 2025-03-14T07:58:33.9055419Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:33.9055804Z env: 2025-03-14T07:58:33.9056037Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:33.9056385Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:33.9056945Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:33.9057450Z ##[endgroup] 2025-03-14T07:58:33.9085183Z Holding runner for 2 hours until all ssh sessions have logged out 2025-03-14T07:58:33.9149335Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T07:58:33.9149910Z # ignore expansion of "docker ps -q" since it could be empty 2025-03-14T07:58:33.9150343Z # shellcheck disable=SC2046 2025-03-14T07:58:33.9150717Z docker stop $(docker ps -q) || true 2025-03-14T07:58:33.9151083Z # Prune all of the docker images 2025-03-14T07:58:33.9151422Z docker system prune -af 2025-03-14T07:58:33.9160116Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:33.9160500Z env: 2025-03-14T07:58:33.9160732Z GIT_DEFAULT_BRANCH: main 2025-03-14T07:58:33.9161081Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-03-14T07:58:33.9161637Z DOCKER_CONTAINER_ID: f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:33.9162130Z ##[endgroup] 2025-03-14T07:58:34.7089629Z f31fd565c532 2025-03-14T07:58:38.5961486Z Deleted Containers: 2025-03-14T07:58:38.5962032Z f31fd565c53286e3fe252c7267bbe58cfa0f1298dfbc73c388a8d6bbca7724de 2025-03-14T07:58:38.5962404Z 2025-03-14T07:58:48.6358790Z Deleted Images: 2025-03-14T07:58:48.6359737Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:aa89d6e739080d90fa18625d57297c6734465849 2025-03-14T07:58:48.6361246Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks@sha256:20eb41577713f879cca4c6a57bd64d737c482cad39fe2b18409f513444e2b522 2025-03-14T07:58:48.6362388Z deleted: sha256:9f77b6c3483857c0bff989bce733b5bd5d6fc70a10591ed0f8d1de80d0e77bfd 2025-03-14T07:58:48.6363026Z deleted: sha256:006617515f4e958af4f9dc6065c779e410071c02b44689d893561937feda4f3d 2025-03-14T07:58:48.6363661Z deleted: sha256:d8dfc9fe28ede8a98ec1fc902c02e20e76ec8f22f58784b12c12343fa12aab46 2025-03-14T07:58:48.6364317Z deleted: sha256:2ce32bbc6df81c5a9f0eedd7f48abb4d8336b02cd0fe1c318c1e6ffbf5595e1d 2025-03-14T07:58:48.6364987Z deleted: sha256:c120dcdc0d53f8756455aeed564dcd3f8ecb5141f494a1d6a0e605e5a9c3d254 2025-03-14T07:58:48.6365633Z deleted: sha256:e5b77ce8ae749a04e3cb4574d097ead9420bada2f025a0a4f9fcf7de22ee6a3b 2025-03-14T07:58:48.6366283Z deleted: sha256:65922f72524fa8bd30f5814c0483c298cc1e363eeb7b032d764285811af93ce5 2025-03-14T07:58:48.6367524Z deleted: sha256:33352e487d1c5d4a1ea6f0e5ec8d4aa690c8d775510ea3e62f860dc97ac64997 2025-03-14T07:58:48.6368173Z deleted: sha256:2878e03ee5cad99652cce4dd49bf5eed60082e76b77e6b947066e957d7b4d828 2025-03-14T07:58:48.6368820Z deleted: sha256:e4447a79ed2e204ebac0f3b48e17d6ec52941cf7a0e5d0b3e9051bea08065f71 2025-03-14T07:58:48.6369535Z deleted: sha256:8aa553275dbba5e374d93de0cef6fc5150eca77aab3ff27289706fa6db36af67 2025-03-14T07:58:48.6370179Z deleted: sha256:dc0cf83dbb0248e366220eed83a75f88c574ebe4ca58830c0fef21e4bec501a4 2025-03-14T07:58:48.6370818Z deleted: sha256:daac181dd53a4d2f2c53329e091cb6a2911be2dc9b38cf8bd09f3bce1c56329c 2025-03-14T07:58:48.6371454Z deleted: sha256:5f7a34c79738cb5ff3137836e023378b646d5b28faea7f76b6ed7ba06bf54130 2025-03-14T07:58:48.6372073Z deleted: sha256:4b0c75c8695ff1444c28e7f0d9f37138c72bb20ce91d65155c07902e7e05a0c5 2025-03-14T07:58:48.6372694Z deleted: sha256:0f379e6e379d2461e4d77a69b7f475ee67a98b758bfa2a6e05f11f20e39966b3 2025-03-14T07:58:48.6373317Z deleted: sha256:3584990d592e96be75dd409691a557456ccf4876a869c0e3dd941bed6fed0b42 2025-03-14T07:58:48.6373947Z deleted: sha256:caf7ae943b4b818733b68d0dfeec2a2fcdcb4b58aae0ff0eb07ad8de3aecc198 2025-03-14T07:58:48.6374582Z deleted: sha256:95c2e1a5d17c51b798f929a84a005e82a2665193d91211021be3a9336b0693bf 2025-03-14T07:58:48.6375213Z deleted: sha256:84fd9d078db4e9840fbe6ae6e4fd539b4a7238f0dae2d0474c17148d3a27d059 2025-03-14T07:58:48.6375852Z deleted: sha256:0fb240cfd150fea9ed47646b4e3c3a7e4e1ea5307f56aaf07fa985d412318768 2025-03-14T07:58:48.6376495Z deleted: sha256:f18d22f63350f207ade7b97d488c59e8fd7cd7408ed04ed6f5b92ada881c61ef 2025-03-14T07:58:48.6377340Z deleted: sha256:d42f54324f663c1982969ed3fe001f4b304fecfa3cfc10132ad9d4dc84a2bacc 2025-03-14T07:58:48.6377963Z deleted: sha256:0b37319755f8628856da1780e49c5e71a173a7a1a4a3b7ac5f8ceb1071071c74 2025-03-14T07:58:48.6378566Z deleted: sha256:76592bb55488254587030a382269d8e529361c5dde7153d975ec0396488ca44d 2025-03-14T07:58:48.6379202Z deleted: sha256:2bffea548a1863fddca3ebaedf1ca51aaefd33da5128a2b1d81da0bbb12d4ad8 2025-03-14T07:58:48.6379845Z deleted: sha256:eed215398a5b664a7a01e0d0f26462615ad7a9cefe2f74257fb14575d08c244c 2025-03-14T07:58:48.6380474Z deleted: sha256:f5282aa578fffba31e0baa84c639a8e1e94a77ed3e552206d6ec3446cbfa89b1 2025-03-14T07:58:48.6381100Z deleted: sha256:f317c307d996656151b7ac4957ac6b2c986f06732e6afce47e43d0b31f917a87 2025-03-14T07:58:48.6381712Z deleted: sha256:fd255efe4773a357812751cffd74b8461b2f92c9a7b5127e7521925001c988d3 2025-03-14T07:58:48.6382334Z deleted: sha256:30e20732ffdb3275b7c129ad98542773192aca3faab2122c6332ab64261bcbd1 2025-03-14T07:58:48.6382961Z deleted: sha256:4eba7dc868086afd8f7068ffc4b1430987109e93f31499cbd0b9186d230f4b64 2025-03-14T07:58:48.6383600Z deleted: sha256:32b75793e91d34d3df888760cfaf84d89bad5549dfedc542bd2f37f4f19c4163 2025-03-14T07:58:48.6384226Z deleted: sha256:264e78fc63ac5701d0309f437fbd1371eef92d4cb1ad885bb7e8711f91876023 2025-03-14T07:58:48.6384851Z deleted: sha256:588d5279ec0827a0c14c7996c8a0de1308053c827072bc5ff633db2f3f8b4ca7 2025-03-14T07:58:48.6385468Z deleted: sha256:f8a41ed324145e99d2949a9ad7e770977a1253f00cc0500f20331914040a1267 2025-03-14T07:58:48.6386086Z deleted: sha256:68428ef9c1e2998605df716b9c01dc38c2bcdbba94c8b31c6430ab212efeac31 2025-03-14T07:58:48.6386840Z deleted: sha256:a1917ce13f328407ea441062f71b5b77ac5f2b3ae9c952851c42f9a82135793b 2025-03-14T07:58:48.6387470Z deleted: sha256:b3fbda5315c62609003c185eff2f901f968fb890b3c1f089bc8b524925368f24 2025-03-14T07:58:48.6388089Z deleted: sha256:5e9f33a8a4d4909d1334953eb99e6043242524e35d555e6622a2551211c26f5e 2025-03-14T07:58:48.6388705Z deleted: sha256:51838fe2fd479c6daa3a7cb4c47d5a084ad59043b17540a456ff88c0723d2250 2025-03-14T07:58:48.6389341Z deleted: sha256:f0272fb272ff749e1d3da1729dc7b5ff83c43465e5c95fb80e773f524e61580a 2025-03-14T07:58:48.6389981Z deleted: sha256:db15f066101d28e941686c1ac5e5a81024495efcd81d880451f9001076918e99 2025-03-14T07:58:48.6390600Z deleted: sha256:5f29f057376f675e8139a0b6af0d4795a0fc5a1930d205c2c24135e7929ca460 2025-03-14T07:58:48.6391358Z deleted: sha256:375b9f836aa6a734930fcd4ce90df2a9b40f094f7ebc68449755c2313caa1081 2025-03-14T07:58:48.6391989Z deleted: sha256:c6f22f833901f861e3c9b30d1620529d27ecaf641c94f336e5d17e50696cccc1 2025-03-14T07:58:48.6392624Z deleted: sha256:d0d45ba336b5db9d314ad4a1a948f4e1711f9e2f527865e29de0c162a66b16d6 2025-03-14T07:58:48.6393260Z deleted: sha256:9b7b0db453ce8cc71cef0616cb4e09b80b053735e533649fb39c3f21f26c7d81 2025-03-14T07:58:48.6393887Z deleted: sha256:441e98624d1376b791d8af11aeee95372e5516769496260a25ad268f7c24c606 2025-03-14T07:58:48.6394504Z deleted: sha256:1b3c121c6718eb9cd2b6e96fe80e390205d000355db4014a598bb86d9d97f955 2025-03-14T07:58:48.6395136Z deleted: sha256:f3d3a1d041a9da9f2d71a3565b2cee16d011f9c676d1db4b51599068bb178a86 2025-03-14T07:58:48.6395773Z deleted: sha256:05cbe0ec3896fa1c6cf4f24845f78a75acde5ae92a1b75928cdeb620af0ab92c 2025-03-14T07:58:48.6396396Z deleted: sha256:9233c2f0227af3f5244baddf5ff281313ef3ab22166d295168266b4024143358 2025-03-14T07:58:48.6397025Z deleted: sha256:bffee652694320768fb4c36c5fbf8d8235d9bcc56f6943f8ce068955e2fdff42 2025-03-14T07:58:48.6397656Z deleted: sha256:0c3bd1563fe41a6680da0f88490a6e3f95c13f717fcf3be921cb91bdc0c70c6d 2025-03-14T07:58:48.6398293Z deleted: sha256:a8e68dfacfd1c48c0f0de41a6f1027138d3594de96f11173259edb1c550d16e1 2025-03-14T07:58:48.6398930Z deleted: sha256:686ea9f39e5c7149e5e3a54c1fe8fd8944a89d1ac77d8ceb89d6baa82d38353e 2025-03-14T07:58:48.6399549Z deleted: sha256:cc54309555c2967d247e976d0615961da3574ba7a10807962394fa9715aa5ba9 2025-03-14T07:58:48.6400148Z deleted: sha256:6943262429b139d69d97e0b5a1c815b30a5e729c419be0b625a4dd752f3ea6d0 2025-03-14T07:58:48.6400770Z deleted: sha256:23f6ecaead1d4d1346b4ca530a520f5fb05c393fc9607ceb5d6d74f00fd95d9d 2025-03-14T07:58:48.6401477Z deleted: sha256:3463127b6781701fd57f819279de4b6a70d206e909ec0c3194e4189a7430a1af 2025-03-14T07:58:48.6402200Z deleted: sha256:c01ac6f42ea62550c7e4049f442abaf30774144a8883656d88a11c75a21f64f9 2025-03-14T07:58:48.6402911Z deleted: sha256:514b4f6b58aa7f2fb8ae46592e1bf3874b0fa104ae15f23e593e1a72fe5b6b58 2025-03-14T07:58:48.6403692Z deleted: sha256:c3843a02971b2998897727ff0a2572a5832df6caa5325182b2ec72d734c1e6ed 2025-03-14T07:58:48.6404323Z deleted: sha256:06563b6c0292b6f05bf75a09a8641f589a0d4b77f70f794425317915d99341a9 2025-03-14T07:58:48.6404926Z deleted: sha256:6c53bb6e0232a44846d29332a3243a22a11dddd51125d8f9734b1916bd20f18c 2025-03-14T07:58:48.6405541Z deleted: sha256:59b9e37805a254e0fd1cb7f05851c0a5dca09e78196334d3b70a50a1745f14ed 2025-03-14T07:58:48.6406159Z deleted: sha256:ebc7d3133ec482ba0278ab995ba56a672c8a3450ec83a176b1a780fd0f85e787 2025-03-14T07:58:48.6406823Z deleted: sha256:c5202300187efa50fa85a1f4d829e8a723f1243a4a395ca444f807d731d5b19f 2025-03-14T07:58:48.6407453Z deleted: sha256:9951141dbd2b4ebb9a96557e8e809ded0c1e449f499c8fd2fb4e05233fe6282f 2025-03-14T07:58:48.6408071Z deleted: sha256:fa382588f8917faec5545e1031108a2f5e3391cd75859197c3a632e47dcd00e9 2025-03-14T07:58:48.6408683Z deleted: sha256:2150f1c968f3842e8df8ae2e1ce3c0ab83e7062b906d8f6a4cd8415f6f822a00 2025-03-14T07:58:48.6409314Z deleted: sha256:46bc902662130b8d13f25b0294fac16ca6248034d5ccfaf3ee51cb56572946c5 2025-03-14T07:58:48.6409921Z deleted: sha256:43659ee56fac7f221f9388ec263ad2042847103213025f0a7a0ec7ca6a81b20f 2025-03-14T07:58:48.6410541Z deleted: sha256:5839c16cf80c38c4f69268b2c73c3d7ec37a33e18c9c5f49e6624239a182dbf8 2025-03-14T07:58:48.6411157Z deleted: sha256:e16d8e25c4393c96891af6d6a42889638fc30a7490ef9c5c5517798ea9860bda 2025-03-14T07:58:48.6411774Z deleted: sha256:868539926d1d83fe2b4fdb0231d8b57fb5b8b34144d733a4cf862d241e16c667 2025-03-14T07:58:48.6412396Z deleted: sha256:cc4224d01139aef3aeb29c568d184283602d52f47bb8e5e29de97bf6aa54d951 2025-03-14T07:58:48.6413026Z deleted: sha256:bd18b4fe898c9279c6e9ad952165522e24de55ab48d83289ca0ff6c8df97d85d 2025-03-14T07:58:48.6413651Z deleted: sha256:c7917cebb757d994f33b0f4bbad70fbb563939c2630d8f96113d437fa09e68cd 2025-03-14T07:58:48.6414276Z deleted: sha256:34fb1ecb50c48f883ea450e0364c90bdf16ce5e35e8a132012851ac3b040addf 2025-03-14T07:58:48.6415013Z deleted: sha256:92300fc3f1a943c0b6a295d87da98cb3d593b18b8ddc4f0b229e3ca909d03206 2025-03-14T07:58:48.6415649Z deleted: sha256:fffe76c64ef2dee2d80a8bb3ad13d65d596d04a45510b1956a976a69215dae92 2025-03-14T07:58:48.6416025Z 2025-03-14T07:58:48.6416155Z Total reclaimed space: 59.02GB 2025-03-14T07:58:48.6510804Z Post job cleanup. 2025-03-14T07:58:48.6576849Z Post job cleanup. 2025-03-14T07:58:48.7607972Z [command]/usr/bin/git version 2025-03-14T07:58:48.7737358Z git version 2.47.1 2025-03-14T07:58:48.7772931Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/94ba4e84-cdb5-43da-b870-5d10c4e35647/.gitconfig' 2025-03-14T07:58:48.7783558Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/94ba4e84-cdb5-43da-b870-5d10c4e35647' before making global git config changes 2025-03-14T07:58:48.7784485Z Adding repository directory to the temporary git global config as a safe directory 2025-03-14T07:58:48.7789627Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-03-14T07:58:48.7834011Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-03-14T07:58:48.7869778Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-03-14T07:58:48.8283584Z Entering 'android/libs/fbjni' 2025-03-14T07:58:48.8355481Z Entering 'third_party/FP16' 2025-03-14T07:58:48.8423988Z Entering 'third_party/FXdiv' 2025-03-14T07:58:48.8494089Z Entering 'third_party/NNPACK' 2025-03-14T07:58:48.8562479Z Entering 'third_party/NVTX' 2025-03-14T07:58:48.8632311Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T07:58:48.8701503Z Entering 'third_party/XNNPACK' 2025-03-14T07:58:48.8790859Z Entering 'third_party/benchmark' 2025-03-14T07:58:48.8860912Z Entering 'third_party/composable_kernel' 2025-03-14T07:58:48.8940004Z Entering 'third_party/cpp-httplib' 2025-03-14T07:58:48.9009705Z Entering 'third_party/cpuinfo' 2025-03-14T07:58:48.9081734Z Entering 'third_party/cudnn_frontend' 2025-03-14T07:58:48.9153104Z Entering 'third_party/cutlass' 2025-03-14T07:58:48.9229702Z Entering 'third_party/eigen' 2025-03-14T07:58:48.9299979Z Entering 'third_party/fbgemm' 2025-03-14T07:58:48.9372894Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T07:58:48.9438785Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T07:58:48.9505746Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T07:58:48.9581713Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T07:58:48.9648953Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T07:58:48.9718728Z Entering 'third_party/flash-attention' 2025-03-14T07:58:48.9792743Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T07:58:48.9869743Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T07:58:48.9948884Z Entering 'third_party/flatbuffers' 2025-03-14T07:58:49.0024023Z Entering 'third_party/fmt' 2025-03-14T07:58:49.0096268Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T07:58:49.0165402Z Entering 'third_party/gloo' 2025-03-14T07:58:49.0234795Z Entering 'third_party/googletest' 2025-03-14T07:58:49.0309759Z Entering 'third_party/ideep' 2025-03-14T07:58:49.0377578Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T07:58:49.0454983Z Entering 'third_party/ittapi' 2025-03-14T07:58:49.0523331Z Entering 'third_party/kineto' 2025-03-14T07:58:49.0591392Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T07:58:49.0659139Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T07:58:49.0734405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T07:58:49.0802613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T07:58:49.0871600Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T07:58:49.0937033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T07:58:49.1009768Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T07:58:49.1081094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T07:58:49.1156318Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T07:58:49.1224347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T07:58:49.1295692Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T07:58:49.1362887Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T07:58:49.1437015Z Entering 'third_party/kleidiai' 2025-03-14T07:58:49.1511914Z Entering 'third_party/mimalloc' 2025-03-14T07:58:49.1581201Z Entering 'third_party/nlohmann' 2025-03-14T07:58:49.1654267Z Entering 'third_party/onnx' 2025-03-14T07:58:49.1737450Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T07:58:49.1811774Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T07:58:49.1882055Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T07:58:49.1949239Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T07:58:49.2016752Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T07:58:49.2087277Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T07:58:49.2156077Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T07:58:49.2222516Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T07:58:49.2290462Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T07:58:49.2356798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T07:58:49.2430434Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T07:58:49.2508625Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T07:58:49.2599841Z Entering 'third_party/pocketfft' 2025-03-14T07:58:49.2671565Z Entering 'third_party/protobuf' 2025-03-14T07:58:49.2742933Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T07:58:49.2811233Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T07:58:49.2881752Z Entering 'third_party/psimd' 2025-03-14T07:58:49.2952993Z Entering 'third_party/pthreadpool' 2025-03-14T07:58:49.3021038Z Entering 'third_party/pybind11' 2025-03-14T07:58:49.3092146Z Entering 'third_party/python-peachpy' 2025-03-14T07:58:49.3161075Z Entering 'third_party/sleef' 2025-03-14T07:58:49.3230485Z Entering 'third_party/tensorpipe' 2025-03-14T07:58:49.3298832Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T07:58:49.3366066Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T07:58:49.3432858Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T07:58:49.3500013Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T07:58:49.3566220Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T07:58:49.3662854Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-03-14T07:58:49.3688095Z http.https://github.com/.extraheader 2025-03-14T07:58:49.3700509Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-03-14T07:58:49.3738132Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-03-14T07:58:49.4099970Z Entering 'android/libs/fbjni' 2025-03-14T07:58:49.4144022Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4190878Z Entering 'third_party/FP16' 2025-03-14T07:58:49.4236216Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4280552Z Entering 'third_party/FXdiv' 2025-03-14T07:58:49.4325177Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4368994Z Entering 'third_party/NNPACK' 2025-03-14T07:58:49.4412737Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4457871Z Entering 'third_party/NVTX' 2025-03-14T07:58:49.4502384Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4548767Z Entering 'third_party/VulkanMemoryAllocator' 2025-03-14T07:58:49.4593042Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4636947Z Entering 'third_party/XNNPACK' 2025-03-14T07:58:49.4681191Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4740587Z Entering 'third_party/benchmark' 2025-03-14T07:58:49.4784552Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4830547Z Entering 'third_party/composable_kernel' 2025-03-14T07:58:49.4875114Z http.https://github.com/.extraheader 2025-03-14T07:58:49.4925519Z Entering 'third_party/cpp-httplib' 2025-03-14T07:58:49.4969919Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5013820Z Entering 'third_party/cpuinfo' 2025-03-14T07:58:49.5058114Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5102355Z Entering 'third_party/cudnn_frontend' 2025-03-14T07:58:49.5146348Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5190109Z Entering 'third_party/cutlass' 2025-03-14T07:58:49.5234724Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5287427Z Entering 'third_party/eigen' 2025-03-14T07:58:49.5331078Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5376739Z Entering 'third_party/fbgemm' 2025-03-14T07:58:49.5420440Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5465276Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-03-14T07:58:49.5509307Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5553431Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-03-14T07:58:49.5596697Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5642059Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-03-14T07:58:49.5684941Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5734990Z Entering 'third_party/fbgemm/third_party/googletest' 2025-03-14T07:58:49.5784492Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5828874Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-03-14T07:58:49.5876745Z http.https://github.com/.extraheader 2025-03-14T07:58:49.5923704Z Entering 'third_party/flash-attention' 2025-03-14T07:58:49.5969525Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6014312Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-03-14T07:58:49.6060383Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6110262Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-03-14T07:58:49.6153792Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6207957Z Entering 'third_party/flatbuffers' 2025-03-14T07:58:49.6253500Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6300708Z Entering 'third_party/fmt' 2025-03-14T07:58:49.6348609Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6395045Z Entering 'third_party/gemmlowp/gemmlowp' 2025-03-14T07:58:49.6440402Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6484388Z Entering 'third_party/gloo' 2025-03-14T07:58:49.6529088Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6573142Z Entering 'third_party/googletest' 2025-03-14T07:58:49.6617246Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6661844Z Entering 'third_party/ideep' 2025-03-14T07:58:49.6705823Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6751867Z Entering 'third_party/ideep/mkl-dnn' 2025-03-14T07:58:49.6795661Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6851745Z Entering 'third_party/ittapi' 2025-03-14T07:58:49.6896557Z http.https://github.com/.extraheader 2025-03-14T07:58:49.6941040Z Entering 'third_party/kineto' 2025-03-14T07:58:49.6985820Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7031201Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-03-14T07:58:49.7077928Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7121254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-03-14T07:58:49.7166516Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7215906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-03-14T07:58:49.7258974Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7303056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-03-14T07:58:49.7346089Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7391717Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-03-14T07:58:49.7434324Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7477655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-03-14T07:58:49.7522010Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7572282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-03-14T07:58:49.7615638Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7661516Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-03-14T07:58:49.7704105Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7748724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-03-14T07:58:49.7791749Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7837867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-03-14T07:58:49.7885380Z http.https://github.com/.extraheader 2025-03-14T07:58:49.7935789Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-03-14T07:58:49.7985123Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8029110Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-03-14T07:58:49.8075123Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8121129Z Entering 'third_party/kleidiai' 2025-03-14T07:58:49.8166822Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8212756Z Entering 'third_party/mimalloc' 2025-03-14T07:58:49.8257648Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8301584Z Entering 'third_party/nlohmann' 2025-03-14T07:58:49.8346763Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8392856Z Entering 'third_party/onnx' 2025-03-14T07:58:49.8440261Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8498897Z Entering 'third_party/onnx/third_party/pybind11' 2025-03-14T07:58:49.8541413Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8604326Z Entering 'third_party/opentelemetry-cpp' 2025-03-14T07:58:49.8636167Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8682266Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-03-14T07:58:49.8724752Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8768125Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-03-14T07:58:49.8817315Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8862498Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-03-14T07:58:49.8907664Z http.https://github.com/.extraheader 2025-03-14T07:58:49.8950614Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-03-14T07:58:49.8997671Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9044377Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-03-14T07:58:49.9086290Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9130485Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-03-14T07:58:49.9174080Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9218376Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-03-14T07:58:49.9261365Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9304181Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-03-14T07:58:49.9348573Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9395464Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-03-14T07:58:49.9439622Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9486732Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-03-14T07:58:49.9535949Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9602266Z Entering 'third_party/pocketfft' 2025-03-14T07:58:49.9650102Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9694252Z Entering 'third_party/protobuf' 2025-03-14T07:58:49.9742083Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9787905Z Entering 'third_party/protobuf/third_party/benchmark' 2025-03-14T07:58:49.9831388Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9874697Z Entering 'third_party/protobuf/third_party/googletest' 2025-03-14T07:58:49.9918295Z http.https://github.com/.extraheader 2025-03-14T07:58:49.9965440Z Entering 'third_party/psimd' 2025-03-14T07:58:50.0009124Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0053070Z Entering 'third_party/pthreadpool' 2025-03-14T07:58:50.0097129Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0140923Z Entering 'third_party/pybind11' 2025-03-14T07:58:50.0184824Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0230961Z Entering 'third_party/python-peachpy' 2025-03-14T07:58:50.0274801Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0318117Z Entering 'third_party/sleef' 2025-03-14T07:58:50.0362713Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0407156Z Entering 'third_party/tensorpipe' 2025-03-14T07:58:50.0452131Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0495246Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-03-14T07:58:50.0538247Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0582553Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-03-14T07:58:50.0628816Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0672481Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-03-14T07:58:50.0718095Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0761087Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-03-14T07:58:50.0804321Z http.https://github.com/.extraheader 2025-03-14T07:58:50.0847958Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-03-14T07:58:50.0891763Z http.https://github.com/.extraheader 2025-03-14T07:58:50.1064074Z A job completed hook has been configured by the self-hosted runner administrator 2025-03-14T07:58:50.1094543Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-03-14T07:58:50.1102450Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-03-14T07:58:50.1102842Z ##[endgroup] 2025-03-14T07:58:58.0754715Z Cleaning up orphan processes