2025-04-21T21:42:05.6322500Z Current runner version: '2.323.0' 2025-04-21T21:42:05.6328152Z Runner name: 'i-02e84e4685011efd7' 2025-04-21T21:42:05.6329144Z Machine name: 'ip-10-0-3-190' 2025-04-21T21:42:05.6333539Z ##[group]GITHUB_TOKEN Permissions 2025-04-21T21:42:05.6335987Z Actions: read 2025-04-21T21:42:05.6336520Z Attestations: read 2025-04-21T21:42:05.6337037Z Checks: read 2025-04-21T21:42:05.6337518Z Contents: read 2025-04-21T21:42:05.6338121Z Deployments: read 2025-04-21T21:42:05.6338626Z Discussions: read 2025-04-21T21:42:05.6339125Z Issues: read 2025-04-21T21:42:05.6339595Z Metadata: read 2025-04-21T21:42:05.6340081Z Models: read 2025-04-21T21:42:05.6340557Z Packages: read 2025-04-21T21:42:05.6341032Z Pages: read 2025-04-21T21:42:05.6341510Z PullRequests: read 2025-04-21T21:42:05.6342037Z RepositoryProjects: read 2025-04-21T21:42:05.6342591Z SecurityEvents: read 2025-04-21T21:42:05.6343118Z Statuses: read 2025-04-21T21:42:05.6343599Z ##[endgroup] 2025-04-21T21:42:05.6345531Z Secret source: Actions 2025-04-21T21:42:05.6346299Z Prepare workflow directory 2025-04-21T21:42:06.1166517Z Prepare all required actions 2025-04-21T21:42:06.1215198Z Getting action download info 2025-04-21T21:42:06.3580756Z Download action repository 'pytorch/test-infra@main' (SHA:d0e960f2e16f063377744672abd0cabc2c9a0167) 2025-04-21T21:42:07.9259482Z Download action repository 'pytorch/pytorch@main' (SHA:c312d8c5018f6c742fa9df7223fee74c71c4a68b) 2025-04-21T21:42:22.2368102Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2025-04-21T21:42:22.4469987Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-04-21T21:42:22.7424023Z Getting action download info 2025-04-21T21:42:22.8509511Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683) 2025-04-21T21:42:23.0871478Z Getting action download info 2025-04-21T21:42:23.2005284Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-04-21T21:42:23.3845091Z Getting action download info 2025-04-21T21:42:23.4868941Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-04-21T21:42:23.6490576Z Getting action download info 2025-04-21T21:42:23.7855978Z Complete job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:42:23.8396424Z A job started hook has been configured by the self-hosted runner administrator 2025-04-21T21:42:23.8493954Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-04-21T21:42:23.8506001Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:23.8506590Z ##[endgroup] 2025-04-21T21:42:25.6747235Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2025-04-21T21:42:25.6747699Z Instance Type: g5.4xlarge 2025-04-21T21:42:25.6747942Z AMI Name: unknown 2025-04-21T21:42:25.6787766Z AMI ID: ami-071226ecf16aa7d96 2025-04-21T21:42:31.1227390Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-04-21T21:42:31.1227802Z with: 2025-04-21T21:42:31.1228314Z github-secret: *** 2025-04-21T21:42:31.1228984Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-04-21T21:42:31.1229712Z activate-with-label: false 2025-04-21T21:42:31.1229974Z label: with-ssh 2025-04-21T21:42:31.1230217Z remove-existing-keys: true 2025-04-21T21:42:31.1230498Z fail-silently: true 2025-04-21T21:42:31.1230754Z env: 2025-04-21T21:42:31.1230966Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:31.1231221Z ##[endgroup] 2025-04-21T21:42:31.2447130Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-04-21T21:42:31.2448575Z ciflow reference detected, attempting to extract PR number 2025-04-21T21:42:31.5224213Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2025-04-21T21:42:31.5705849Z No SSH keys found for user pytorch-bot[bot] 2025-04-21T21:42:31.5706561Z Grabbing public ssh keys from https://github.com/laithsakka.keys 2025-04-21T21:42:31.6267088Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2025-04-21T21:42:31.6280766Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2025-04-21T21:42:31.6314306Z Login using: ssh ec2-user@ec2-54-162-213-130.compute-1.amazonaws.com 2025-04-21T21:42:31.6314865Z All testing is done inside the container, to start an interactive session run: 2025-04-21T21:42:31.6315383Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-04-21T21:42:31.6480932Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-04-21T21:42:31.6481352Z with: 2025-04-21T21:42:31.6481564Z no-sudo: true 2025-04-21T21:42:31.6481806Z submodules: recursive 2025-04-21T21:42:31.6482065Z fetch-depth: 0 2025-04-21T21:42:31.6482281Z env: 2025-04-21T21:42:31.6482492Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:31.6482759Z ##[endgroup] 2025-04-21T21:42:31.6620771Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:42:31.6621689Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:42:31.6632997Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:31.6633354Z env: 2025-04-21T21:42:31.6633560Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:31.6633817Z ##[endgroup] 2025-04-21T21:42:31.6727193Z ##[group]Run # Use all available CPUs for fetching 2025-04-21T21:42:31.6727603Z # Use all available CPUs for fetching 2025-04-21T21:42:31.6727937Z cd "${GITHUB_WORKSPACE}" 2025-04-21T21:42:31.6728276Z git config --global fetch.parallel 0 2025-04-21T21:42:31.6728654Z git config --global submodule.fetchJobs 0 2025-04-21T21:42:31.6729002Z  2025-04-21T21:42:31.6729343Z # Clean workspace. The default checkout action should also do this, but 2025-04-21T21:42:31.6729818Z # do it here as well just in case 2025-04-21T21:42:31.6730166Z if [[ -d .git ]]; then 2025-04-21T21:42:31.6730445Z  if [ -z "${NO_SUDO}" ]; then 2025-04-21T21:42:31.6730744Z  sudo git clean -ffdx 2025-04-21T21:42:31.6731093Z  else 2025-04-21T21:42:31.6731336Z  git clean -ffdx 2025-04-21T21:42:31.6731587Z  fi 2025-04-21T21:42:31.6731794Z fi 2025-04-21T21:42:31.6740219Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:31.6740582Z env: 2025-04-21T21:42:31.6740793Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:31.6741045Z NO_SUDO: true 2025-04-21T21:42:31.6741255Z ##[endgroup] 2025-04-21T21:42:31.9664537Z Removing .additional_ci_files/ 2025-04-21T21:42:31.9665071Z Removing .pytest_cache/ 2025-04-21T21:42:31.9665424Z Removing build/ 2025-04-21T21:42:31.9665738Z Removing dist/ 2025-04-21T21:42:31.9666256Z Removing logs-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_40895218916.zip 2025-04-21T21:42:31.9666922Z Removing test-jsons-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_40895218916.zip 2025-04-21T21:42:31.9667613Z Removing test-reports-test-inductor-2-2-linux.g5.4xlarge.nvidia.gpu_40895218916.zip 2025-04-21T21:42:31.9668183Z Removing test/.pytorch-disabled-tests.json 2025-04-21T21:42:31.9668523Z Removing test/__pycache__/ 2025-04-21T21:42:31.9668810Z Removing test/inductor/__pycache__/ 2025-04-21T21:42:31.9669110Z Removing test/test-reports/ 2025-04-21T21:42:31.9669410Z Removing test/torch_compile_debug/ 2025-04-21T21:42:31.9669725Z Removing tools/__pycache__/ 2025-04-21T21:42:31.9670000Z Removing tools/stats/__pycache__/ 2025-04-21T21:42:31.9670374Z Removing tools/stats/upload_utilization_stats/__pycache__/ 2025-04-21T21:42:31.9670763Z Removing tools/testing/__pycache__/ 2025-04-21T21:42:31.9671140Z Removing tools/testing/target_determination/__pycache__/ 2025-04-21T21:42:31.9671874Z Removing tools/testing/target_determination/heuristics/__pycache__/ 2025-04-21T21:42:31.9672289Z Removing usage_log.txt 2025-04-21T21:42:31.9747848Z ##[group]Run actions/checkout@v4 2025-04-21T21:42:31.9748136Z with: 2025-04-21T21:42:31.9748378Z ref: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:42:31.9748706Z fetch-depth: 0 2025-04-21T21:42:31.9748946Z submodules: recursive 2025-04-21T21:42:31.9749196Z show-progress: false 2025-04-21T21:42:31.9749460Z repository: pytorch/pytorch 2025-04-21T21:42:31.9749870Z token: *** 2025-04-21T21:42:31.9750094Z ssh-strict: true 2025-04-21T21:42:31.9750528Z ssh-user: git 2025-04-21T21:42:31.9750775Z persist-credentials: true 2025-04-21T21:42:31.9751035Z clean: true 2025-04-21T21:42:31.9751269Z sparse-checkout-cone-mode: true 2025-04-21T21:42:31.9751557Z fetch-tags: false 2025-04-21T21:42:31.9751777Z lfs: false 2025-04-21T21:42:31.9751997Z set-safe-directory: true 2025-04-21T21:42:31.9752249Z env: 2025-04-21T21:42:31.9752466Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:31.9752706Z ##[endgroup] 2025-04-21T21:42:32.0882498Z Syncing repository: pytorch/pytorch 2025-04-21T21:42:32.0883770Z ##[group]Getting Git version info 2025-04-21T21:42:32.0884219Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-04-21T21:42:32.0884835Z [command]/usr/bin/git version 2025-04-21T21:42:32.0885126Z git version 2.47.1 2025-04-21T21:42:32.0908675Z ##[endgroup] 2025-04-21T21:42:32.0919617Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/8fbe8433-623d-4c20-a273-dc9a92e3040c/.gitconfig' 2025-04-21T21:42:32.0941242Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/8fbe8433-623d-4c20-a273-dc9a92e3040c' before making global git config changes 2025-04-21T21:42:32.0942146Z Adding repository directory to the temporary git global config as a safe directory 2025-04-21T21:42:32.0946430Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-04-21T21:42:32.0991369Z [command]/usr/bin/git config --local --get remote.origin.url 2025-04-21T21:42:32.1016208Z https://github.com/pytorch/pytorch 2025-04-21T21:42:32.1034637Z ##[group]Removing previously created refs, to avoid conflicts 2025-04-21T21:42:32.1038371Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD 2025-04-21T21:42:32.1065509Z HEAD 2025-04-21T21:42:32.1108709Z ##[endgroup] 2025-04-21T21:42:32.1112612Z [command]/usr/bin/git submodule status 2025-04-21T21:42:32.1521314Z 7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe) 2025-04-21T21:42:32.1617974Z 4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081) 2025-04-21T21:42:32.1715468Z b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327) 2025-04-21T21:42:32.1827652Z c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0) 2025-04-21T21:42:32.1857780Z e170594ac7cf1dac584da473d4ca9301087090c1 third_party/NVTX (v3.1.0) 2025-04-21T21:42:32.1946927Z a6bfc237255a6bac1513f7c1ebde6d8aed6b5191 third_party/VulkanMemoryAllocator (v2.1.0-705-ga6bfc23) 2025-04-21T21:42:32.2562071Z 51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a010365) 2025-04-21T21:42:32.2592389Z 0d98dba29d66e93259db7daa53a9327df767a415 third_party/benchmark (v1.6.1) 2025-04-21T21:42:32.2628526Z 8086bbe3a78d931eb96fe12fdc014082e18d18d3 third_party/composable_kernel (rocm-6.4.0-6-g8086bbe3a) 2025-04-21T21:42:32.2786483Z 3b6597bba913d51161383657829b7e644e59c006 third_party/cpp-httplib (v0.15.3-20-g3b6597b) 2025-04-21T21:42:32.2927954Z 5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244) 2025-04-21T21:42:32.2969251Z 8801fd7b31c2f798732ed1e1dd4711a9cde4217d third_party/cudnn_frontend (v0.5-46-g8801fd7) 2025-04-21T21:42:32.3074391Z afa1772203677c5118fcd82537a9c8fefbcc7008 third_party/cutlass (v3.8.0) 2025-04-21T21:42:32.3907652Z 3147391d946bb4b6c68edd901f2add6ac1f31f8c third_party/eigen (3.4.0) 2025-04-21T21:42:32.4302888Z dbc3157bf256f1339b3fa1fef2be89ac4078be0e third_party/fbgemm (v0.4.1-446-gdbc3157b) 2025-04-21T21:42:32.4405630Z 979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4) 2025-04-21T21:42:32.4434778Z 01834de25e4bf3975a9a00e816292b1ad0fe184b third_party/flatbuffers (v23.3.3) 2025-04-21T21:42:32.4909587Z 123913715afeb8a437e6388b4473fcc4753e1c9a third_party/fmt (11.1.4) 2025-04-21T21:42:32.5046505Z 3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17) 2025-04-21T21:42:32.5147075Z c61070427610ccd923efe3e7f8b3eca12bbcc31a third_party/gloo (heads/main) 2025-04-21T21:42:32.5437750Z b514bdc898e2951020cbdca1304b75f5950d1f59 third_party/googletest (release-1.8.0-3484-gb514bdc8) 2025-04-21T21:42:32.5535579Z 719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1) 2025-04-21T21:42:32.5608868Z 5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42 third_party/ittapi (v3.23.0-14-g5b8a7d7) 2025-04-21T21:42:32.5925162Z 5fa4bd8c8fb581a621f72957d0cb11431cc4714f third_party/kineto (remotes/origin/sraikund/test-64-g5fa4bd8) 2025-04-21T21:42:32.5956838Z ef685a13cfbe8d418aa2ed34350e21e4938358b6 third_party/kleidiai (v1.3.0) 2025-04-21T21:42:32.5986245Z b66e3214d8a104669c2ec05ae91ebc26a8f5ab78 third_party/mimalloc (v1.8.2) 2025-04-21T21:42:32.6548739Z 87cda1d6646592ac5866dc703c8e1839046a6806 third_party/nlohmann (v3.10.1-113-g87cda1d6) 2025-04-21T21:42:32.6900234Z b8baa8446686496da4cc8fda09f2b6fe65c2a02c third_party/onnx (v1.17.0) 2025-04-21T21:42:32.6929084Z a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2) 2025-04-21T21:42:32.6959317Z 9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe third_party/pocketfft (release_for_eigen-11-g9d3ab05) 2025-04-21T21:42:32.7413151Z d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b) 2025-04-21T21:42:32.7510012Z 072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master) 2025-04-21T21:42:32.7580840Z 4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e) 2025-04-21T21:42:32.7639483Z a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e) 2025-04-21T21:42:32.7733432Z f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated) 2025-04-21T21:42:32.7871792Z 56e1f79cb140fb9326d612d0be06b5250565cade third_party/sleef (3.7-33-g56e1f79) 2025-04-21T21:42:32.7960446Z 52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master) 2025-04-21T21:42:32.7979622Z ##[group]Cleaning the repository 2025-04-21T21:42:32.7983890Z [command]/usr/bin/git clean -ffdx 2025-04-21T21:42:32.8220149Z [command]/usr/bin/git reset --hard HEAD 2025-04-21T21:42:33.8829970Z Updating files: 55% (9167/16510) 2025-04-21T21:42:33.8915470Z Updating files: 56% (9246/16510) 2025-04-21T21:42:33.9002083Z Updating files: 57% (9411/16510) 2025-04-21T21:42:33.9084532Z Updating files: 58% (9576/16510) 2025-04-21T21:42:33.9172079Z Updating files: 59% (9741/16510) 2025-04-21T21:42:33.9258410Z Updating files: 60% (9906/16510) 2025-04-21T21:42:33.9343896Z Updating files: 61% (10072/16510) 2025-04-21T21:42:33.9433106Z Updating files: 62% (10237/16510) 2025-04-21T21:42:33.9520189Z Updating files: 63% (10402/16510) 2025-04-21T21:42:33.9606021Z Updating files: 64% (10567/16510) 2025-04-21T21:42:33.9699707Z Updating files: 65% (10732/16510) 2025-04-21T21:42:33.9965292Z Updating files: 66% (10897/16510) 2025-04-21T21:42:34.0208194Z Updating files: 67% (11062/16510) 2025-04-21T21:42:34.0285380Z Updating files: 68% (11227/16510) 2025-04-21T21:42:34.0371486Z Updating files: 69% (11392/16510) 2025-04-21T21:42:34.0578750Z Updating files: 70% (11557/16510) 2025-04-21T21:42:34.0949254Z Updating files: 71% (11723/16510) 2025-04-21T21:42:34.1261181Z Updating files: 72% (11888/16510) 2025-04-21T21:42:34.1707270Z Updating files: 73% (12053/16510) 2025-04-21T21:42:34.1895229Z Updating files: 74% (12218/16510) 2025-04-21T21:42:34.2041256Z Updating files: 75% (12383/16510) 2025-04-21T21:42:34.2324742Z Updating files: 76% (12548/16510) 2025-04-21T21:42:34.2547904Z Updating files: 77% (12713/16510) 2025-04-21T21:42:34.2877672Z Updating files: 78% (12878/16510) 2025-04-21T21:42:34.3155403Z Updating files: 79% (13043/16510) 2025-04-21T21:42:34.3439241Z Updating files: 80% (13208/16510) 2025-04-21T21:42:34.3621175Z Updating files: 81% (13374/16510) 2025-04-21T21:42:34.3775154Z Updating files: 82% (13539/16510) 2025-04-21T21:42:34.3974683Z Updating files: 83% (13704/16510) 2025-04-21T21:42:34.4180779Z Updating files: 84% (13869/16510) 2025-04-21T21:42:34.4359725Z Updating files: 85% (14034/16510) 2025-04-21T21:42:34.4554684Z Updating files: 86% (14199/16510) 2025-04-21T21:42:34.4717933Z Updating files: 87% (14364/16510) 2025-04-21T21:42:34.4896910Z Updating files: 88% (14529/16510) 2025-04-21T21:42:34.5129598Z Updating files: 89% (14694/16510) 2025-04-21T21:42:34.5310530Z Updating files: 90% (14859/16510) 2025-04-21T21:42:34.5485757Z Updating files: 91% (15025/16510) 2025-04-21T21:42:34.5663229Z Updating files: 92% (15190/16510) 2025-04-21T21:42:34.5900744Z Updating files: 93% (15355/16510) 2025-04-21T21:42:34.6145921Z Updating files: 94% (15520/16510) 2025-04-21T21:42:34.6360546Z Updating files: 95% (15685/16510) 2025-04-21T21:42:34.6567432Z Updating files: 96% (15850/16510) 2025-04-21T21:42:34.6902148Z Updating files: 97% (16015/16510) 2025-04-21T21:42:34.7109132Z Updating files: 98% (16180/16510) 2025-04-21T21:42:34.7308901Z Updating files: 99% (16345/16510) 2025-04-21T21:42:34.7309224Z Updating files: 100% (16510/16510) 2025-04-21T21:42:34.7309561Z Updating files: 100% (16510/16510), done. 2025-04-21T21:42:34.7485582Z HEAD is now at cfca14cb8da Back out "Do not propagate real tensor in extern kernel" (#151813) 2025-04-21T21:42:34.7655781Z ##[endgroup] 2025-04-21T21:42:34.7657857Z ##[group]Disabling automatic garbage collection 2025-04-21T21:42:34.7663510Z [command]/usr/bin/git config --local gc.auto 0 2025-04-21T21:42:34.7700210Z ##[endgroup] 2025-04-21T21:42:34.7700748Z ##[group]Setting up auth 2025-04-21T21:42:34.7707342Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-04-21T21:42:34.7740777Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-04-21T21:42:34.8116445Z Entering 'android/libs/fbjni' 2025-04-21T21:42:34.8185935Z Entering 'third_party/FP16' 2025-04-21T21:42:34.8256705Z Entering 'third_party/FXdiv' 2025-04-21T21:42:34.8326272Z Entering 'third_party/NNPACK' 2025-04-21T21:42:34.8395834Z Entering 'third_party/NVTX' 2025-04-21T21:42:34.8465012Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:34.8533650Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:34.8618954Z Entering 'third_party/benchmark' 2025-04-21T21:42:34.8689344Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:34.8764067Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:34.8832265Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:34.8902861Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:34.8970554Z Entering 'third_party/cutlass' 2025-04-21T21:42:34.9048184Z Entering 'third_party/eigen' 2025-04-21T21:42:34.9118138Z Entering 'third_party/fbgemm' 2025-04-21T21:42:34.9187813Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:34.9254447Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:34.9320831Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:34.9393678Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:34.9460850Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:34.9530701Z Entering 'third_party/flash-attention' 2025-04-21T21:42:34.9598747Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:34.9671162Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:34.9747227Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:34.9818142Z Entering 'third_party/fmt' 2025-04-21T21:42:34.9885469Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:34.9954658Z Entering 'third_party/gloo' 2025-04-21T21:42:35.0024020Z Entering 'third_party/googletest' 2025-04-21T21:42:35.0092399Z Entering 'third_party/ideep' 2025-04-21T21:42:35.0159032Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:35.0236949Z Entering 'third_party/ittapi' 2025-04-21T21:42:35.0305461Z Entering 'third_party/kineto' 2025-04-21T21:42:35.0372294Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:35.0443820Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:35.0512807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:35.0579508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:35.0647550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:35.0717557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:35.0788215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:35.0857155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:35.0924816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:35.0992053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:35.1063719Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:35.1130721Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:35.1201556Z Entering 'third_party/kleidiai' 2025-04-21T21:42:35.1271493Z Entering 'third_party/mimalloc' 2025-04-21T21:42:35.1341100Z Entering 'third_party/nlohmann' 2025-04-21T21:42:35.1416751Z Entering 'third_party/onnx' 2025-04-21T21:42:35.1501083Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:35.1573481Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:35.1644539Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:35.1714002Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:35.1779087Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:35.1845889Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:35.1912958Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:35.1978005Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:35.2047366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:35.2113195Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:35.2181934Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:35.2250783Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:35.2341425Z Entering 'third_party/pocketfft' 2025-04-21T21:42:35.2412519Z Entering 'third_party/protobuf' 2025-04-21T21:42:35.2488004Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:35.2554926Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:35.2625012Z Entering 'third_party/psimd' 2025-04-21T21:42:35.2693286Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:35.2761444Z Entering 'third_party/pybind11' 2025-04-21T21:42:35.2829725Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:35.2899403Z Entering 'third_party/sleef' 2025-04-21T21:42:35.2966307Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:35.3033993Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:35.3100526Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:35.3165405Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:35.3233895Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:35.3298092Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:35.3393292Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-04-21T21:42:35.3426069Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-04-21T21:42:35.3789963Z Entering 'android/libs/fbjni' 2025-04-21T21:42:35.3859558Z Entering 'third_party/FP16' 2025-04-21T21:42:35.3928759Z Entering 'third_party/FXdiv' 2025-04-21T21:42:35.3997534Z Entering 'third_party/NNPACK' 2025-04-21T21:42:35.4065902Z Entering 'third_party/NVTX' 2025-04-21T21:42:35.4135772Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:35.4203919Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:35.4287780Z Entering 'third_party/benchmark' 2025-04-21T21:42:35.4356369Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:35.4431134Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:35.4501559Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:35.4575667Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:35.4644952Z Entering 'third_party/cutlass' 2025-04-21T21:42:35.4721736Z Entering 'third_party/eigen' 2025-04-21T21:42:35.4792507Z Entering 'third_party/fbgemm' 2025-04-21T21:42:35.4860761Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:35.4927709Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:35.4993939Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:35.5067209Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:35.5134551Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:35.5204021Z Entering 'third_party/flash-attention' 2025-04-21T21:42:35.5272998Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:35.5346101Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:35.5430198Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:35.5504441Z Entering 'third_party/fmt' 2025-04-21T21:42:35.5574589Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:35.5644631Z Entering 'third_party/gloo' 2025-04-21T21:42:35.5713943Z Entering 'third_party/googletest' 2025-04-21T21:42:35.5783563Z Entering 'third_party/ideep' 2025-04-21T21:42:35.5851112Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:35.5930156Z Entering 'third_party/ittapi' 2025-04-21T21:42:35.5999761Z Entering 'third_party/kineto' 2025-04-21T21:42:35.6068423Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:35.6135382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:35.6207170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:35.6275164Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:35.6343149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:35.6408808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:35.6481440Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:35.6549650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:35.6619027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:35.6688122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:35.6759631Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:35.6826725Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:35.6895528Z Entering 'third_party/kleidiai' 2025-04-21T21:42:35.6966637Z Entering 'third_party/mimalloc' 2025-04-21T21:42:35.7036370Z Entering 'third_party/nlohmann' 2025-04-21T21:42:35.7106525Z Entering 'third_party/onnx' 2025-04-21T21:42:35.7190366Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:35.7263373Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:35.7332357Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:35.7400058Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:35.7470052Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:35.7536610Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:35.7605051Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:35.7670780Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:35.7736927Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:35.7800677Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:35.7869538Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:35.7942423Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:35.8031897Z Entering 'third_party/pocketfft' 2025-04-21T21:42:35.8103619Z Entering 'third_party/protobuf' 2025-04-21T21:42:35.8175757Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:35.8243212Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:35.8314040Z Entering 'third_party/psimd' 2025-04-21T21:42:35.8383452Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:35.8453361Z Entering 'third_party/pybind11' 2025-04-21T21:42:35.8523575Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:35.8592822Z Entering 'third_party/sleef' 2025-04-21T21:42:35.8662883Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:35.8731949Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:35.8800830Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:35.8866427Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:35.8935194Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:35.9000411Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:35.9098629Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-04-21T21:42:35.9152374Z ##[endgroup] 2025-04-21T21:42:35.9152914Z ##[group]Fetching the repository 2025-04-21T21:42:35.9159521Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-04-21T21:42:39.1836327Z From https://github.com/pytorch/pytorch 2025-04-21T21:42:39.1836861Z - [deleted] (none) -> ciflow/inductor/148622 2025-04-21T21:42:39.2521894Z - [deleted] (none) -> ciflow/inductor/149690 2025-04-21T21:42:39.2522349Z - [deleted] (none) -> ciflow/inductor/149711 2025-04-21T21:42:39.2523245Z - [deleted] (none) -> ciflow/inductor/151506 2025-04-21T21:42:39.2525200Z - [deleted] (none) -> ciflow/trunk/148622 2025-04-21T21:42:39.2526240Z - [deleted] (none) -> ciflow/trunk/149711 2025-04-21T21:42:39.2528031Z - [deleted] (none) -> ciflow/trunk/151046 2025-04-21T21:42:39.2529158Z - [deleted] (none) -> ciflow/trunk/151672 2025-04-21T21:42:39.2530938Z - [deleted] (none) -> ciflow/trunk/151704 2025-04-21T21:42:39.2532019Z - [deleted] (none) -> ciflow/trunk/151716 2025-04-21T21:42:39.2533857Z - [deleted] (none) -> ciflow/trunk/151794 2025-04-21T21:42:40.4171058Z ed1e3930b3f..a840b9b1ba8 bf/fa-small-block-mask -> origin/bf/fa-small-block-mask 2025-04-21T21:42:40.4174642Z * [new branch] eqy-patch-6 -> origin/eqy-patch-6 2025-04-21T21:42:40.4214445Z b95d854985a..3e5aca84441 gh/laithsakka/119/base -> origin/gh/laithsakka/119/base 2025-04-21T21:42:40.4221027Z 2e26d078915..b68c0effe97 gh/laithsakka/119/head -> origin/gh/laithsakka/119/head 2025-04-21T21:42:40.4227214Z + 19dc41198c9...55d0c158068 gh/laithsakka/119/orig -> origin/gh/laithsakka/119/orig (forced update) 2025-04-21T21:42:40.4235160Z eaba5732110..a9a371e2cbb gh/laithsakka/143/base -> origin/gh/laithsakka/143/base 2025-04-21T21:42:40.4242649Z 12685772308..e408d0201d5 gh/laithsakka/143/head -> origin/gh/laithsakka/143/head 2025-04-21T21:42:40.4249995Z + 5a4cbcd8a74...e3fa1be9109 gh/laithsakka/143/orig -> origin/gh/laithsakka/143/orig (forced update) 2025-04-21T21:42:40.4274857Z bbd593720e2..0de0a192d66 gh/laithsakka/144/base -> origin/gh/laithsakka/144/base 2025-04-21T21:42:40.4282365Z 8b8f4016ba3..90c985a64c3 gh/laithsakka/144/head -> origin/gh/laithsakka/144/head 2025-04-21T21:42:40.4289642Z + 5cae6c1dc5e...c76ceea1d31 gh/laithsakka/144/orig -> origin/gh/laithsakka/144/orig (forced update) 2025-04-21T21:42:40.4297110Z 8b8f4016ba3..91110def275 gh/laithsakka/145/base -> origin/gh/laithsakka/145/base 2025-04-21T21:42:40.4305706Z 61c653a8a41..79bd94ff19f gh/laithsakka/145/head -> origin/gh/laithsakka/145/head 2025-04-21T21:42:40.4314849Z + 1c79f4f257d...cf312ce1ac8 gh/laithsakka/145/orig -> origin/gh/laithsakka/145/orig (forced update) 2025-04-21T21:42:40.4330341Z 5c603ac1ba8..f71a3d90cd1 gh/mlazos/29/base -> origin/gh/mlazos/29/base 2025-04-21T21:42:40.4337705Z d5699eb0ca4..ba361774d76 gh/mlazos/29/head -> origin/gh/mlazos/29/head 2025-04-21T21:42:40.4344510Z + ceacf9edd65...7463b297ab7 gh/mlazos/29/orig -> origin/gh/mlazos/29/orig (forced update) 2025-04-21T21:42:40.4351494Z fea3e205db8..f0051bf4cf3 gh/mlazos/30/base -> origin/gh/mlazos/30/base 2025-04-21T21:42:40.4358687Z b367c377e05..36d13fd7746 gh/mlazos/30/head -> origin/gh/mlazos/30/head 2025-04-21T21:42:40.4366042Z + 16f0eb41fb0...386d8af2dab gh/mlazos/30/orig -> origin/gh/mlazos/30/orig (forced update) 2025-04-21T21:42:40.4373358Z 4a9a383ded5..7423b341c9f gh/mlazos/31/base -> origin/gh/mlazos/31/base 2025-04-21T21:42:40.4380709Z 9dd543831cf..c2cf8791922 gh/mlazos/31/head -> origin/gh/mlazos/31/head 2025-04-21T21:42:40.4387334Z + 0ca0b7d756f...71abc0a02df gh/mlazos/31/orig -> origin/gh/mlazos/31/orig (forced update) 2025-04-21T21:42:40.4395329Z edd39a0d4ee..4447800f753 gh/syed-ahmed/1/head -> origin/gh/syed-ahmed/1/head 2025-04-21T21:42:40.4397204Z + 469a4dc35e1...0f317493c86 gh/syed-ahmed/1/orig -> origin/gh/syed-ahmed/1/orig (forced update) 2025-04-21T21:42:40.4403203Z 4c18280dfe5..0bb3c87007f gh/xmfan/214/base -> origin/gh/xmfan/214/base 2025-04-21T21:42:40.4405126Z f07fd36146a..ad7c2ba8b2d gh/xmfan/214/head -> origin/gh/xmfan/214/head 2025-04-21T21:42:40.4406927Z + 257a8caa97f...f7dd7d4da17 gh/xmfan/214/orig -> origin/gh/xmfan/214/orig (forced update) 2025-04-21T21:42:40.4409334Z 10eb069c930..1aeb85157e3 gh/xmfan/224/base -> origin/gh/xmfan/224/base 2025-04-21T21:42:40.4411070Z c5cad517a32..0c5e455e3eb gh/xmfan/224/head -> origin/gh/xmfan/224/head 2025-04-21T21:42:40.4412969Z + 9cf7d1444e7...29777b78279 gh/xmfan/224/orig -> origin/gh/xmfan/224/orig (forced update) 2025-04-21T21:42:40.4420145Z b7c70007280..c312d8c5018 main -> origin/main 2025-04-21T21:42:40.4429023Z + 0ca0b7d756f...71abc0a02df mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 (forced update) 2025-04-21T21:42:40.4432685Z * [new branch] pianpwk/01_auto_warning -> origin/pianpwk/01_auto_warning 2025-04-21T21:42:40.4434702Z * [new branch] pianpwk/dynamo_export_ctx -> origin/pianpwk/dynamo_export_ctx 2025-04-21T21:42:40.4443467Z 30cb4fc17d5..cceac144146 pianpwk/oblivious_expand -> origin/pianpwk/oblivious_expand 2025-04-21T21:42:40.4458308Z 10001864160..9d80fe4b89f pianpwk/skt_user_code -> origin/pianpwk/skt_user_code 2025-04-21T21:42:40.4468470Z 2eacdb91c3d..33808f0ebdc viable/strict -> origin/viable/strict 2025-04-21T21:42:40.4475234Z t [tag update] ciflow/inductor/148893 -> ciflow/inductor/148893 2025-04-21T21:42:40.4477282Z t [tag update] ciflow/inductor/149707 -> ciflow/inductor/149707 2025-04-21T21:42:40.4479296Z t [tag update] ciflow/inductor/150236 -> ciflow/inductor/150236 2025-04-21T21:42:40.4481852Z t [tag update] ciflow/inductor/151023 -> ciflow/inductor/151023 2025-04-21T21:42:40.4483511Z t [tag update] ciflow/inductor/151038 -> ciflow/inductor/151038 2025-04-21T21:42:40.4485334Z t [tag update] ciflow/inductor/151041 -> ciflow/inductor/151041 2025-04-21T21:42:40.4487159Z t [tag update] ciflow/inductor/151293 -> ciflow/inductor/151293 2025-04-21T21:42:40.4488719Z t [tag update] ciflow/inductor/151294 -> ciflow/inductor/151294 2025-04-21T21:42:40.4490356Z t [tag update] ciflow/inductor/151295 -> ciflow/inductor/151295 2025-04-21T21:42:40.4492410Z t [tag update] ciflow/inductor/151601 -> ciflow/inductor/151601 2025-04-21T21:42:40.4494130Z t [tag update] ciflow/inductor/151634 -> ciflow/inductor/151634 2025-04-21T21:42:40.4496498Z t [tag update] ciflow/inductor/151731 -> ciflow/inductor/151731 2025-04-21T21:42:40.4498985Z * [new tag] ciflow/inductor/151824 -> ciflow/inductor/151824 2025-04-21T21:42:40.4500518Z * [new tag] ciflow/inductor/151825 -> ciflow/inductor/151825 2025-04-21T21:42:40.4502289Z * [new tag] ciflow/inductor/151828 -> ciflow/inductor/151828 2025-04-21T21:42:40.4503756Z * [new tag] ciflow/inductor/151833 -> ciflow/inductor/151833 2025-04-21T21:42:40.4506510Z t [tag update] ciflow/mps/147644 -> ciflow/mps/147644 2025-04-21T21:42:40.4510922Z * [new tag] ciflow/trunk/147476 -> ciflow/trunk/147476 2025-04-21T21:42:40.4513630Z * [new tag] ciflow/trunk/150897 -> ciflow/trunk/150897 2025-04-21T21:42:40.4515271Z t [tag update] ciflow/trunk/151023 -> ciflow/trunk/151023 2025-04-21T21:42:40.4517201Z t [tag update] ciflow/trunk/151253 -> ciflow/trunk/151253 2025-04-21T21:42:40.4518764Z t [tag update] ciflow/trunk/151295 -> ciflow/trunk/151295 2025-04-21T21:42:40.4520773Z t [tag update] ciflow/trunk/151634 -> ciflow/trunk/151634 2025-04-21T21:42:40.4522319Z * [new tag] ciflow/trunk/151701 -> ciflow/trunk/151701 2025-04-21T21:42:40.4524105Z * [new tag] ciflow/trunk/151795 -> ciflow/trunk/151795 2025-04-21T21:42:40.4525667Z * [new tag] ciflow/trunk/151797 -> ciflow/trunk/151797 2025-04-21T21:42:40.4527585Z t [tag update] ciflow/trunk/151798 -> ciflow/trunk/151798 2025-04-21T21:42:40.4529116Z * [new tag] ciflow/trunk/151816 -> ciflow/trunk/151816 2025-04-21T21:42:40.5356121Z [command]/usr/bin/git rev-parse --verify --quiet b68c0effe97351b0c24b2f7aef671559be6aa977^{object} 2025-04-21T21:42:40.5385858Z b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:42:40.5391786Z ##[endgroup] 2025-04-21T21:42:40.5392204Z ##[group]Determining the checkout info 2025-04-21T21:42:40.5392690Z ##[endgroup] 2025-04-21T21:42:40.5398580Z [command]/usr/bin/git sparse-checkout disable 2025-04-21T21:42:40.8415267Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-04-21T21:42:40.8450882Z ##[group]Checking out the ref 2025-04-21T21:42:40.8453583Z [command]/usr/bin/git checkout --progress --force b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:42:41.0091484Z Previous HEAD position was cfca14cb8da Back out "Do not propagate real tensor in extern kernel" (#151813) 2025-04-21T21:42:41.0100974Z HEAD is now at b68c0effe97 Update on "use statically known true instead of guard size oblivious in bmm and mm decompositions . " 2025-04-21T21:42:41.0156148Z ##[endgroup] 2025-04-21T21:42:41.0156610Z ##[group]Setting up auth for fetching submodules 2025-04-21T21:42:41.0162767Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-04-21T21:42:41.0218060Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-04-21T21:42:41.0260291Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-04-21T21:42:41.0283687Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-04-21T21:42:41.0313602Z ##[endgroup] 2025-04-21T21:42:41.0314260Z ##[group]Fetching submodules 2025-04-21T21:42:41.0316916Z [command]/usr/bin/git submodule sync --recursive 2025-04-21T21:42:41.0683920Z Synchronizing submodule url for 'android/libs/fbjni' 2025-04-21T21:42:41.0711056Z Synchronizing submodule url for 'third_party/FP16' 2025-04-21T21:42:41.0738906Z Synchronizing submodule url for 'third_party/FXdiv' 2025-04-21T21:42:41.0765318Z Synchronizing submodule url for 'third_party/NNPACK' 2025-04-21T21:42:41.0792584Z Synchronizing submodule url for 'third_party/NVTX' 2025-04-21T21:42:41.0821490Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:41.0848012Z Synchronizing submodule url for 'third_party/XNNPACK' 2025-04-21T21:42:41.0890309Z Synchronizing submodule url for 'third_party/benchmark' 2025-04-21T21:42:41.0919351Z Synchronizing submodule url for 'third_party/composable_kernel' 2025-04-21T21:42:41.0953608Z Synchronizing submodule url for 'third_party/cpp-httplib' 2025-04-21T21:42:41.0980605Z Synchronizing submodule url for 'third_party/cpuinfo' 2025-04-21T21:42:41.1009264Z Synchronizing submodule url for 'third_party/cudnn_frontend' 2025-04-21T21:42:41.1035801Z Synchronizing submodule url for 'third_party/cutlass' 2025-04-21T21:42:41.1071571Z Synchronizing submodule url for 'third_party/eigen' 2025-04-21T21:42:41.1103145Z Synchronizing submodule url for 'third_party/fbgemm' 2025-04-21T21:42:41.1130286Z Synchronizing submodule url for 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:41.1155180Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:41.1181605Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:41.1215174Z Synchronizing submodule url for 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:41.1241279Z Synchronizing submodule url for 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:41.1271846Z Synchronizing submodule url for 'third_party/flash-attention' 2025-04-21T21:42:41.1298784Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:41.1331433Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:41.1369688Z Synchronizing submodule url for 'third_party/flatbuffers' 2025-04-21T21:42:41.1406134Z Synchronizing submodule url for 'third_party/fmt' 2025-04-21T21:42:41.1433414Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:41.1460785Z Synchronizing submodule url for 'third_party/gloo' 2025-04-21T21:42:41.1487802Z Synchronizing submodule url for 'third_party/googletest' 2025-04-21T21:42:41.1516674Z Synchronizing submodule url for 'third_party/ideep' 2025-04-21T21:42:41.1539816Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:41.1575357Z Synchronizing submodule url for 'third_party/ittapi' 2025-04-21T21:42:41.1605095Z Synchronizing submodule url for 'third_party/kineto' 2025-04-21T21:42:41.1631782Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:41.1657170Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:41.1686017Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:41.1714984Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:41.1744099Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:41.1768692Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:41.1800108Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:41.1828148Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:41.1856121Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:41.1884631Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:41.1915441Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:41.1941573Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:41.1971593Z Synchronizing submodule url for 'third_party/kleidiai' 2025-04-21T21:42:41.2001184Z Synchronizing submodule url for 'third_party/mimalloc' 2025-04-21T21:42:41.2033570Z Synchronizing submodule url for 'third_party/nlohmann' 2025-04-21T21:42:41.2062416Z Synchronizing submodule url for 'third_party/onnx' 2025-04-21T21:42:41.2103459Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:41.2136720Z Synchronizing submodule url for 'third_party/opentelemetry-cpp' 2025-04-21T21:42:41.2164189Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:41.2190098Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:41.2217016Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:41.2242876Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:41.2270988Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:41.2297883Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:41.2324211Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:41.2348780Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:41.2377969Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:41.2410359Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:41.2459022Z Synchronizing submodule url for 'third_party/pocketfft' 2025-04-21T21:42:41.2487825Z Synchronizing submodule url for 'third_party/protobuf' 2025-04-21T21:42:41.2516279Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:41.2542447Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:41.2572020Z Synchronizing submodule url for 'third_party/psimd' 2025-04-21T21:42:41.2599526Z Synchronizing submodule url for 'third_party/pthreadpool' 2025-04-21T21:42:41.2627334Z Synchronizing submodule url for 'third_party/pybind11' 2025-04-21T21:42:41.2654970Z Synchronizing submodule url for 'third_party/python-peachpy' 2025-04-21T21:42:41.2682185Z Synchronizing submodule url for 'third_party/sleef' 2025-04-21T21:42:41.2716398Z Synchronizing submodule url for 'third_party/tensorpipe' 2025-04-21T21:42:41.2743353Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:41.2769052Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:41.2793951Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:41.2820631Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:41.2843212Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:41.2888765Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-04-21T21:42:41.3482559Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-04-21T21:42:41.3648311Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-04-21T21:42:41.3772046Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-04-21T21:42:41.4147473Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-04-21T21:42:41.4758358Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2025-04-21T21:42:41.5338482Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2025-04-21T21:42:42.7914357Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-04-21T21:42:42.8248443Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2025-04-21T21:42:43.2562811Z Submodule path 'third_party/composable_kernel': checked out '8086bbe3a78d931eb96fe12fdc014082e18d18d3' 2025-04-21T21:42:43.3168899Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2025-04-21T21:42:43.4604382Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-04-21T21:42:43.5113082Z Submodule path 'third_party/cudnn_frontend': checked out '8801fd7b31c2f798732ed1e1dd4711a9cde4217d' 2025-04-21T21:42:44.5015976Z Submodule path 'third_party/cutlass': checked out 'afa1772203677c5118fcd82537a9c8fefbcc7008' 2025-04-21T21:42:44.8521385Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2025-04-21T21:42:44.9482926Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2025-04-21T21:42:45.0062902Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2025-04-21T21:42:45.1328333Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2025-04-21T21:42:45.8332560Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2025-04-21T21:42:45.8952724Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2025-04-21T21:42:45.9098296Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2025-04-21T21:42:46.0488629Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-04-21T21:42:46.4662802Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-04-21T21:42:47.3795059Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-04-21T21:42:47.6046305Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2025-04-21T21:42:47.6498610Z Submodule path 'third_party/fmt': checked out '123913715afeb8a437e6388b4473fcc4753e1c9a' 2025-04-21T21:42:47.7004037Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-04-21T21:42:47.7386343Z Submodule path 'third_party/gloo': checked out 'c61070427610ccd923efe3e7f8b3eca12bbcc31a' 2025-04-21T21:42:47.8056278Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2025-04-21T21:42:47.8250711Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-04-21T21:42:48.6393124Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-04-21T21:42:48.6620563Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2025-04-21T21:42:48.7897346Z Submodule path 'third_party/kineto': checked out '5fa4bd8c8fb581a621f72957d0cb11431cc4714f' 2025-04-21T21:42:48.9012050Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-04-21T21:42:49.1427705Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-04-21T21:42:49.1713808Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-04-21T21:42:49.2204397Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-04-21T21:42:49.2406766Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-04-21T21:42:49.2530776Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-04-21T21:42:49.2784237Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-04-21T21:42:49.3376752Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-04-21T21:42:49.5009119Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-04-21T21:42:49.5248424Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-04-21T21:42:49.5747008Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-04-21T21:42:49.6355150Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-04-21T21:42:49.6957207Z Submodule path 'third_party/kleidiai': checked out 'ef685a13cfbe8d418aa2ed34350e21e4938358b6' 2025-04-21T21:42:49.7528696Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2025-04-21T21:42:49.9368531Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2025-04-21T21:42:50.6061981Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2025-04-21T21:42:50.6593276Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2025-04-21T21:42:50.7857821Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-04-21T21:42:50.8172508Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-04-21T21:42:50.8725011Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-04-21T21:42:50.8903041Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-04-21T21:42:51.0518882Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-04-21T21:42:51.0691930Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-04-21T21:42:51.0917053Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-04-21T21:42:51.1177776Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-04-21T21:42:51.4514948Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-04-21T21:42:51.5134791Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-04-21T21:42:52.4331854Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-04-21T21:42:52.4487071Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2025-04-21T21:42:52.8490184Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-04-21T21:42:52.8710817Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-04-21T21:42:52.9400417Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-04-21T21:42:52.9533283Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-04-21T21:42:52.9704327Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-04-21T21:42:53.0253582Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-04-21T21:42:53.0670430Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-04-21T21:42:53.1351907Z Submodule path 'third_party/sleef': checked out '56e1f79cb140fb9326d612d0be06b5250565cade' 2025-04-21T21:42:53.1817384Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2025-04-21T21:42:53.2428011Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-04-21T21:42:53.2743636Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-04-21T21:42:53.3689507Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2025-04-21T21:42:53.4104203Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-04-21T21:42:53.4224596Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-04-21T21:42:53.4277805Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-04-21T21:42:53.4638829Z Entering 'android/libs/fbjni' 2025-04-21T21:42:53.4693373Z Entering 'third_party/FP16' 2025-04-21T21:42:53.4747899Z Entering 'third_party/FXdiv' 2025-04-21T21:42:53.4802272Z Entering 'third_party/NNPACK' 2025-04-21T21:42:53.4854942Z Entering 'third_party/NVTX' 2025-04-21T21:42:53.4908362Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:53.4963574Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:53.5032828Z Entering 'third_party/benchmark' 2025-04-21T21:42:53.5085154Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:53.5144956Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:53.5200527Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:53.5253677Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:53.5307927Z Entering 'third_party/cutlass' 2025-04-21T21:42:53.5369478Z Entering 'third_party/eigen' 2025-04-21T21:42:53.5426140Z Entering 'third_party/fbgemm' 2025-04-21T21:42:53.5479881Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:53.5530288Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:53.5580448Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:53.5636992Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:53.5686106Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:53.5740762Z Entering 'third_party/flash-attention' 2025-04-21T21:42:53.5793295Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:53.5851597Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:53.5914008Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:53.5969639Z Entering 'third_party/fmt' 2025-04-21T21:42:53.6027133Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:53.6080019Z Entering 'third_party/gloo' 2025-04-21T21:42:53.6132787Z Entering 'third_party/googletest' 2025-04-21T21:42:53.6185678Z Entering 'third_party/ideep' 2025-04-21T21:42:53.6237208Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:53.6299490Z Entering 'third_party/ittapi' 2025-04-21T21:42:53.6352060Z Entering 'third_party/kineto' 2025-04-21T21:42:53.6404337Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:53.6454847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:53.6508228Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:53.6559707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:53.6611599Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:53.6661620Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:53.6716790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:53.6767704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:53.6820581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:53.6872544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:53.6929377Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:53.6979260Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:53.7034288Z Entering 'third_party/kleidiai' 2025-04-21T21:42:53.7086634Z Entering 'third_party/mimalloc' 2025-04-21T21:42:53.7140105Z Entering 'third_party/nlohmann' 2025-04-21T21:42:53.7194768Z Entering 'third_party/onnx' 2025-04-21T21:42:53.7263158Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:53.7320829Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:53.7374186Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:53.7426622Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:53.7477887Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:53.7532584Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:53.7584424Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:53.7635395Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:53.7685465Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:53.7740141Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:53.7792887Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:53.7847209Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:53.7920355Z Entering 'third_party/pocketfft' 2025-04-21T21:42:53.7975046Z Entering 'third_party/protobuf' 2025-04-21T21:42:53.8033312Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:53.8083431Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:53.8137499Z Entering 'third_party/psimd' 2025-04-21T21:42:53.8190617Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:53.8244743Z Entering 'third_party/pybind11' 2025-04-21T21:42:53.8298388Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:53.8350952Z Entering 'third_party/sleef' 2025-04-21T21:42:53.8405858Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:53.8458326Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:53.8510363Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:53.8563291Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:53.8614986Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:53.8664743Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:53.8738222Z ##[endgroup] 2025-04-21T21:42:53.8738895Z ##[group]Persisting credentials for submodules 2025-04-21T21:42:53.8750366Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-04-21T21:42:53.9110418Z Entering 'android/libs/fbjni' 2025-04-21T21:42:53.9154394Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9154751Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9199033Z Entering 'third_party/FP16' 2025-04-21T21:42:53.9242863Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9243210Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9286211Z Entering 'third_party/FXdiv' 2025-04-21T21:42:53.9329615Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9329958Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9373184Z Entering 'third_party/NNPACK' 2025-04-21T21:42:53.9416407Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9416744Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9460185Z Entering 'third_party/NVTX' 2025-04-21T21:42:53.9504081Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9504428Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9548475Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:53.9591058Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9591403Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9634373Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:53.9677578Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9677905Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9735746Z Entering 'third_party/benchmark' 2025-04-21T21:42:53.9778578Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9778914Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9822653Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:53.9865386Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9865706Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9916571Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:53.9959414Z url.https://github.com/.insteadof 2025-04-21T21:42:53.9959872Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0003225Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:54.0047015Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0047493Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0091786Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:54.0135430Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0135908Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0179326Z Entering 'third_party/cutlass' 2025-04-21T21:42:54.0223132Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0223462Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0274901Z Entering 'third_party/eigen' 2025-04-21T21:42:54.0321700Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0322159Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0367142Z Entering 'third_party/fbgemm' 2025-04-21T21:42:54.0411116Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0411585Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0456229Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:54.0500078Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0500400Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0543394Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:54.0589296Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0589735Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0633693Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:54.0676705Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0677045Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0726531Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:54.0768192Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0768687Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0814557Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:54.0857915Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0858563Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0904701Z Entering 'third_party/flash-attention' 2025-04-21T21:42:54.0947939Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0948266Z url.https://github.com/.insteadof 2025-04-21T21:42:54.0991271Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:54.1033511Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1033846Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1082261Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:54.1125399Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1125879Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1179465Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:54.1223017Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1223356Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1268823Z Entering 'third_party/fmt' 2025-04-21T21:42:54.1314306Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1314676Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1357518Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:54.1401189Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1401529Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1445214Z Entering 'third_party/gloo' 2025-04-21T21:42:54.1488746Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1489403Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1535619Z Entering 'third_party/googletest' 2025-04-21T21:42:54.1579291Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1579925Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1627860Z Entering 'third_party/ideep' 2025-04-21T21:42:54.1671429Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1671774Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1714461Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:54.1756895Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1757235Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1811932Z Entering 'third_party/ittapi' 2025-04-21T21:42:54.1855412Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1855750Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1899852Z Entering 'third_party/kineto' 2025-04-21T21:42:54.1944181Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1944508Z url.https://github.com/.insteadof 2025-04-21T21:42:54.1987601Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:54.2030131Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2030464Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2072405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:54.2116998Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2117345Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2166705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:54.2211113Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2211457Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2256057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:54.2300158Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2300499Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2345180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:54.2388915Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2389281Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2432321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:54.2479417Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2479813Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2528174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:54.2571117Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2571465Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2616868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:54.2659547Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2660088Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2704442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:54.2748295Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2748661Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2796767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:54.2839830Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2840183Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2888431Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:54.2932006Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2932358Z url.https://github.com/.insteadof 2025-04-21T21:42:54.2974857Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:54.3018675Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3019135Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3067988Z Entering 'third_party/kleidiai' 2025-04-21T21:42:54.3113191Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3113625Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3157619Z Entering 'third_party/mimalloc' 2025-04-21T21:42:54.3202103Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3202560Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3246241Z Entering 'third_party/nlohmann' 2025-04-21T21:42:54.3290408Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3290849Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3337247Z Entering 'third_party/onnx' 2025-04-21T21:42:54.3381765Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3382358Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3441403Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:54.3487090Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3487446Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3541415Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:54.3585941Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3586410Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3631236Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:54.3673774Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3674167Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3717705Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:54.3760270Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3760622Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3806666Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:54.3851428Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3851883Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3893327Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:54.3936659Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3937123Z url.https://github.com/.insteadof 2025-04-21T21:42:54.3981025Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:54.4023312Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4023755Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4066024Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:54.4110597Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4111104Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4153542Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:54.4196627Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4196965Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4239768Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:54.4282231Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4282653Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4328724Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:54.4371434Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4371866Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4419028Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:54.4461755Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4462107Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4529562Z Entering 'third_party/pocketfft' 2025-04-21T21:42:54.4574387Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4574758Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4619596Z Entering 'third_party/protobuf' 2025-04-21T21:42:54.4663455Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4663803Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4709940Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:54.4752537Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4752978Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4796046Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:54.4838451Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4838900Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4884380Z Entering 'third_party/psimd' 2025-04-21T21:42:54.4929131Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4929569Z url.https://github.com/.insteadof 2025-04-21T21:42:54.4972542Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:54.5017101Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5017456Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5061200Z Entering 'third_party/pybind11' 2025-04-21T21:42:54.5105462Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5105803Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5151031Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:54.5196780Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5197271Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5242889Z Entering 'third_party/sleef' 2025-04-21T21:42:54.5287089Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5287433Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5331811Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:54.5375939Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5376275Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5420766Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:54.5462664Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5462987Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5506376Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:54.5548773Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5549115Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5593346Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:54.5636317Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5636795Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5685361Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:54.5727848Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5728189Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5772680Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:54.5815809Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5816141Z url.https://github.com/.insteadof 2025-04-21T21:42:54.5886445Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-04-21T21:42:54.6241902Z Entering 'android/libs/fbjni' 2025-04-21T21:42:54.6305645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-04-21T21:42:54.6328026Z Entering 'third_party/FP16' 2025-04-21T21:42:54.6390246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-04-21T21:42:54.6412628Z Entering 'third_party/FXdiv' 2025-04-21T21:42:54.6474570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-04-21T21:42:54.6496551Z Entering 'third_party/NNPACK' 2025-04-21T21:42:54.6559459Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-04-21T21:42:54.6580653Z Entering 'third_party/NVTX' 2025-04-21T21:42:54.6641870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-04-21T21:42:54.6664325Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:54.6726025Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-04-21T21:42:54.6748170Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:54.6809908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-04-21T21:42:54.6846960Z Entering 'third_party/benchmark' 2025-04-21T21:42:54.6908992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-04-21T21:42:54.6930650Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:54.6995232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-04-21T21:42:54.7023403Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:54.7084696Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-04-21T21:42:54.7106476Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:54.7166915Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-04-21T21:42:54.7189755Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:54.7251657Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-04-21T21:42:54.7273725Z Entering 'third_party/cutlass' 2025-04-21T21:42:54.7336419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-04-21T21:42:54.7366691Z Entering 'third_party/eigen' 2025-04-21T21:42:54.7433124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2025-04-21T21:42:54.7456074Z Entering 'third_party/fbgemm' 2025-04-21T21:42:54.7518780Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-04-21T21:42:54.7540593Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:54.7603002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2025-04-21T21:42:54.7623971Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:54.7683882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2025-04-21T21:42:54.7705609Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:54.7765679Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2025-04-21T21:42:54.7792728Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:54.7854371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:54.7874701Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:54.7936462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2025-04-21T21:42:54.7959406Z Entering 'third_party/flash-attention' 2025-04-21T21:42:54.8026200Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-04-21T21:42:54.8047147Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:54.8107959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-04-21T21:42:54.8134888Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:54.8195223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-04-21T21:42:54.8226157Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:54.8287465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-04-21T21:42:54.8313617Z Entering 'third_party/fmt' 2025-04-21T21:42:54.8374873Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-04-21T21:42:54.8397227Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:54.8458357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-04-21T21:42:54.8480176Z Entering 'third_party/gloo' 2025-04-21T21:42:54.8542350Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-04-21T21:42:54.8564320Z Entering 'third_party/googletest' 2025-04-21T21:42:54.8626682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:54.8648305Z Entering 'third_party/ideep' 2025-04-21T21:42:54.8710240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-04-21T21:42:54.8730327Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:54.8789385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-04-21T21:42:54.8819653Z Entering 'third_party/ittapi' 2025-04-21T21:42:54.8881000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-04-21T21:42:54.8906639Z Entering 'third_party/kineto' 2025-04-21T21:42:54.8973579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-04-21T21:42:54.8994933Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:54.9056483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-04-21T21:42:54.9075245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:54.9138625Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-04-21T21:42:54.9165276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:54.9228044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-04-21T21:42:54.9249613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:54.9312102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-04-21T21:42:54.9333407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:54.9401296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-04-21T21:42:54.9422283Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:54.9487733Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-04-21T21:42:54.9512223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:54.9573566Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-04-21T21:42:54.9594603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:54.9656223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:54.9678937Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:54.9741183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-04-21T21:42:54.9763524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:54.9831038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-04-21T21:42:54.9854715Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:54.9918105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-04-21T21:42:54.9939548Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:55.0004326Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-04-21T21:42:55.0028559Z Entering 'third_party/kleidiai' 2025-04-21T21:42:55.0090414Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-04-21T21:42:55.0114074Z Entering 'third_party/mimalloc' 2025-04-21T21:42:55.0174417Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-04-21T21:42:55.0197465Z Entering 'third_party/nlohmann' 2025-04-21T21:42:55.0259032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-04-21T21:42:55.0282164Z Entering 'third_party/onnx' 2025-04-21T21:42:55.0345158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-04-21T21:42:55.0380607Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:55.0442769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-04-21T21:42:55.0469158Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:55.0532837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-04-21T21:42:55.0554499Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:55.0617643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-04-21T21:42:55.0640034Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:55.0701670Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:55.0722276Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:55.0784436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-04-21T21:42:55.0804036Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:55.0866141Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-04-21T21:42:55.0888437Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:55.0950700Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-04-21T21:42:55.0971138Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:55.1033111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-04-21T21:42:55.1054171Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:55.1120754Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-04-21T21:42:55.1139713Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:55.1203309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-04-21T21:42:55.1227186Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:55.1287758Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-04-21T21:42:55.1312611Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:55.1374202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-04-21T21:42:55.1419227Z Entering 'third_party/pocketfft' 2025-04-21T21:42:55.1481469Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-04-21T21:42:55.1503619Z Entering 'third_party/protobuf' 2025-04-21T21:42:55.1564974Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-04-21T21:42:55.1588376Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:55.1648960Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-04-21T21:42:55.1669481Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:55.1731860Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:55.1755937Z Entering 'third_party/psimd' 2025-04-21T21:42:55.1820091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-04-21T21:42:55.1840971Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:55.1904049Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-04-21T21:42:55.1925898Z Entering 'third_party/pybind11' 2025-04-21T21:42:55.1988188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-04-21T21:42:55.2012147Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:55.2074351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-04-21T21:42:55.2096773Z Entering 'third_party/sleef' 2025-04-21T21:42:55.2162410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-04-21T21:42:55.2184608Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:55.2248065Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-04-21T21:42:55.2272378Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:55.2333882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-04-21T21:42:55.2355089Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:55.2417782Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-04-21T21:42:55.2438801Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:55.2500426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-04-21T21:42:55.2523027Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:55.2584720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-04-21T21:42:55.2603781Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:55.2666802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-04-21T21:42:55.3766499Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-04-21T21:42:55.4129658Z Entering 'android/libs/fbjni' 2025-04-21T21:42:55.4185467Z Entering 'third_party/FP16' 2025-04-21T21:42:55.4242639Z Entering 'third_party/FXdiv' 2025-04-21T21:42:55.4295209Z Entering 'third_party/NNPACK' 2025-04-21T21:42:55.4347771Z Entering 'third_party/NVTX' 2025-04-21T21:42:55.4401074Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:55.4453321Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:55.4521142Z Entering 'third_party/benchmark' 2025-04-21T21:42:55.4573205Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:55.4635738Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:55.4701748Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:55.4743265Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:55.4796048Z Entering 'third_party/cutlass' 2025-04-21T21:42:55.4856708Z Entering 'third_party/eigen' 2025-04-21T21:42:55.4916241Z Entering 'third_party/fbgemm' 2025-04-21T21:42:55.4969154Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:55.5020869Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:55.5070243Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:55.5128796Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:55.5179362Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:55.5232501Z Entering 'third_party/flash-attention' 2025-04-21T21:42:55.5285252Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:55.5343576Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:55.5406011Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:55.5461476Z Entering 'third_party/fmt' 2025-04-21T21:42:55.5514217Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:55.5567549Z Entering 'third_party/gloo' 2025-04-21T21:42:55.5625715Z Entering 'third_party/googletest' 2025-04-21T21:42:55.5683769Z Entering 'third_party/ideep' 2025-04-21T21:42:55.5735191Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:55.5795682Z Entering 'third_party/ittapi' 2025-04-21T21:42:55.5848049Z Entering 'third_party/kineto' 2025-04-21T21:42:55.5904719Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:55.5954912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:55.6008632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:55.6060359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:55.6113541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:55.6163108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:55.6218395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:55.6275029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:55.6328803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:55.6383590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:55.6440612Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:55.6491027Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:55.6544880Z Entering 'third_party/kleidiai' 2025-04-21T21:42:55.6599760Z Entering 'third_party/mimalloc' 2025-04-21T21:42:55.6651912Z Entering 'third_party/nlohmann' 2025-04-21T21:42:55.6705852Z Entering 'third_party/onnx' 2025-04-21T21:42:55.6772822Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:55.6829098Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:55.6885428Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:55.6934420Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:55.6984701Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:55.7035246Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:55.7087614Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:55.7138076Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:55.7194761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:55.7243997Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:55.7297242Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:55.7353109Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:55.7426482Z Entering 'third_party/pocketfft' 2025-04-21T21:42:55.7480090Z Entering 'third_party/protobuf' 2025-04-21T21:42:55.7535772Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:55.7585623Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:55.7641217Z Entering 'third_party/psimd' 2025-04-21T21:42:55.7693556Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:55.7748183Z Entering 'third_party/pybind11' 2025-04-21T21:42:55.7801473Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:55.7853811Z Entering 'third_party/sleef' 2025-04-21T21:42:55.7907029Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:55.7958885Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:55.8010500Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:55.8061643Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:55.8112962Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:55.8162395Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:55.8238933Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-04-21T21:42:55.8596779Z Entering 'android/libs/fbjni' 2025-04-21T21:42:55.8650368Z Entering 'third_party/FP16' 2025-04-21T21:42:55.8708241Z Entering 'third_party/FXdiv' 2025-04-21T21:42:55.8761492Z Entering 'third_party/NNPACK' 2025-04-21T21:42:55.8813301Z Entering 'third_party/NVTX' 2025-04-21T21:42:55.8865306Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-21T21:42:55.8918016Z Entering 'third_party/XNNPACK' 2025-04-21T21:42:55.8985397Z Entering 'third_party/benchmark' 2025-04-21T21:42:55.9038207Z Entering 'third_party/composable_kernel' 2025-04-21T21:42:55.9097531Z Entering 'third_party/cpp-httplib' 2025-04-21T21:42:55.9149991Z Entering 'third_party/cpuinfo' 2025-04-21T21:42:55.9203099Z Entering 'third_party/cudnn_frontend' 2025-04-21T21:42:55.9254693Z Entering 'third_party/cutlass' 2025-04-21T21:42:55.9319937Z Entering 'third_party/eigen' 2025-04-21T21:42:55.9377249Z Entering 'third_party/fbgemm' 2025-04-21T21:42:55.9430285Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-21T21:42:55.9480205Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-21T21:42:55.9532476Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-21T21:42:55.9588945Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-21T21:42:55.9640052Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-21T21:42:55.9696488Z Entering 'third_party/flash-attention' 2025-04-21T21:42:55.9749097Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-21T21:42:55.9807424Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-21T21:42:55.9872049Z Entering 'third_party/flatbuffers' 2025-04-21T21:42:55.9927086Z Entering 'third_party/fmt' 2025-04-21T21:42:55.9983604Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-21T21:42:56.0036539Z Entering 'third_party/gloo' 2025-04-21T21:42:56.0088146Z Entering 'third_party/googletest' 2025-04-21T21:42:56.0145316Z Entering 'third_party/ideep' 2025-04-21T21:42:56.0195877Z Entering 'third_party/ideep/mkl-dnn' 2025-04-21T21:42:56.0256179Z Entering 'third_party/ittapi' 2025-04-21T21:42:56.0308594Z Entering 'third_party/kineto' 2025-04-21T21:42:56.0363399Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-21T21:42:56.0413826Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-21T21:42:56.0467973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-21T21:42:56.0521640Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-21T21:42:56.0573357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-21T21:42:56.0627071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-21T21:42:56.0682773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-21T21:42:56.0738005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-21T21:42:56.0790066Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-21T21:42:56.0844765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-21T21:42:56.0899787Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-21T21:42:56.0950088Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-21T21:42:56.1004190Z Entering 'third_party/kleidiai' 2025-04-21T21:42:56.1056699Z Entering 'third_party/mimalloc' 2025-04-21T21:42:56.1110481Z Entering 'third_party/nlohmann' 2025-04-21T21:42:56.1164523Z Entering 'third_party/onnx' 2025-04-21T21:42:56.1232451Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-21T21:42:56.1291979Z Entering 'third_party/opentelemetry-cpp' 2025-04-21T21:42:56.1349187Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-21T21:42:56.1399792Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-21T21:42:56.1450947Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-21T21:42:56.1505262Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-21T21:42:56.1556891Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-21T21:42:56.1607094Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-21T21:42:56.1656006Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-21T21:42:56.1705021Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-21T21:42:56.1757051Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-21T21:42:56.1813026Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-21T21:42:56.1885607Z Entering 'third_party/pocketfft' 2025-04-21T21:42:56.1942836Z Entering 'third_party/protobuf' 2025-04-21T21:42:56.1999769Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-21T21:42:56.2049656Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-21T21:42:56.2105552Z Entering 'third_party/psimd' 2025-04-21T21:42:56.2156920Z Entering 'third_party/pthreadpool' 2025-04-21T21:42:56.2211386Z Entering 'third_party/pybind11' 2025-04-21T21:42:56.2264657Z Entering 'third_party/python-peachpy' 2025-04-21T21:42:56.2317917Z Entering 'third_party/sleef' 2025-04-21T21:42:56.2375023Z Entering 'third_party/tensorpipe' 2025-04-21T21:42:56.2428573Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-21T21:42:56.2479988Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-21T21:42:56.2531841Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-21T21:42:56.2582576Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-21T21:42:56.2631691Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-21T21:42:56.2704208Z ##[endgroup] 2025-04-21T21:42:56.2747079Z [command]/usr/bin/git log -1 --format=%H 2025-04-21T21:42:56.2775816Z b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:42:56.2990022Z Prepare all required actions 2025-04-21T21:42:56.2990528Z Getting action download info 2025-04-21T21:42:56.4033216Z ##[group]Run ./.github/actions/setup-linux 2025-04-21T21:42:56.4033520Z env: 2025-04-21T21:42:56.4033725Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:56.4033976Z ##[endgroup] 2025-04-21T21:42:56.4096107Z ##[group]Run set -euo pipefail 2025-04-21T21:42:56.4096435Z set -euo pipefail 2025-04-21T21:42:56.4096728Z function get_ec2_metadata() { 2025-04-21T21:42:56.4097119Z  # Pulled from instance metadata endpoint for EC2 2025-04-21T21:42:56.4097853Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-04-21T21:42:56.4098403Z  category=$1 2025-04-21T21:42:56.4098757Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-04-21T21:42:56.4099225Z  runner_name_str=i-02e84e4685011efd7 2025-04-21T21:42:56.4099594Z  if [[ -f /.inarc ]]; then 2025-04-21T21:42:56.4099937Z  echo "ARC Runner, no info on ec2 metadata" 2025-04-21T21:42:56.4100324Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-04-21T21:42:56.4100793Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-04-21T21:42:56.4101207Z  else 2025-04-21T21:42:56.4102016Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-04-21T21:42:56.4102890Z  fi 2025-04-21T21:42:56.4103099Z } 2025-04-21T21:42:56.4103355Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-04-21T21:42:56.4103759Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-04-21T21:42:56.4104221Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-04-21T21:42:56.4104622Z echo "system info $(uname -a)" 2025-04-21T21:42:56.4113645Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:56.4114011Z env: 2025-04-21T21:42:56.4114224Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:56.4114472Z ##[endgroup] 2025-04-21T21:42:56.4268559Z ami-id: ami-071226ecf16aa7d96 2025-04-21T21:42:56.4378993Z instance-id: i-02e84e4685011efd7 2025-04-21T21:42:56.4486763Z instance-type: g5.4xlarge 2025-04-21T21:42:56.4500856Z system info Linux ip-10-0-3-190.ec2.internal 6.1.130-139.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 11 01:10:58 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-04-21T21:42:56.4532853Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:42:56.4533738Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:42:56.4542982Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:56.4543351Z env: 2025-04-21T21:42:56.4543570Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:56.4543822Z ##[endgroup] 2025-04-21T21:42:56.4608762Z ##[group]Run if systemctl is-active --quiet docker; then 2025-04-21T21:42:56.4609203Z if systemctl is-active --quiet docker; then 2025-04-21T21:42:56.4609596Z  echo "Docker daemon is running..."; 2025-04-21T21:42:56.4609928Z else 2025-04-21T21:42:56.4610274Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2025-04-21T21:42:56.4610704Z fi 2025-04-21T21:42:56.4618447Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:56.4618807Z env: 2025-04-21T21:42:56.4619019Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:56.4619273Z ##[endgroup] 2025-04-21T21:42:56.4703666Z Docker daemon is running... 2025-04-21T21:42:56.4752159Z ##[group]Run nick-fields/retry@v3.0.0 2025-04-21T21:42:56.4752460Z with: 2025-04-21T21:42:56.4752663Z shell: bash 2025-04-21T21:42:56.4753085Z timeout_minutes: 5 2025-04-21T21:42:56.4753328Z max_attempts: 3 2025-04-21T21:42:56.4753562Z retry_wait_seconds: 30 2025-04-21T21:42:56.4755692Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-04-21T21:42:56.4757879Z polling_interval_seconds: 1 2025-04-21T21:42:56.4758152Z warning_on_retry: true 2025-04-21T21:42:56.4758409Z continue_on_error: false 2025-04-21T21:42:56.4758652Z env: 2025-04-21T21:42:56.4758884Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:56.4759170Z AWS_RETRY_MODE: standard 2025-04-21T21:42:56.4759420Z AWS_MAX_ATTEMPTS: 5 2025-04-21T21:42:56.4759668Z AWS_DEFAULT_REGION: us-east-1 2025-04-21T21:42:56.4759938Z ##[endgroup] 2025-04-21T21:42:57.6115691Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-04-21T21:42:57.6116307Z Configure a credential helper to remove this warning. See 2025-04-21T21:42:57.6116870Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-04-21T21:42:57.6117299Z 2025-04-21T21:42:57.6117399Z Login Succeeded 2025-04-21T21:42:58.5596472Z Command completed after 1 attempt(s). 2025-04-21T21:42:58.5675269Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-04-21T21:42:58.5675780Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-04-21T21:42:58.5676227Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-04-21T21:42:58.5685837Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:58.5701519Z env: 2025-04-21T21:42:58.5701732Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:58.5701978Z ##[endgroup] 2025-04-21T21:42:58.5814331Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-04-21T21:42:58.5814871Z # ignore expansion of "docker ps -q" since it could be empty 2025-04-21T21:42:58.5815284Z # shellcheck disable=SC2046 2025-04-21T21:42:58.5815610Z docker stop $(docker ps -q) || true 2025-04-21T21:42:58.5816101Z # Prune all of the docker images 2025-04-21T21:42:58.5816416Z docker system prune -af 2025-04-21T21:42:58.5824779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:58.5825137Z env: 2025-04-21T21:42:58.5825346Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:58.5825599Z ##[endgroup] 2025-04-21T21:42:58.6118493Z "docker stop" requires at least 1 argument. 2025-04-21T21:42:58.6118870Z See 'docker stop --help'. 2025-04-21T21:42:58.6119047Z 2025-04-21T21:42:58.6119228Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-04-21T21:42:58.6119499Z 2025-04-21T21:42:58.6119618Z Stop one or more running containers 2025-04-21T21:42:58.6293187Z Total reclaimed space: 0B 2025-04-21T21:42:58.6345933Z ##[group]Run set +e 2025-04-21T21:42:58.6346217Z set +e 2025-04-21T21:42:58.6346434Z set -x 2025-04-21T21:42:58.6346637Z  2025-04-21T21:42:58.6346871Z PT_DOMAIN=download.pytorch.org 2025-04-21T21:42:58.6347415Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-04-21T21:42:58.6348154Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-04-21T21:42:58.6348648Z # one is returned at random 2025-04-21T21:42:58.6349024Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-04-21T21:42:58.6349383Z  2025-04-21T21:42:58.6349813Z if [ -z "${RESOLVED_IP}" ]; then 2025-04-21T21:42:58.6350229Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-04-21T21:42:58.6350722Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-04-21T21:42:58.6351092Z  2025-04-21T21:42:58.6351312Z  if [ -z "${RESOLVED_IP}" ]; then 2025-04-21T21:42:58.6351680Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-04-21T21:42:58.6352023Z  exit 1 2025-04-21T21:42:58.6352238Z  fi 2025-04-21T21:42:58.6352443Z fi 2025-04-21T21:42:58.6352639Z  2025-04-21T21:42:58.6352881Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-04-21T21:42:58.6353234Z  # Clean up any old records first 2025-04-21T21:42:58.6353579Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-04-21T21:42:58.6353881Z fi 2025-04-21T21:42:58.6354084Z  2025-04-21T21:42:58.6354395Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-04-21T21:42:58.6354777Z cat /etc/hosts 2025-04-21T21:42:58.6363453Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:58.6363815Z env: 2025-04-21T21:42:58.6364026Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:58.6364275Z ##[endgroup] 2025-04-21T21:42:58.6390756Z + PT_DOMAIN=download.pytorch.org 2025-04-21T21:42:58.6397527Z ++ dig -4 +short download.pytorch.org 2025-04-21T21:42:58.6397951Z ++ tail -n1 2025-04-21T21:42:58.6711985Z + RESOLVED_IP=18.160.10.28 2025-04-21T21:42:58.6712316Z + '[' -z 18.160.10.28 ']' 2025-04-21T21:42:58.6712656Z + grep -r download.pytorch.org /etc/hosts 2025-04-21T21:42:58.6725113Z 18.160.10.76 download.pytorch.org 2025-04-21T21:42:58.6727199Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2025-04-21T21:42:58.8001311Z + echo '18.160.10.28 download.pytorch.org' 2025-04-21T21:42:58.8002011Z + sudo tee -a /etc/hosts 2025-04-21T21:42:59.0754275Z 18.160.10.28 download.pytorch.org 2025-04-21T21:42:59.0775962Z + cat /etc/hosts 2025-04-21T21:42:59.0786365Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-04-21T21:42:59.0791558Z ::1 localhost6 localhost6.localdomain6 2025-04-21T21:42:59.0792018Z 18.160.10.28 download.pytorch.org 2025-04-21T21:42:59.0950968Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-04-21T21:42:59.0951418Z with: 2025-04-21T21:42:59.0952221Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.0953284Z docker-build-dir: .ci/docker 2025-04-21T21:42:59.0953582Z docker-build-script: ./build.sh 2025-04-21T21:42:59.0953870Z working-directory: . 2025-04-21T21:42:59.0954216Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.0954605Z force-push: false 2025-04-21T21:42:59.0954822Z env: 2025-04-21T21:42:59.0955032Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:59.0955284Z ##[endgroup] 2025-04-21T21:42:59.1001771Z ##[group]Run set -ex 2025-04-21T21:42:59.1002042Z set -ex 2025-04-21T21:42:59.1002245Z  2025-04-21T21:42:59.1002640Z # If the docker build directory or the build script doesn't exist, the action will 2025-04-21T21:42:59.1003261Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-04-21T21:42:59.1003786Z # job could then download the pre-built image as usual 2025-04-21T21:42:59.1004316Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]]; then 2025-04-21T21:42:59.1004802Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1005220Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1005597Z  2025-04-21T21:42:59.1005923Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2025-04-21T21:42:59.1006324Z  exit 0 2025-04-21T21:42:59.1006530Z else 2025-04-21T21:42:59.1006782Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1007086Z fi 2025-04-21T21:42:59.1007279Z  2025-04-21T21:42:59.1007600Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-04-21T21:42:59.1008165Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-04-21T21:42:59.1008673Z  # use it as it is, but first let's extract the tag 2025-04-21T21:42:59.1009125Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-04-21T21:42:59.1009608Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1010098Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1010503Z else 2025-04-21T21:42:59.1010930Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-04-21T21:42:59.1011504Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1012113Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1012636Z fi 2025-04-21T21:42:59.1022712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:59.1023071Z env: 2025-04-21T21:42:59.1023273Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:59.1023526Z REPO_NAME: pytorch 2025-04-21T21:42:59.1024337Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1025204Z DOCKER_BUILD_DIR: .ci/docker 2025-04-21T21:42:59.1025475Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-04-21T21:42:59.1025839Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.1026212Z CUSTOM_TAG_PREFIX: 2025-04-21T21:42:59.1026437Z ##[endgroup] 2025-04-21T21:42:59.1054186Z + [[ ! -d .ci/docker ]] 2025-04-21T21:42:59.1054580Z + [[ ! -f .ci/docker/./build.sh ]] 2025-04-21T21:42:59.1054967Z + echo skip=false 2025-04-21T21:42:59.1056447Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-04-21T21:42:59.1062663Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1063524Z ++ awk -F '[:,]' '{print $2}' 2025-04-21T21:42:59.1090427Z + DOCKER_TAG=f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1090830Z + echo docker-tag=f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1091756Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1126563Z ##[group]Run set +e 2025-04-21T21:42:59.1126858Z set +e 2025-04-21T21:42:59.1127098Z set -x 2025-04-21T21:42:59.1127316Z  2025-04-21T21:42:59.1127523Z login() { 2025-04-21T21:42:59.1127981Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-04-21T21:42:59.1128477Z } 2025-04-21T21:42:59.1128691Z  2025-04-21T21:42:59.1128890Z retry () { 2025-04-21T21:42:59.1129149Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-04-21T21:42:59.1129454Z } 2025-04-21T21:42:59.1129672Z  2025-04-21T21:42:59.1129925Z retry login "${DOCKER_REGISTRY}" 2025-04-21T21:42:59.1130215Z  2025-04-21T21:42:59.1130418Z START_TIME=$(date +%s) 2025-04-21T21:42:59.1130704Z # Wait up to 120 minutes 2025-04-21T21:42:59.1131054Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-04-21T21:42:59.1131526Z  # Check if image already exists, if it does then skip building it 2025-04-21T21:42:59.1132004Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-04-21T21:42:59.1132359Z  exit 0 2025-04-21T21:42:59.1132582Z  fi 2025-04-21T21:42:59.1132783Z  2025-04-21T21:42:59.1133150Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-04-21T21:42:59.1133789Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-04-21T21:42:59.1134422Z  # latter, it will wait for the Docker images to become available before continuing 2025-04-21T21:42:59.1134917Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-04-21T21:42:59.1135310Z  # It's a Docker build job, let's build the image 2025-04-21T21:42:59.1135646Z  break 2025-04-21T21:42:59.1135867Z  else 2025-04-21T21:42:59.1136193Z  # It's a regular build job, wait for the image to become available 2025-04-21T21:42:59.1136587Z  sleep 300 2025-04-21T21:42:59.1136824Z  fi 2025-04-21T21:42:59.1137035Z done 2025-04-21T21:42:59.1137243Z  2025-04-21T21:42:59.1137587Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-04-21T21:42:59.1138256Z # be empty. The default action would be to continue rebuild the image 2025-04-21T21:42:59.1138759Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-04-21T21:42:59.1139200Z  # if we're on the base branch then use the parent commit 2025-04-21T21:42:59.1139585Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-04-21T21:42:59.1139884Z else 2025-04-21T21:42:59.1140193Z  # otherwise we're on a PR, so use the most recent base commit 2025-04-21T21:42:59.1140642Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-04-21T21:42:59.1140984Z fi 2025-04-21T21:42:59.1141190Z  2025-04-21T21:42:59.1141414Z if [[ -z "${MERGE_BASE}" ]]; then 2025-04-21T21:42:59.1141936Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1142257Z  2025-04-21T21:42:59.1142701Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-04-21T21:42:59.1143354Z  exit 0 2025-04-21T21:42:59.1143563Z fi 2025-04-21T21:42:59.1143765Z  2025-04-21T21:42:59.1144062Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-04-21T21:42:59.1144707Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-04-21T21:42:59.1145262Z  exit 1 2025-04-21T21:42:59.1145479Z fi 2025-04-21T21:42:59.1145674Z  2025-04-21T21:42:59.1146017Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-04-21T21:42:59.1146646Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-04-21T21:42:59.1147208Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-04-21T21:42:59.1147852Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-04-21T21:42:59.1148584Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-04-21T21:42:59.1149023Z fi 2025-04-21T21:42:59.1149223Z  2025-04-21T21:42:59.1149468Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-04-21T21:42:59.1157569Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:59.1157930Z env: 2025-04-21T21:42:59.1158140Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:59.1158419Z DOCKER_BUILD_DIR: .ci/docker 2025-04-21T21:42:59.1158754Z BASE_REVISION: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:42:59.1159673Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1160565Z DOCKER_TAG: f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.1160986Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.1161362Z DOCKER_PUSH: 2025-04-21T21:42:59.1161583Z ##[endgroup] 2025-04-21T21:42:59.1188062Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.1188486Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.1191326Z + aws ecr get-login-password --region us-east-1 2025-04-21T21:42:59.1192318Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:42:59.6340144Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-04-21T21:42:59.6340743Z Configure a credential helper to remove this warning. See 2025-04-21T21:42:59.6341296Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-04-21T21:42:59.6341756Z 2025-04-21T21:42:59.6341895Z Login Succeeded 2025-04-21T21:42:59.6368726Z ++ date +%s 2025-04-21T21:42:59.6380278Z + START_TIME=1745271779 2025-04-21T21:42:59.6384725Z ++ date +%s 2025-04-21T21:42:59.6396975Z + [[ 1745264579 -lt 1745271779 ]] 2025-04-21T21:42:59.6397900Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:42:59.8587798Z { 2025-04-21T21:42:59.8588089Z "schemaVersion": 2, 2025-04-21T21:42:59.8589145Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-04-21T21:42:59.8589606Z "config": { 2025-04-21T21:42:59.8589937Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-04-21T21:42:59.8590320Z "size": 46736, 2025-04-21T21:42:59.8590731Z "digest": "sha256:7b29eacbf93f5bdd9e6b666f27edb34c0d2f289aa58ceaf3bdfcf4e06f505e55" 2025-04-21T21:42:59.8591208Z }, 2025-04-21T21:42:59.8591398Z "layers": [ 2025-04-21T21:42:59.8591874Z { 2025-04-21T21:42:59.8592203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8592601Z "size": 28583948, 2025-04-21T21:42:59.8593006Z "digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1" 2025-04-21T21:42:59.8593644Z }, 2025-04-21T21:42:59.8593891Z { 2025-04-21T21:42:59.8594571Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8595066Z "size": 7964619, 2025-04-21T21:42:59.8595460Z "digest": "sha256:49e139a3d6c2f1801aa0cea1eb34e57c5314065b679325df205026fb175383b8" 2025-04-21T21:42:59.8595894Z }, 2025-04-21T21:42:59.8596079Z { 2025-04-21T21:42:59.8596397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8596790Z "size": 57379226, 2025-04-21T21:42:59.8597194Z "digest": "sha256:a14844c8c51f98ebee2b5eda8bff8742dd804385e1dbbeb56928946113000293" 2025-04-21T21:42:59.8597643Z }, 2025-04-21T21:42:59.8597825Z { 2025-04-21T21:42:59.8598149Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8598544Z "size": 187, 2025-04-21T21:42:59.8598932Z "digest": "sha256:18fb524087fbed3307d23afde734ef1df452f55e7a22e28b56ce1e6eb9c6b3d9" 2025-04-21T21:42:59.8599387Z }, 2025-04-21T21:42:59.8599579Z { 2025-04-21T21:42:59.8599887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8600278Z "size": 6885, 2025-04-21T21:42:59.8600679Z "digest": "sha256:efd686a7b2c8bb052a3bb919973c4bee33c2bbcff08fd431362b90c9469fbc4d" 2025-04-21T21:42:59.8601129Z }, 2025-04-21T21:42:59.8601309Z { 2025-04-21T21:42:59.8601619Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8602015Z "size": 1438282733, 2025-04-21T21:42:59.8602409Z "digest": "sha256:52c648e213347a29001a45b2c4f834cc60621a3d6ab68fa06ebd2075f6487ec5" 2025-04-21T21:42:59.8602841Z }, 2025-04-21T21:42:59.8603026Z { 2025-04-21T21:42:59.8603341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8603733Z "size": 62694, 2025-04-21T21:42:59.8604131Z "digest": "sha256:56e384e4e5aa7ebddf31686fc6ae62ef19fdc5311bbea8186563fc2aeca2fb52" 2025-04-21T21:42:59.8604576Z }, 2025-04-21T21:42:59.8604762Z { 2025-04-21T21:42:59.8605072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8605457Z "size": 1684, 2025-04-21T21:42:59.8605845Z "digest": "sha256:fb71b792ec6c458c338cace5ce42f653e4ae4b24442a49b7c47af3a38342b5bc" 2025-04-21T21:42:59.8606287Z }, 2025-04-21T21:42:59.8606471Z { 2025-04-21T21:42:59.8606774Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8607166Z "size": 1523, 2025-04-21T21:42:59.8607542Z "digest": "sha256:5509576f2693abf94a50a70a59bbf8b519f20705f147d495ddde85981c8189fa" 2025-04-21T21:42:59.8607971Z }, 2025-04-21T21:42:59.8608147Z { 2025-04-21T21:42:59.8608463Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8608849Z "size": 2515216700, 2025-04-21T21:42:59.8609252Z "digest": "sha256:1e6c6f2d245956001435a5df27fd4defea08affa7f6c22252464f758447db401" 2025-04-21T21:42:59.8609692Z }, 2025-04-21T21:42:59.8609869Z { 2025-04-21T21:42:59.8610181Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8610578Z "size": 86619, 2025-04-21T21:42:59.8610963Z "digest": "sha256:a1fe8922734d371e0c0fd5b04183758331e08f45553720ecff4b062824e0c8e1" 2025-04-21T21:42:59.8611387Z }, 2025-04-21T21:42:59.8611573Z { 2025-04-21T21:42:59.8611880Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8612263Z "size": 1893, 2025-04-21T21:42:59.8612645Z "digest": "sha256:84e5433913c60a912877721ebf3a4d115b8bd55fb1c5a298b58fdc1f3ef945b1" 2025-04-21T21:42:59.8613080Z }, 2025-04-21T21:42:59.8613259Z { 2025-04-21T21:42:59.8613570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8614113Z "size": 248991124, 2025-04-21T21:42:59.8614510Z "digest": "sha256:c68e78d48b95f6272003f9de9b80c623c4b361ed6e860801cbe1bd2db4d97263" 2025-04-21T21:42:59.8614948Z }, 2025-04-21T21:42:59.8615133Z { 2025-04-21T21:42:59.8615436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8615929Z "size": 703, 2025-04-21T21:42:59.8616309Z "digest": "sha256:b14b320c459cd13b3ef4a06575f1b29513d8b0e6c1672ea68ae5b34a977cd975" 2025-04-21T21:42:59.8616744Z }, 2025-04-21T21:42:59.8616921Z { 2025-04-21T21:42:59.8617305Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8617854Z "size": 1262, 2025-04-21T21:42:59.8618234Z "digest": "sha256:0d929bfaefee4bb307c42773c6f04675af8b94398a51b1c8575447099bbf8607" 2025-04-21T21:42:59.8618673Z }, 2025-04-21T21:42:59.8618856Z { 2025-04-21T21:42:59.8619158Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8619584Z "size": 484, 2025-04-21T21:42:59.8620258Z "digest": "sha256:a5a8ea5a3eabb6655fad5c54e28c86000dca6ac1c3a5a485c718f87b60e28711" 2025-04-21T21:42:59.8620722Z }, 2025-04-21T21:42:59.8620904Z { 2025-04-21T21:42:59.8621217Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8621612Z "size": 91731730, 2025-04-21T21:42:59.8622012Z "digest": "sha256:281f13ef6929e0c7c8259f29b3dc71103ec70018f2711ec9f8a8f04093099171" 2025-04-21T21:42:59.8622448Z }, 2025-04-21T21:42:59.8622624Z { 2025-04-21T21:42:59.8622935Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8623328Z "size": 3767, 2025-04-21T21:42:59.8623702Z "digest": "sha256:a8695f1c94c34953b85e2090d713494e7fc84adc148495937b7a7c3deed10275" 2025-04-21T21:42:59.8624132Z }, 2025-04-21T21:42:59.8624312Z { 2025-04-21T21:42:59.8624564Z + exit 0 2025-04-21T21:42:59.8624879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8625271Z "size": 1868, 2025-04-21T21:42:59.8625672Z "digest": "sha256:c72f323be8018695bb0b260acf986dd615ac7ce067ad92b0eb1ac2e72925a39a" 2025-04-21T21:42:59.8626117Z }, 2025-04-21T21:42:59.8626299Z { 2025-04-21T21:42:59.8626612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8627005Z "size": 701, 2025-04-21T21:42:59.8627395Z "digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03" 2025-04-21T21:42:59.8627841Z }, 2025-04-21T21:42:59.8628018Z { 2025-04-21T21:42:59.8628332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8628727Z "size": 477, 2025-04-21T21:42:59.8629119Z "digest": "sha256:a9bdca78eb50cb709cb6b39f9f2bcd879bc2bb990b944eca5bc0b41d8ac75114" 2025-04-21T21:42:59.8629575Z }, 2025-04-21T21:42:59.8629766Z { 2025-04-21T21:42:59.8630118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8630519Z "size": 3018885224, 2025-04-21T21:42:59.8630917Z "digest": "sha256:87b13320574046b541e9fdaa0186880e7082d07d823639a45f77e9da1e325fda" 2025-04-21T21:42:59.8631347Z }, 2025-04-21T21:42:59.8631522Z { 2025-04-21T21:42:59.8631836Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8632227Z "size": 380, 2025-04-21T21:42:59.8632609Z "digest": "sha256:1a1f3a6b6e2805950beb3918166ea519e8237d7b6493d050bce0ecb65b3750e6" 2025-04-21T21:42:59.8633044Z }, 2025-04-21T21:42:59.8633224Z { 2025-04-21T21:42:59.8633528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8633923Z "size": 12918, 2025-04-21T21:42:59.8634307Z "digest": "sha256:1e726e5b69dff0c7b1285406bf76144f96e624de579ac59e252da98586616399" 2025-04-21T21:42:59.8634742Z }, 2025-04-21T21:42:59.8634927Z { 2025-04-21T21:42:59.8635242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8635631Z "size": 791, 2025-04-21T21:42:59.8647097Z "digest": "sha256:e55ee61cc4aaf6da32e72efeef162eda299bb6a632e8d8dcb487521c151204a0" 2025-04-21T21:42:59.8647586Z }, 2025-04-21T21:42:59.8647767Z { 2025-04-21T21:42:59.8648089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8648493Z "size": 106, 2025-04-21T21:42:59.8648886Z "digest": "sha256:d2c9e5f6d068cc6a329295a862e67a4110accba3b0c35efaab96121dc83e2c79" 2025-04-21T21:42:59.8649426Z }, 2025-04-21T21:42:59.8649612Z { 2025-04-21T21:42:59.8649925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8650327Z "size": 503, 2025-04-21T21:42:59.8650720Z "digest": "sha256:a9e23e3c0a187a40d8af9500ce12468894283cc71e0ba2fb1fcb6624dc40ddbf" 2025-04-21T21:42:59.8651169Z }, 2025-04-21T21:42:59.8651350Z { 2025-04-21T21:42:59.8651666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8652068Z "size": 121477387, 2025-04-21T21:42:59.8652469Z "digest": "sha256:956077b157237c043dfffd54a6aee7e4f7d3e03584e329d427ab36bcf285b595" 2025-04-21T21:42:59.8652921Z }, 2025-04-21T21:42:59.8653110Z { 2025-04-21T21:42:59.8653421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8653823Z "size": 109, 2025-04-21T21:42:59.8654212Z "digest": "sha256:b840d58dab14ffab9220e7e5dba7751d9809c049d77a61b93924e4947527bb65" 2025-04-21T21:42:59.8654660Z }, 2025-04-21T21:42:59.8654848Z { 2025-04-21T21:42:59.8655168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8655566Z "size": 1472, 2025-04-21T21:42:59.8655959Z "digest": "sha256:135c1f91f3df5257432cc2d37486ab3cf2ff585d3a2ec4ac2304cb0d25e5fac1" 2025-04-21T21:42:59.8656406Z }, 2025-04-21T21:42:59.8656596Z { 2025-04-21T21:42:59.8656907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8657312Z "size": 429635236, 2025-04-21T21:42:59.8657807Z "digest": "sha256:2fba85b4232347228dbfdef797160ace15ad71face213ee558c1180161dfe7c6" 2025-04-21T21:42:59.8658249Z }, 2025-04-21T21:42:59.8658438Z { 2025-04-21T21:42:59.8658755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8659148Z "size": 162, 2025-04-21T21:42:59.8659537Z "digest": "sha256:7e377528c596bece391a67c4dfc9bab633557a7bd77e7a650595f6f4ef162b1a" 2025-04-21T21:42:59.8659994Z }, 2025-04-21T21:42:59.8660174Z { 2025-04-21T21:42:59.8660490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8660884Z "size": 801, 2025-04-21T21:42:59.8661280Z "digest": "sha256:7c1fe59ec87a4d7e8d2d9c14b3aa7254a4dbeb2566473fceaa3d6ef086a29e5a" 2025-04-21T21:42:59.8661733Z }, 2025-04-21T21:42:59.8661920Z { 2025-04-21T21:42:59.8662233Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8662624Z "size": 35860969, 2025-04-21T21:42:59.8663023Z "digest": "sha256:bb83f375ed7d52c4af28b418a77af53bdb24511aad990c9e8f017c3b6145d401" 2025-04-21T21:42:59.8663473Z }, 2025-04-21T21:42:59.8663653Z { 2025-04-21T21:42:59.8663970Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8664364Z "size": 104, 2025-04-21T21:42:59.8664743Z "digest": "sha256:d1ee3dcc43c51d791d948668680e601c3394068ce014a7395b2565facd8e1554" 2025-04-21T21:42:59.8665183Z }, 2025-04-21T21:42:59.8665374Z { 2025-04-21T21:42:59.8665683Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8666078Z "size": 424, 2025-04-21T21:42:59.8666460Z "digest": "sha256:54687ddf5aab346564b45b9340d55679e17969e1a9290a068a71acd75e445a68" 2025-04-21T21:42:59.8666894Z }, 2025-04-21T21:42:59.8667073Z { 2025-04-21T21:42:59.8667384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8667780Z "size": 20262168, 2025-04-21T21:42:59.8668170Z "digest": "sha256:6b673e167d0fd859f7f81837936c87feb0f3540e9d83ccb685e01c2d2b1a105c" 2025-04-21T21:42:59.8668606Z }, 2025-04-21T21:42:59.8668783Z { 2025-04-21T21:42:59.8669188Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8669585Z "size": 643, 2025-04-21T21:42:59.8669965Z "digest": "sha256:a4f475d6d4d727b2776f8de8fb347830ba056f3485b53102d7502bf63a1c08b4" 2025-04-21T21:42:59.8670394Z }, 2025-04-21T21:42:59.8670574Z { 2025-04-21T21:42:59.8670884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8671348Z "size": 701, 2025-04-21T21:42:59.8671731Z "digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03" 2025-04-21T21:42:59.8672171Z }, 2025-04-21T21:42:59.8672355Z { 2025-04-21T21:42:59.8672660Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8673049Z "size": 141, 2025-04-21T21:42:59.8673427Z "digest": "sha256:48e60e8ea738df82c6a874ed4b7e587451261d7675d166a68e4b882ce5a932e7" 2025-04-21T21:42:59.8673859Z }, 2025-04-21T21:42:59.8674040Z { 2025-04-21T21:42:59.8674352Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8674748Z "size": 134, 2025-04-21T21:42:59.8675130Z "digest": "sha256:d52c79949221ec29bacb5c73e1630c76a77c93a6dcd0a36570ef858cb7ff2491" 2025-04-21T21:42:59.8675567Z }, 2025-04-21T21:42:59.8675745Z { 2025-04-21T21:42:59.8676057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8676459Z "size": 5221737836, 2025-04-21T21:42:59.8676842Z "digest": "sha256:846d8aa2652360437687998e7786934ac4a2680471da233060f0e42acca36db8" 2025-04-21T21:42:59.8677270Z }, 2025-04-21T21:42:59.8677453Z { 2025-04-21T21:42:59.8677759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8678149Z "size": 194, 2025-04-21T21:42:59.8678531Z "digest": "sha256:595e9954171617ada6037b7869515bf0e2d1dff54b868e3ffba1f652ef8c56f5" 2025-04-21T21:42:59.8678962Z }, 2025-04-21T21:42:59.8679137Z { 2025-04-21T21:42:59.8679446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8679838Z "size": 563, 2025-04-21T21:42:59.8680221Z "digest": "sha256:013dfd09e68f59d76a79ddcc8b47d99f10ad62529abcd1725885de25818509ae" 2025-04-21T21:42:59.8680703Z }, 2025-04-21T21:42:59.8680898Z { 2025-04-21T21:42:59.8681202Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8681602Z "size": 43164291, 2025-04-21T21:42:59.8682003Z "digest": "sha256:4402edece585e892e3dcd04507c4e9315cd4537bacd16ed748aebfec5e8af5e8" 2025-04-21T21:42:59.8682449Z }, 2025-04-21T21:42:59.8682625Z { 2025-04-21T21:42:59.8682936Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8683330Z "size": 106, 2025-04-21T21:42:59.8683709Z "digest": "sha256:82aa68866e535166ba9028ae8d5fe7bf3d13ea4c69a43469aaefdffccc00b8a9" 2025-04-21T21:42:59.8684150Z }, 2025-04-21T21:42:59.8684325Z { 2025-04-21T21:42:59.8684628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8685025Z "size": 307403395, 2025-04-21T21:42:59.8685424Z "digest": "sha256:e624d917de213d9b06921538cf6d41f55585fc181f15c727920ca878e0d432c1" 2025-04-21T21:42:59.8685850Z }, 2025-04-21T21:42:59.8686034Z { 2025-04-21T21:42:59.8686345Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8686735Z "size": 5761507135, 2025-04-21T21:42:59.8687143Z "digest": "sha256:79d2c9f506eccd71c934a8639db0b5fb46a4c03b1e4c68535c972001a0b3a1f6" 2025-04-21T21:42:59.8687581Z }, 2025-04-21T21:42:59.8687769Z { 2025-04-21T21:42:59.8688074Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8688467Z "size": 127, 2025-04-21T21:42:59.8688854Z "digest": "sha256:3b25d713acb3675cc21cfaf3e73036b34d21dd039d0339ebd5874e41f29d4c73" 2025-04-21T21:42:59.8689291Z }, 2025-04-21T21:42:59.8689472Z { 2025-04-21T21:42:59.8689784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8690220Z "size": 1012, 2025-04-21T21:42:59.8690750Z "digest": "sha256:c0ad696e18d34210a240d661b9a6994f1fcf548fb2eeaeccef011fd9fbb01d79" 2025-04-21T21:42:59.8691201Z }, 2025-04-21T21:42:59.8691383Z { 2025-04-21T21:42:59.8691702Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8692101Z "size": 701, 2025-04-21T21:42:59.8692496Z "digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03" 2025-04-21T21:42:59.8693015Z }, 2025-04-21T21:42:59.8693209Z { 2025-04-21T21:42:59.8693523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8693915Z "size": 135, 2025-04-21T21:42:59.8694524Z "digest": "sha256:767566b693177188e1a3b60bd320a787e121ffd7dd9f0cd79f75f94f17d2197b" 2025-04-21T21:42:59.8695053Z }, 2025-04-21T21:42:59.8695233Z { 2025-04-21T21:42:59.8695550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8695943Z "size": 32, 2025-04-21T21:42:59.8696327Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-04-21T21:42:59.8696780Z }, 2025-04-21T21:42:59.8696964Z { 2025-04-21T21:42:59.8697270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8697759Z "size": 159, 2025-04-21T21:42:59.8698147Z "digest": "sha256:d83ab53d6ab47f0836a1d834f483ea68f6bb74717f9b5f435958be607a1e25de" 2025-04-21T21:42:59.8698594Z }, 2025-04-21T21:42:59.8698772Z { 2025-04-21T21:42:59.8699085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8699482Z "size": 1898, 2025-04-21T21:42:59.8699909Z "digest": "sha256:e9ae24bb3b5b9eaace2576d0bc4ce0ccdea8f446743644ae2f5d99fa4d2704ed" 2025-04-21T21:42:59.8700393Z }, 2025-04-21T21:42:59.8700578Z { 2025-04-21T21:42:59.8700884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8701286Z "size": 237571532, 2025-04-21T21:42:59.8701687Z "digest": "sha256:8c11aa345fff3b36368b1fc7fc8268149038691d5e7a11054e8c029d0b9a6675" 2025-04-21T21:42:59.8702118Z }, 2025-04-21T21:42:59.8702308Z { 2025-04-21T21:42:59.8702621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8703012Z "size": 162, 2025-04-21T21:42:59.8703412Z "digest": "sha256:cc9ed3f492faeb8df7ef7ca746a2c85d1189181e3a2516a19c0126bd224ad5d9" 2025-04-21T21:42:59.8703868Z }, 2025-04-21T21:42:59.8704051Z { 2025-04-21T21:42:59.8704359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8704759Z "size": 7944, 2025-04-21T21:42:59.8706597Z "digest": "sha256:6133da88e6692508c12cae6643582782b9a424aa73892cf6e4c0befebe516269" 2025-04-21T21:42:59.8707028Z }, 2025-04-21T21:42:59.8707217Z { 2025-04-21T21:42:59.8707537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8707933Z "size": 8071, 2025-04-21T21:42:59.8708314Z "digest": "sha256:f55162ff6427c5c71e1737cb77e4c9913c679981ec16b241d92084fe621b74c9" 2025-04-21T21:42:59.8708751Z }, 2025-04-21T21:42:59.8708931Z { 2025-04-21T21:42:59.8709258Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8709658Z "size": 304, 2025-04-21T21:42:59.8710047Z "digest": "sha256:f2fb53f519b30fceaf1192ac5221e9c1130032af58e6dd7312a7d82f6be9e66e" 2025-04-21T21:42:59.8710490Z }, 2025-04-21T21:42:59.8710681Z { 2025-04-21T21:42:59.8710994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8711388Z "size": 7631978, 2025-04-21T21:42:59.8711782Z "digest": "sha256:d93b86c61192d758035dbbef6102f4c29339a3bc439c06bde06e3e7efb73c071" 2025-04-21T21:42:59.8712227Z }, 2025-04-21T21:42:59.8712418Z { 2025-04-21T21:42:59.8712727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8713126Z "size": 108, 2025-04-21T21:42:59.8713526Z "digest": "sha256:2d75b00aef3ef33cd0a6aca03e1b81df01b8d5c874850e5ee0a580fdaeffafda" 2025-04-21T21:42:59.8713974Z }, 2025-04-21T21:42:59.8714160Z { 2025-04-21T21:42:59.8714635Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8715030Z "size": 54145661, 2025-04-21T21:42:59.8715424Z "digest": "sha256:165cb697180f8d5f9e2677a69ad5431ab294be16a39bc423f0d8593513c041f0" 2025-04-21T21:42:59.8715863Z }, 2025-04-21T21:42:59.8716045Z { 2025-04-21T21:42:59.8716358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8716885Z "size": 495, 2025-04-21T21:42:59.8717259Z "digest": "sha256:952482b7353e3317938e0c2a2c86c92dc2301bec5f1683ae1fc4f72ef2e8ac4b" 2025-04-21T21:42:59.8717698Z }, 2025-04-21T21:42:59.8717887Z { 2025-04-21T21:42:59.8718203Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8718595Z "size": 1179582084, 2025-04-21T21:42:59.8718993Z "digest": "sha256:f4fb942e12736de733008a26bdd886b81205f8a80595e32db7584540ac193776" 2025-04-21T21:42:59.8719429Z }, 2025-04-21T21:42:59.8719609Z { 2025-04-21T21:42:59.8719927Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8720379Z "size": 106, 2025-04-21T21:42:59.8720767Z "digest": "sha256:22d7ad62e40c8f69aaedce849745730ac752c1b564e4a7d39bbae1f12e7f03c6" 2025-04-21T21:42:59.8721215Z }, 2025-04-21T21:42:59.8721397Z { 2025-04-21T21:42:59.8721704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8722104Z "size": 612, 2025-04-21T21:42:59.8722482Z "digest": "sha256:340ddaf5579e22644b795f56a17c3697533215e26d5761d5cca8bae92205e078" 2025-04-21T21:42:59.8722917Z }, 2025-04-21T21:42:59.8723095Z { 2025-04-21T21:42:59.8723408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8723807Z "size": 317359363, 2025-04-21T21:42:59.8724203Z "digest": "sha256:abb8b598f42085f2a06e7d8095a6ddb1407e5611c878d0ec1c605c6b62565e39" 2025-04-21T21:42:59.8724648Z }, 2025-04-21T21:42:59.8724835Z { 2025-04-21T21:42:59.8725143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8725538Z "size": 111, 2025-04-21T21:42:59.8725930Z "digest": "sha256:410b8ad7da03358202afe62f6f95a90ffc13b6c289c59f77d52078e0cfb3b427" 2025-04-21T21:42:59.8726369Z }, 2025-04-21T21:42:59.8726557Z { 2025-04-21T21:42:59.8726871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8727267Z "size": 475, 2025-04-21T21:42:59.8727657Z "digest": "sha256:79d2e9034f2b1b76f47e712bb3ab767c7b0a8aa2a9bdfc69526ea55345939d3c" 2025-04-21T21:42:59.8728102Z }, 2025-04-21T21:42:59.8728291Z { 2025-04-21T21:42:59.8728600Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8728997Z "size": 180, 2025-04-21T21:42:59.8729381Z "digest": "sha256:c2128146ebc3a8a388e3ae687d3a8509470f980106006e2c07f01878a39cb1e2" 2025-04-21T21:42:59.8729811Z }, 2025-04-21T21:42:59.8730002Z { 2025-04-21T21:42:59.8730366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8730758Z "size": 529690496, 2025-04-21T21:42:59.8731172Z "digest": "sha256:4b1e8ee3de3882b3d9b74a17aa0895307d6b7f5aa4bde3fe3338dc7599d38fcf" 2025-04-21T21:42:59.8731621Z }, 2025-04-21T21:42:59.8731800Z { 2025-04-21T21:42:59.8732114Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8732511Z "size": 183, 2025-04-21T21:42:59.8732906Z "digest": "sha256:02d23db28b3d929d4f9be01edc8371656c3215d2b4c08df0f24fb897e8b9c4eb" 2025-04-21T21:42:59.8733347Z }, 2025-04-21T21:42:59.8733532Z { 2025-04-21T21:42:59.8733846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8734237Z "size": 529, 2025-04-21T21:42:59.8734632Z "digest": "sha256:24d4ae1408767f7ceda609b9b1eef8ff3b3f7ee876a8a92ba1de0800db9328d1" 2025-04-21T21:42:59.8735090Z }, 2025-04-21T21:42:59.8735269Z { 2025-04-21T21:42:59.8735580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8735976Z "size": 26134, 2025-04-21T21:42:59.8736451Z "digest": "sha256:2604f3be2ad176777da15e3755dea31789b0e584891d00680961ee3095a8b9e0" 2025-04-21T21:42:59.8736890Z }, 2025-04-21T21:42:59.8737073Z { 2025-04-21T21:42:59.8737380Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8737867Z "size": 106, 2025-04-21T21:42:59.8738248Z "digest": "sha256:ba564700c169e37253f9d67c6224e478ab8ee66a1906b540eaa06a77e3c5b75a" 2025-04-21T21:42:59.8738765Z }, 2025-04-21T21:42:59.8738946Z { 2025-04-21T21:42:59.8739264Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8739658Z "size": 32, 2025-04-21T21:42:59.8740040Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-04-21T21:42:59.8740489Z }, 2025-04-21T21:42:59.8740674Z { 2025-04-21T21:42:59.8740981Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8741378Z "size": 32, 2025-04-21T21:42:59.8741769Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-04-21T21:42:59.8742217Z }, 2025-04-21T21:42:59.8742403Z { 2025-04-21T21:42:59.8742717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8743109Z "size": 32, 2025-04-21T21:42:59.8743497Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-04-21T21:42:59.8743950Z }, 2025-04-21T21:42:59.8744135Z { 2025-04-21T21:42:59.8744443Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-04-21T21:42:59.8744839Z "size": 32, 2025-04-21T21:42:59.8745228Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-04-21T21:42:59.8745668Z } 2025-04-21T21:42:59.8745856Z ] 2025-04-21T21:42:59.8746039Z } 2025-04-21T21:42:59.8782699Z ##[group]Run set -eux 2025-04-21T21:42:59.8782964Z set -eux 2025-04-21T21:42:59.8783792Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-04-21T21:42:59.8793996Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:42:59.8794714Z env: 2025-04-21T21:42:59.8794930Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:42:59.8795185Z ##[endgroup] 2025-04-21T21:42:59.8826950Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-04-21T21:42:59.8827949Z + jq --raw-output .SecretString 2025-04-21T21:42:59.8828725Z + jq -r .docker_hub_readonly_token 2025-04-21T21:42:59.8831069Z + docker login --username pytorchbot --password-stdin 2025-04-21T21:43:00.4534390Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-04-21T21:43:00.4535000Z Configure a credential helper to remove this warning. See 2025-04-21T21:43:00.4535577Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-04-21T21:43:00.4535968Z 2025-04-21T21:43:00.4536068Z Login Succeeded 2025-04-21T21:43:00.4635165Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2025-04-21T21:43:00.4635537Z tag=${ECR_DOCKER_IMAGE##*/} 2025-04-21T21:43:00.4635918Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-04-21T21:43:00.4644847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:43:00.4645211Z env: 2025-04-21T21:43:00.4645439Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:43:00.4646270Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:00.4647125Z ##[endgroup] 2025-04-21T21:43:00.4677728Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks-f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:00.4734373Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-04-21T21:43:00.4734803Z with: 2025-04-21T21:43:00.4735576Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:00.4736501Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:43:00.4736870Z env: 2025-04-21T21:43:00.4737079Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:43:00.4737491Z ##[endgroup] 2025-04-21T21:43:00.4761546Z ##[group]Run set -x 2025-04-21T21:43:00.4761812Z set -x 2025-04-21T21:43:00.4762036Z set +e 2025-04-21T21:43:00.4762251Z  2025-04-21T21:43:00.4762452Z login() { 2025-04-21T21:43:00.4762908Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-04-21T21:43:00.4763405Z } 2025-04-21T21:43:00.4763606Z  2025-04-21T21:43:00.4763840Z retry () { 2025-04-21T21:43:00.4764104Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-04-21T21:43:00.4764411Z } 2025-04-21T21:43:00.4764620Z  2025-04-21T21:43:00.4764851Z retry login "${DOCKER_REGISTRY}" 2025-04-21T21:43:00.4765139Z  2025-04-21T21:43:00.4765601Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-04-21T21:43:00.4766232Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-04-21T21:43:00.4766593Z  2025-04-21T21:43:00.4766797Z set -e 2025-04-21T21:43:00.4767132Z # ignore output since only exit code is used for conditional 2025-04-21T21:43:00.4767605Z # only pull docker image if it's not available locally 2025-04-21T21:43:00.4768134Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-04-21T21:43:00.4768628Z  retry docker pull "${DOCKER_IMAGE}" 2025-04-21T21:43:00.4768937Z fi 2025-04-21T21:43:00.4777794Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:43:00.4778162Z env: 2025-04-21T21:43:00.4778378Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:43:00.4779203Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:00.4780156Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:43:00.4780552Z ##[endgroup] 2025-04-21T21:43:00.4808346Z + set +e 2025-04-21T21:43:00.4808667Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:43:00.4809072Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:43:00.4812089Z + aws ecr get-login-password --region us-east-1 2025-04-21T21:43:00.4813847Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-04-21T21:43:01.0054679Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-04-21T21:43:01.0056073Z Configure a credential helper to remove this warning. See 2025-04-21T21:43:01.0057385Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-04-21T21:43:01.0058444Z 2025-04-21T21:43:01.0058636Z Login Succeeded 2025-04-21T21:43:01.0088191Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:01.0089198Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-04-21T21:43:01.3992454Z + IMAGE_SIZE=20670.273096084595 2025-04-21T21:43:01.3992839Z + echo 'Compressed size of image in MB: 20670.273096084595' 2025-04-21T21:43:01.3993202Z + set -e 2025-04-21T21:43:01.3994617Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:01.3995638Z Compressed size of image in MB: 20670.273096084595 2025-04-21T21:43:01.4146212Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:01.4147648Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:43:01.6458777Z f583c1656c56372ad0e1c57f3463a017fbf03138: Pulling from pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks 2025-04-21T21:43:01.6465164Z 86e5016c2693: Pulling fs layer 2025-04-21T21:43:01.6465684Z 49e139a3d6c2: Pulling fs layer 2025-04-21T21:43:01.6466114Z a14844c8c51f: Pulling fs layer 2025-04-21T21:43:01.6466527Z 18fb524087fb: Pulling fs layer 2025-04-21T21:43:01.6466932Z efd686a7b2c8: Pulling fs layer 2025-04-21T21:43:01.6467359Z 52c648e21334: Pulling fs layer 2025-04-21T21:43:01.6467703Z 56e384e4e5aa: Pulling fs layer 2025-04-21T21:43:01.6467971Z fb71b792ec6c: Pulling fs layer 2025-04-21T21:43:01.6468274Z 5509576f2693: Pulling fs layer 2025-04-21T21:43:01.6468545Z 1e6c6f2d2459: Pulling fs layer 2025-04-21T21:43:01.6468806Z a1fe8922734d: Pulling fs layer 2025-04-21T21:43:01.6469070Z 84e5433913c6: Pulling fs layer 2025-04-21T21:43:01.6469334Z efd686a7b2c8: Waiting 2025-04-21T21:43:01.6469601Z c68e78d48b95: Pulling fs layer 2025-04-21T21:43:01.6469927Z b14b320c459c: Pulling fs layer 2025-04-21T21:43:01.6470214Z 0d929bfaefee: Pulling fs layer 2025-04-21T21:43:01.6470525Z a5a8ea5a3eab: Pulling fs layer 2025-04-21T21:43:01.6470807Z 281f13ef6929: Pulling fs layer 2025-04-21T21:43:01.6471081Z a8695f1c94c3: Pulling fs layer 2025-04-21T21:43:01.6471349Z c72f323be801: Pulling fs layer 2025-04-21T21:43:01.6471613Z 4afa91cd0662: Pulling fs layer 2025-04-21T21:43:01.6471889Z a9bdca78eb50: Pulling fs layer 2025-04-21T21:43:01.6472150Z 18fb524087fb: Waiting 2025-04-21T21:43:01.6472396Z 87b133205740: Pulling fs layer 2025-04-21T21:43:01.6472647Z fb71b792ec6c: Waiting 2025-04-21T21:43:01.6472892Z 1a1f3a6b6e28: Pulling fs layer 2025-04-21T21:43:01.6473154Z 56e384e4e5aa: Waiting 2025-04-21T21:43:01.6473433Z 1e726e5b69df: Pulling fs layer 2025-04-21T21:43:01.6473801Z e55ee61cc4aa: Pulling fs layer 2025-04-21T21:43:01.6474061Z 52c648e21334: Waiting 2025-04-21T21:43:01.6474278Z 5509576f2693: Waiting 2025-04-21T21:43:01.6474503Z a1fe8922734d: Waiting 2025-04-21T21:43:01.6474745Z d2c9e5f6d068: Pulling fs layer 2025-04-21T21:43:01.6475017Z a9e23e3c0a18: Pulling fs layer 2025-04-21T21:43:01.6475284Z 956077b15723: Pulling fs layer 2025-04-21T21:43:01.6485387Z b840d58dab14: Pulling fs layer 2025-04-21T21:43:01.6485757Z b14b320c459c: Waiting 2025-04-21T21:43:01.6486068Z 84e5433913c6: Waiting 2025-04-21T21:43:01.6486381Z a5a8ea5a3eab: Waiting 2025-04-21T21:43:01.6486769Z 1e6c6f2d2459: Waiting 2025-04-21T21:43:01.6487108Z 135c1f91f3df: Pulling fs layer 2025-04-21T21:43:01.6487521Z a8695f1c94c3: Waiting 2025-04-21T21:43:01.6487901Z 2fba85b42323: Pulling fs layer 2025-04-21T21:43:01.6488282Z 7e377528c596: Pulling fs layer 2025-04-21T21:43:01.6488658Z 7c1fe59ec87a: Pulling fs layer 2025-04-21T21:43:01.6489055Z bb83f375ed7d: Pulling fs layer 2025-04-21T21:43:01.6489443Z d1ee3dcc43c5: Pulling fs layer 2025-04-21T21:43:01.6489831Z 54687ddf5aab: Pulling fs layer 2025-04-21T21:43:01.6490203Z 956077b15723: Waiting 2025-04-21T21:43:01.6490547Z 6b673e167d0f: Pulling fs layer 2025-04-21T21:43:01.6490913Z b840d58dab14: Waiting 2025-04-21T21:43:01.6491270Z a4f475d6d4d7: Pulling fs layer 2025-04-21T21:43:01.6491645Z 2fba85b42323: Waiting 2025-04-21T21:43:01.6491960Z a9bdca78eb50: Waiting 2025-04-21T21:43:01.6492284Z bb83f375ed7d: Waiting 2025-04-21T21:43:01.6492629Z 48e60e8ea738: Pulling fs layer 2025-04-21T21:43:01.6492931Z 7e377528c596: Waiting 2025-04-21T21:43:01.6493202Z a4f475d6d4d7: Waiting 2025-04-21T21:43:01.6493443Z d52c79949221: Pulling fs layer 2025-04-21T21:43:01.6493974Z 846d8aa26523: Pulling fs layer 2025-04-21T21:43:01.6494616Z 595e99541716: Pulling fs layer 2025-04-21T21:43:01.6494885Z 6b673e167d0f: Waiting 2025-04-21T21:43:01.6495109Z 54687ddf5aab: Waiting 2025-04-21T21:43:01.6495352Z 013dfd09e68f: Pulling fs layer 2025-04-21T21:43:01.6495679Z 4402edece585: Pulling fs layer 2025-04-21T21:43:01.6495971Z 82aa68866e53: Pulling fs layer 2025-04-21T21:43:01.6496232Z e624d917de21: Pulling fs layer 2025-04-21T21:43:01.6496495Z 79d2c9f506ec: Pulling fs layer 2025-04-21T21:43:01.6496751Z 3b25d713acb3: Pulling fs layer 2025-04-21T21:43:01.6497321Z c0ad696e18d3: Pulling fs layer 2025-04-21T21:43:01.6497720Z a9e23e3c0a18: Waiting 2025-04-21T21:43:01.6497951Z 281f13ef6929: Waiting 2025-04-21T21:43:01.6498171Z 48e60e8ea738: Waiting 2025-04-21T21:43:01.6498402Z 767566b69317: Pulling fs layer 2025-04-21T21:43:01.6498646Z 1e726e5b69df: Waiting 2025-04-21T21:43:01.6498876Z 4f4fb700ef54: Pulling fs layer 2025-04-21T21:43:01.6499145Z d83ab53d6ab4: Pulling fs layer 2025-04-21T21:43:01.6499408Z 135c1f91f3df: Waiting 2025-04-21T21:43:01.6499640Z e9ae24bb3b5b: Pulling fs layer 2025-04-21T21:43:01.6499909Z 8c11aa345fff: Pulling fs layer 2025-04-21T21:43:01.6500182Z cc9ed3f492fa: Pulling fs layer 2025-04-21T21:43:01.6500432Z 87b133205740: Waiting 2025-04-21T21:43:01.6500663Z d1ee3dcc43c5: Waiting 2025-04-21T21:43:01.6500901Z 6133da88e669: Pulling fs layer 2025-04-21T21:43:01.6501158Z f55162ff6427: Pulling fs layer 2025-04-21T21:43:01.6501412Z 846d8aa26523: Waiting 2025-04-21T21:43:01.6501651Z f2fb53f519b3: Pulling fs layer 2025-04-21T21:43:01.6501917Z d93b86c61192: Pulling fs layer 2025-04-21T21:43:01.6502174Z d83ab53d6ab4: Waiting 2025-04-21T21:43:01.6502410Z cc9ed3f492fa: Waiting 2025-04-21T21:43:01.6502638Z e9ae24bb3b5b: Waiting 2025-04-21T21:43:01.6502870Z 2d75b00aef3e: Pulling fs layer 2025-04-21T21:43:01.6503118Z 6133da88e669: Waiting 2025-04-21T21:43:01.6503353Z 165cb697180f: Pulling fs layer 2025-04-21T21:43:01.6503607Z 952482b7353e: Pulling fs layer 2025-04-21T21:43:01.6503857Z c0ad696e18d3: Waiting 2025-04-21T21:43:01.6504101Z f4fb942e1273: Pulling fs layer 2025-04-21T21:43:01.6504344Z f55162ff6427: Waiting 2025-04-21T21:43:01.6504636Z 22d7ad62e40c: Pulling fs layer 2025-04-21T21:43:01.6504907Z 340ddaf5579e: Pulling fs layer 2025-04-21T21:43:01.6505198Z abb8b598f420: Pulling fs layer 2025-04-21T21:43:01.6505481Z 410b8ad7da03: Pulling fs layer 2025-04-21T21:43:01.6505730Z 013dfd09e68f: Waiting 2025-04-21T21:43:01.6505945Z 82aa68866e53: Waiting 2025-04-21T21:43:01.6506169Z d2c9e5f6d068: Waiting 2025-04-21T21:43:01.6506406Z 79d2e9034f2b: Pulling fs layer 2025-04-21T21:43:01.6506666Z 1a1f3a6b6e28: Waiting 2025-04-21T21:43:01.6506901Z c2128146ebc3: Pulling fs layer 2025-04-21T21:43:01.6507169Z 4b1e8ee3de38: Pulling fs layer 2025-04-21T21:43:01.6507432Z 02d23db28b3d: Pulling fs layer 2025-04-21T21:43:01.6507680Z 767566b69317: Waiting 2025-04-21T21:43:01.6507896Z 2d75b00aef3e: Waiting 2025-04-21T21:43:01.6508130Z 24d4ae140876: Pulling fs layer 2025-04-21T21:43:01.6508386Z 2604f3be2ad1: Pulling fs layer 2025-04-21T21:43:01.6508656Z ba564700c169: Pulling fs layer 2025-04-21T21:43:01.6508911Z 22d7ad62e40c: Waiting 2025-04-21T21:43:01.6509126Z 595e99541716: Waiting 2025-04-21T21:43:01.6509348Z 4f4fb700ef54: Waiting 2025-04-21T21:43:01.6509568Z f4fb942e1273: Waiting 2025-04-21T21:43:01.6509783Z 7c1fe59ec87a: Waiting 2025-04-21T21:43:01.6510009Z 24d4ae140876: Waiting 2025-04-21T21:43:01.6510230Z 2604f3be2ad1: Waiting 2025-04-21T21:43:01.6510445Z ba564700c169: Waiting 2025-04-21T21:43:01.6510666Z c68e78d48b95: Waiting 2025-04-21T21:43:01.6510889Z 8c11aa345fff: Waiting 2025-04-21T21:43:01.6511117Z 340ddaf5579e: Waiting 2025-04-21T21:43:01.6511341Z 410b8ad7da03: Waiting 2025-04-21T21:43:01.6511569Z 02d23db28b3d: Waiting 2025-04-21T21:43:01.6511823Z 79d2e9034f2b: Waiting 2025-04-21T21:43:01.6512080Z 165cb697180f: Waiting 2025-04-21T21:43:01.6512302Z d52c79949221: Waiting 2025-04-21T21:43:01.6512516Z e624d917de21: Waiting 2025-04-21T21:43:01.6512740Z 79d2c9f506ec: Waiting 2025-04-21T21:43:01.6512967Z 3b25d713acb3: Waiting 2025-04-21T21:43:01.6513345Z 952482b7353e: Waiting 2025-04-21T21:43:01.6513579Z 4b1e8ee3de38: Waiting 2025-04-21T21:43:01.6513806Z f2fb53f519b3: Waiting 2025-04-21T21:43:01.8018038Z 49e139a3d6c2: Verifying Checksum 2025-04-21T21:43:01.8018352Z 49e139a3d6c2: Download complete 2025-04-21T21:43:01.8933671Z 18fb524087fb: Verifying Checksum 2025-04-21T21:43:01.8933981Z 18fb524087fb: Download complete 2025-04-21T21:43:01.9821650Z efd686a7b2c8: Verifying Checksum 2025-04-21T21:43:01.9821965Z efd686a7b2c8: Download complete 2025-04-21T21:43:02.0113657Z 86e5016c2693: Verifying Checksum 2025-04-21T21:43:02.0114159Z 86e5016c2693: Download complete 2025-04-21T21:43:02.1022122Z 56e384e4e5aa: Verifying Checksum 2025-04-21T21:43:02.1022585Z 56e384e4e5aa: Download complete 2025-04-21T21:43:02.1991420Z fb71b792ec6c: Download complete 2025-04-21T21:43:02.2762217Z 5509576f2693: Download complete 2025-04-21T21:43:02.2778135Z a14844c8c51f: Verifying Checksum 2025-04-21T21:43:02.2778431Z a14844c8c51f: Download complete 2025-04-21T21:43:02.3894658Z a1fe8922734d: Verifying Checksum 2025-04-21T21:43:02.3895115Z a1fe8922734d: Download complete 2025-04-21T21:43:02.4822987Z 84e5433913c6: Verifying Checksum 2025-04-21T21:43:02.4823296Z 84e5433913c6: Download complete 2025-04-21T21:43:03.0972117Z 86e5016c2693: Pull complete 2025-04-21T21:43:03.3954866Z 49e139a3d6c2: Pull complete 2025-04-21T21:43:04.1423559Z a14844c8c51f: Pull complete 2025-04-21T21:43:04.1648354Z 18fb524087fb: Pull complete 2025-04-21T21:43:04.1874210Z efd686a7b2c8: Pull complete 2025-04-21T21:43:05.0254221Z c68e78d48b95: Verifying Checksum 2025-04-21T21:43:05.0254601Z c68e78d48b95: Download complete 2025-04-21T21:43:05.1195661Z b14b320c459c: Verifying Checksum 2025-04-21T21:43:05.1196135Z b14b320c459c: Download complete 2025-04-21T21:43:05.1948740Z 0d929bfaefee: Verifying Checksum 2025-04-21T21:43:05.1949247Z 0d929bfaefee: Download complete 2025-04-21T21:43:05.3683809Z a5a8ea5a3eab: Verifying Checksum 2025-04-21T21:43:05.3684165Z a5a8ea5a3eab: Download complete 2025-04-21T21:43:06.5479496Z 281f13ef6929: Verifying Checksum 2025-04-21T21:43:06.5479951Z 281f13ef6929: Download complete 2025-04-21T21:43:06.6514851Z a8695f1c94c3: Verifying Checksum 2025-04-21T21:43:06.6515280Z a8695f1c94c3: Download complete 2025-04-21T21:43:06.7500368Z c72f323be801: Verifying Checksum 2025-04-21T21:43:06.7500707Z c72f323be801: Download complete 2025-04-21T21:43:06.8518015Z 4afa91cd0662: Download complete 2025-04-21T21:43:06.9481517Z a9bdca78eb50: Verifying Checksum 2025-04-21T21:43:06.9481963Z a9bdca78eb50: Download complete 2025-04-21T21:43:17.4441861Z 52c648e21334: Verifying Checksum 2025-04-21T21:43:17.4442266Z 52c648e21334: Download complete 2025-04-21T21:43:17.5387890Z 1a1f3a6b6e28: Download complete 2025-04-21T21:43:17.6177816Z 1e726e5b69df: Verifying Checksum 2025-04-21T21:43:17.6178323Z 1e726e5b69df: Download complete 2025-04-21T21:43:17.6992007Z e55ee61cc4aa: Verifying Checksum 2025-04-21T21:43:17.6992396Z e55ee61cc4aa: Download complete 2025-04-21T21:43:17.7983234Z d2c9e5f6d068: Verifying Checksum 2025-04-21T21:43:17.7983696Z d2c9e5f6d068: Download complete 2025-04-21T21:43:17.8928725Z a9e23e3c0a18: Download complete 2025-04-21T21:43:20.2144753Z 956077b15723: Verifying Checksum 2025-04-21T21:43:20.2145832Z 956077b15723: Download complete 2025-04-21T21:43:20.3561024Z b840d58dab14: Verifying Checksum 2025-04-21T21:43:20.3561368Z b840d58dab14: Download complete 2025-04-21T21:43:20.4580147Z 135c1f91f3df: Verifying Checksum 2025-04-21T21:43:20.4580599Z 135c1f91f3df: Download complete 2025-04-21T21:43:27.2128212Z 2fba85b42323: Verifying Checksum 2025-04-21T21:43:27.2128614Z 2fba85b42323: Download complete 2025-04-21T21:43:27.3066051Z 7e377528c596: Verifying Checksum 2025-04-21T21:43:27.3066485Z 7e377528c596: Download complete 2025-04-21T21:43:27.3963159Z 7c1fe59ec87a: Verifying Checksum 2025-04-21T21:43:27.3963493Z 7c1fe59ec87a: Download complete 2025-04-21T21:43:28.4531876Z bb83f375ed7d: Verifying Checksum 2025-04-21T21:43:28.4532377Z bb83f375ed7d: Download complete 2025-04-21T21:43:28.5570411Z d1ee3dcc43c5: Verifying Checksum 2025-04-21T21:43:28.5570956Z d1ee3dcc43c5: Download complete 2025-04-21T21:43:28.6528466Z 54687ddf5aab: Download complete 2025-04-21T21:43:28.9562186Z 1e6c6f2d2459: Verifying Checksum 2025-04-21T21:43:28.9562530Z 1e6c6f2d2459: Download complete 2025-04-21T21:43:28.9655126Z 6b673e167d0f: Verifying Checksum 2025-04-21T21:43:28.9655427Z 6b673e167d0f: Download complete 2025-04-21T21:43:29.0447707Z a4f475d6d4d7: Verifying Checksum 2025-04-21T21:43:29.0448059Z a4f475d6d4d7: Download complete 2025-04-21T21:43:29.0919968Z 48e60e8ea738: Verifying Checksum 2025-04-21T21:43:29.0920816Z 48e60e8ea738: Download complete 2025-04-21T21:43:29.1448935Z d52c79949221: Download complete 2025-04-21T21:43:29.2300793Z 595e99541716: Verifying Checksum 2025-04-21T21:43:29.2301259Z 595e99541716: Download complete 2025-04-21T21:43:29.3251033Z 013dfd09e68f: Download complete 2025-04-21T21:43:30.0187052Z 4402edece585: Verifying Checksum 2025-04-21T21:43:30.0187500Z 4402edece585: Download complete 2025-04-21T21:43:30.0646797Z 52c648e21334: Pull complete 2025-04-21T21:43:30.2016692Z 56e384e4e5aa: Pull complete 2025-04-21T21:43:30.2132654Z 82aa68866e53: Verifying Checksum 2025-04-21T21:43:30.2133078Z 82aa68866e53: Download complete 2025-04-21T21:43:30.3035120Z fb71b792ec6c: Pull complete 2025-04-21T21:43:30.3521340Z 5509576f2693: Pull complete 2025-04-21T21:43:35.3468106Z e624d917de21: Verifying Checksum 2025-04-21T21:43:35.3468560Z e624d917de21: Download complete 2025-04-21T21:43:53.1040285Z 87b133205740: Verifying Checksum 2025-04-21T21:43:53.1040637Z 87b133205740: Download complete 2025-04-21T21:43:53.2014059Z 3b25d713acb3: Verifying Checksum 2025-04-21T21:43:53.2014556Z 3b25d713acb3: Download complete 2025-04-21T21:43:53.3102641Z c0ad696e18d3: Verifying Checksum 2025-04-21T21:43:53.3103521Z c0ad696e18d3: Download complete 2025-04-21T21:43:53.4193486Z 767566b69317: Verifying Checksum 2025-04-21T21:43:53.4193795Z 767566b69317: Download complete 2025-04-21T21:43:53.4279631Z 4f4fb700ef54: Verifying Checksum 2025-04-21T21:43:53.4279960Z 4f4fb700ef54: Download complete 2025-04-21T21:43:53.5292278Z d83ab53d6ab4: Verifying Checksum 2025-04-21T21:43:53.5292598Z d83ab53d6ab4: Download complete 2025-04-21T21:43:53.6033122Z e9ae24bb3b5b: Verifying Checksum 2025-04-21T21:43:53.6033472Z e9ae24bb3b5b: Download complete 2025-04-21T21:44:00.3108464Z 8c11aa345fff: Verifying Checksum 2025-04-21T21:44:00.3108812Z 8c11aa345fff: Download complete 2025-04-21T21:44:00.4069083Z cc9ed3f492fa: Verifying Checksum 2025-04-21T21:44:00.4069416Z cc9ed3f492fa: Download complete 2025-04-21T21:44:00.4841507Z 6133da88e669: Verifying Checksum 2025-04-21T21:44:00.4841866Z 6133da88e669: Download complete 2025-04-21T21:44:00.5396892Z f55162ff6427: Verifying Checksum 2025-04-21T21:44:00.5397307Z f55162ff6427: Download complete 2025-04-21T21:44:00.6355618Z f2fb53f519b3: Verifying Checksum 2025-04-21T21:44:00.6356002Z f2fb53f519b3: Download complete 2025-04-21T21:44:01.0098896Z d93b86c61192: Verifying Checksum 2025-04-21T21:44:01.0099223Z d93b86c61192: Download complete 2025-04-21T21:44:01.0991867Z 2d75b00aef3e: Verifying Checksum 2025-04-21T21:44:01.0992199Z 2d75b00aef3e: Download complete 2025-04-21T21:44:02.8430925Z 165cb697180f: Verifying Checksum 2025-04-21T21:44:02.8431260Z 165cb697180f: Download complete 2025-04-21T21:44:02.9107460Z 952482b7353e: Download complete 2025-04-21T21:44:34.9609188Z f4fb942e1273: Verifying Checksum 2025-04-21T21:44:34.9609626Z f4fb942e1273: Download complete 2025-04-21T21:44:35.0584291Z 22d7ad62e40c: Verifying Checksum 2025-04-21T21:44:35.0584627Z 22d7ad62e40c: Download complete 2025-04-21T21:44:35.1490847Z 340ddaf5579e: Verifying Checksum 2025-04-21T21:44:35.1491502Z 340ddaf5579e: Download complete 2025-04-21T21:44:44.7607771Z abb8b598f420: Verifying Checksum 2025-04-21T21:44:44.7608106Z abb8b598f420: Download complete 2025-04-21T21:44:44.9108327Z 410b8ad7da03: Download complete 2025-04-21T21:44:45.0600946Z 79d2e9034f2b: Download complete 2025-04-21T21:44:45.1503878Z c2128146ebc3: Verifying Checksum 2025-04-21T21:44:45.1508375Z c2128146ebc3: Download complete 2025-04-21T21:44:59.9101335Z 4b1e8ee3de38: Verifying Checksum 2025-04-21T21:44:59.9101713Z 4b1e8ee3de38: Download complete 2025-04-21T21:44:59.9981441Z 02d23db28b3d: Verifying Checksum 2025-04-21T21:44:59.9981789Z 02d23db28b3d: Download complete 2025-04-21T21:45:00.1011308Z 24d4ae140876: Verifying Checksum 2025-04-21T21:45:00.1011636Z 24d4ae140876: Download complete 2025-04-21T21:45:00.1851035Z 2604f3be2ad1: Verifying Checksum 2025-04-21T21:45:00.1851359Z 2604f3be2ad1: Download complete 2025-04-21T21:45:00.2552895Z ba564700c169: Verifying Checksum 2025-04-21T21:45:00.2553536Z ba564700c169: Download complete 2025-04-21T21:45:18.8965153Z 846d8aa26523: Verifying Checksum 2025-04-21T21:45:18.8965480Z 846d8aa26523: Download complete 2025-04-21T21:45:33.8417848Z 1e6c6f2d2459: Pull complete 2025-04-21T21:45:33.9311254Z 79d2c9f506ec: Verifying Checksum 2025-04-21T21:45:33.9311593Z 79d2c9f506ec: Download complete 2025-04-21T21:45:34.0649877Z a1fe8922734d: Pull complete 2025-04-21T21:45:34.2574905Z 84e5433913c6: Pull complete 2025-04-21T21:45:41.7887354Z c68e78d48b95: Pull complete 2025-04-21T21:45:41.9021982Z b14b320c459c: Pull complete 2025-04-21T21:45:42.0379759Z 0d929bfaefee: Pull complete 2025-04-21T21:45:42.2319915Z a5a8ea5a3eab: Pull complete 2025-04-21T21:45:44.5179461Z 281f13ef6929: Pull complete 2025-04-21T21:45:44.7239011Z a8695f1c94c3: Pull complete 2025-04-21T21:45:44.9299676Z c72f323be801: Pull complete 2025-04-21T21:45:45.1488110Z 4afa91cd0662: Pull complete 2025-04-21T21:45:45.3506046Z a9bdca78eb50: Pull complete 2025-04-21T21:46:40.3799489Z 87b133205740: Pull complete 2025-04-21T21:46:40.5970722Z 1a1f3a6b6e28: Pull complete 2025-04-21T21:46:40.7737398Z 1e726e5b69df: Pull complete 2025-04-21T21:46:40.9876928Z e55ee61cc4aa: Pull complete 2025-04-21T21:46:41.1599901Z d2c9e5f6d068: Pull complete 2025-04-21T21:46:41.2998153Z a9e23e3c0a18: Pull complete 2025-04-21T21:46:43.7477433Z 956077b15723: Pull complete 2025-04-21T21:46:43.9577285Z b840d58dab14: Pull complete 2025-04-21T21:46:44.0891850Z 135c1f91f3df: Pull complete 2025-04-21T21:46:52.4444912Z 2fba85b42323: Pull complete 2025-04-21T21:46:52.6372694Z 7e377528c596: Pull complete 2025-04-21T21:46:52.8358103Z 7c1fe59ec87a: Pull complete 2025-04-21T21:46:53.6026371Z bb83f375ed7d: Pull complete 2025-04-21T21:46:53.7762771Z d1ee3dcc43c5: Pull complete 2025-04-21T21:46:53.9062902Z 54687ddf5aab: Pull complete 2025-04-21T21:46:54.2482908Z 6b673e167d0f: Pull complete 2025-04-21T21:46:54.3960351Z a4f475d6d4d7: Pull complete 2025-04-21T21:46:54.6542090Z 48e60e8ea738: Pull complete 2025-04-21T21:46:54.7814707Z d52c79949221: Pull complete 2025-04-21T21:47:31.2274866Z 846d8aa26523: Pull complete 2025-04-21T21:47:31.4495118Z 595e99541716: Pull complete 2025-04-21T21:47:31.6530245Z 013dfd09e68f: Pull complete 2025-04-21T21:47:34.1877145Z 4402edece585: Pull complete 2025-04-21T21:47:34.4115659Z 82aa68866e53: Pull complete 2025-04-21T21:47:35.8145607Z e624d917de21: Pull complete 2025-04-21T21:49:05.8703087Z 79d2c9f506ec: Pull complete 2025-04-21T21:49:05.8933011Z 3b25d713acb3: Pull complete 2025-04-21T21:49:05.9157920Z c0ad696e18d3: Pull complete 2025-04-21T21:49:05.9601989Z 767566b69317: Pull complete 2025-04-21T21:49:05.9820004Z 4f4fb700ef54: Pull complete 2025-04-21T21:49:06.0045265Z d83ab53d6ab4: Pull complete 2025-04-21T21:49:06.0295069Z e9ae24bb3b5b: Pull complete 2025-04-21T21:49:13.1887915Z 8c11aa345fff: Pull complete 2025-04-21T21:49:13.3938569Z cc9ed3f492fa: Pull complete 2025-04-21T21:49:13.6055151Z 6133da88e669: Pull complete 2025-04-21T21:49:13.8298680Z f55162ff6427: Pull complete 2025-04-21T21:49:14.0401723Z f2fb53f519b3: Pull complete 2025-04-21T21:49:15.3901074Z d93b86c61192: Pull complete 2025-04-21T21:49:15.5884508Z 2d75b00aef3e: Pull complete 2025-04-21T21:49:17.2500988Z 165cb697180f: Pull complete 2025-04-21T21:49:17.4567432Z 952482b7353e: Pull complete 2025-04-21T21:49:28.0616106Z f4fb942e1273: Pull complete 2025-04-21T21:49:28.1970140Z 22d7ad62e40c: Pull complete 2025-04-21T21:49:28.3184168Z 340ddaf5579e: Pull complete 2025-04-21T21:49:31.3231300Z abb8b598f420: Pull complete 2025-04-21T21:49:31.5501999Z 410b8ad7da03: Pull complete 2025-04-21T21:49:31.7644255Z 79d2e9034f2b: Pull complete 2025-04-21T21:49:31.9876078Z c2128146ebc3: Pull complete 2025-04-21T21:49:36.3548383Z 4b1e8ee3de38: Pull complete 2025-04-21T21:49:36.4934747Z 02d23db28b3d: Pull complete 2025-04-21T21:49:36.6106850Z 24d4ae140876: Pull complete 2025-04-21T21:49:36.7610826Z 2604f3be2ad1: Pull complete 2025-04-21T21:49:36.9693341Z ba564700c169: Pull complete 2025-04-21T21:49:37.8760301Z Digest: sha256:745e08885687216ce5adf3196dea7f49a9467042491210addb990693d7fef90a 2025-04-21T21:49:37.9192171Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:49:37.9373882Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:49:37.9432606Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:49:37.9433507Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-04-21T21:49:37.9444559Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:49:37.9444909Z env: 2025-04-21T21:49:37.9445117Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:49:37.9445366Z ##[endgroup] 2025-04-21T21:49:37.9709759Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2025-04-21T21:49:37.9710167Z with: 2025-04-21T21:49:37.9710381Z driver-version: 570.133.07 2025-04-21T21:49:37.9710637Z env: 2025-04-21T21:49:37.9710846Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:49:37.9711092Z ##[endgroup] 2025-04-21T21:49:37.9807041Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2025-04-21T21:49:37.9807447Z with: 2025-04-21T21:49:37.9807661Z timeout_minutes: 10 2025-04-21T21:49:37.9807909Z max_attempts: 3 2025-04-21T21:49:37.9833756Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" # Turn off persistent mode so that the installation script can unload the kernel module sudo killall nvidia-persistenced || true else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2025-04-21T21:49:37.9858697Z retry_wait_seconds: 10 2025-04-21T21:49:37.9858970Z polling_interval_seconds: 1 2025-04-21T21:49:37.9859242Z warning_on_retry: true 2025-04-21T21:49:37.9859501Z continue_on_error: false 2025-04-21T21:49:37.9859747Z env: 2025-04-21T21:49:37.9859949Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:49:37.9860210Z DRIVER_VERSION: 570.133.07 2025-04-21T21:49:37.9860467Z ##[endgroup] 2025-04-21T21:49:38.0764457Z == Installing nvidia driver NVIDIA-Linux-x86_64-570.133.07.run == 2025-04-21T21:49:38.0765237Z + pre_install_nvidia_driver_amzn2 2025-04-21T21:49:38.0765851Z + sudo yum remove -y nvidia-driver-latest-dkms 2025-04-21T21:49:38.7043110Z No match for argument: nvidia-driver-latest-dkms 2025-04-21T21:49:38.7043737Z No packages marked for removal. 2025-04-21T21:49:38.7229119Z Dependencies resolved. 2025-04-21T21:49:38.7239946Z Nothing to do. 2025-04-21T21:49:38.7241042Z Complete! 2025-04-21T21:49:38.7858931Z + install_nvidia_driver_common 2025-04-21T21:49:38.7863178Z + echo 'Before installing NVIDIA driver' 2025-04-21T21:49:38.7863606Z + lspci 2025-04-21T21:49:38.7864616Z Before installing NVIDIA driver 2025-04-21T21:49:38.7986269Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-04-21T21:49:38.7986802Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-04-21T21:49:38.7987513Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-04-21T21:49:38.7988297Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-04-21T21:49:38.7988969Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-04-21T21:49:38.7989707Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-04-21T21:49:38.7990794Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2025-04-21T21:49:38.7991515Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-04-21T21:49:38.7992101Z + lsmod 2025-04-21T21:49:38.8242586Z Module Size Used by 2025-04-21T21:49:38.8242900Z veth 36864 0 2025-04-21T21:49:38.8243182Z nvidia_modeset 1716224 0 2025-04-21T21:49:38.8243479Z video 65536 1 nvidia_modeset 2025-04-21T21:49:38.8243785Z wmi 36864 1 video 2025-04-21T21:49:38.8244105Z nvidia_uvm 1884160 0 2025-04-21T21:49:38.8244414Z nvidia 11583488 7 nvidia_uvm,nvidia_modeset 2025-04-21T21:49:38.8244756Z drm 602112 1 nvidia 2025-04-21T21:49:38.8245067Z drm_panel_orientation_quirks 32768 1 drm 2025-04-21T21:49:38.8245444Z backlight 24576 3 video,drm,nvidia_modeset 2025-04-21T21:49:38.8245858Z i2c_core 110592 2 nvidia,drm 2025-04-21T21:49:38.8246170Z xt_conntrack 16384 1 2025-04-21T21:49:38.8246464Z nft_chain_nat 16384 3 2025-04-21T21:49:38.8246742Z xt_MASQUERADE 20480 1 2025-04-21T21:49:38.8247047Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-04-21T21:49:38.8247401Z nf_conntrack_netlink 57344 0 2025-04-21T21:49:38.8247817Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-04-21T21:49:38.8248267Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-04-21T21:49:38.8248595Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-04-21T21:49:38.8248912Z xfrm_user 57344 1 2025-04-21T21:49:38.8249192Z xfrm_algo 16384 1 xfrm_user 2025-04-21T21:49:38.8249491Z xt_addrtype 16384 2 2025-04-21T21:49:38.8249770Z nft_compat 20480 4 2025-04-21T21:49:38.8250095Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-04-21T21:49:38.8250513Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-04-21T21:49:38.8250916Z br_netfilter 36864 0 2025-04-21T21:49:38.8251211Z bridge 323584 1 br_netfilter 2025-04-21T21:49:38.8251508Z stp 16384 1 bridge 2025-04-21T21:49:38.8251804Z llc 16384 2 bridge,stp 2025-04-21T21:49:38.8252098Z overlay 167936 0 2025-04-21T21:49:38.8252358Z tls 135168 0 2025-04-21T21:49:38.8252618Z nls_ascii 16384 1 2025-04-21T21:49:38.8252879Z nls_cp437 20480 1 2025-04-21T21:49:38.8253137Z vfat 24576 1 2025-04-21T21:49:38.8253385Z fat 86016 1 vfat 2025-04-21T21:49:38.8253664Z sunrpc 696320 1 2025-04-21T21:49:38.8253934Z ena 180224 0 2025-04-21T21:49:38.8254183Z i8042 45056 0 2025-04-21T21:49:38.8254449Z serio 28672 3 i8042 2025-04-21T21:49:38.8254731Z button 24576 0 2025-04-21T21:49:38.8254992Z ghash_clmulni_intel 16384 0 2025-04-21T21:49:38.8255273Z sch_fq_codel 20480 17 2025-04-21T21:49:38.8257312Z dm_mod 188416 0 2025-04-21T21:49:38.8257601Z loop 36864 0 2025-04-21T21:49:38.8258001Z fuse 163840 1 2025-04-21T21:49:38.8258267Z configfs 57344 1 2025-04-21T21:49:38.8258529Z dax 45056 1 dm_mod 2025-04-21T21:49:38.8258822Z dmi_sysfs 20480 0 2025-04-21T21:49:38.8259091Z crc32_pclmul 16384 0 2025-04-21T21:49:38.8259363Z crc32c_intel 24576 0 2025-04-21T21:49:38.8259624Z efivarfs 24576 1 2025-04-21T21:49:38.8259891Z + modinfo nvidia 2025-04-21T21:49:38.8263140Z filename: /lib/modules/6.1.130-139.222.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-04-21T21:49:38.8263630Z import_ns: DMA_BUF 2025-04-21T21:49:38.8263890Z alias: char-major-195-* 2025-04-21T21:49:38.8264178Z version: 570.133.07 2025-04-21T21:49:38.8264439Z supported: external 2025-04-21T21:49:38.8264697Z license: Dual MIT/GPL 2025-04-21T21:49:38.8265184Z firmware: nvidia/570.133.07/gsp_tu10x.bin 2025-04-21T21:49:38.8265551Z firmware: nvidia/570.133.07/gsp_ga10x.bin 2025-04-21T21:49:38.8265881Z srcversion: 49515739FD8F721A3F2F714 2025-04-21T21:49:38.8266222Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-04-21T21:49:38.8266584Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-04-21T21:49:38.8266932Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-04-21T21:49:38.8267266Z depends: i2c-core,drm 2025-04-21T21:49:38.8267536Z retpoline: Y 2025-04-21T21:49:38.8267770Z name: nvidia 2025-04-21T21:49:38.8268141Z vermagic: 6.1.130-139.222.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-04-21T21:49:38.8268640Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-04-21T21:49:38.8269112Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-04-21T21:49:38.8269652Z parm: NVreg_ResmanDebugLevel:int 2025-04-21T21:49:38.8269987Z parm: NVreg_RmLogonRC:int 2025-04-21T21:49:38.8270313Z parm: NVreg_ModifyDeviceFiles:int 2025-04-21T21:49:38.8270639Z parm: NVreg_DeviceFileUID:int 2025-04-21T21:49:38.8271005Z parm: NVreg_DeviceFileGID:int 2025-04-21T21:49:38.8271334Z parm: NVreg_DeviceFileMode:int 2025-04-21T21:49:38.8271714Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-04-21T21:49:38.8272117Z parm: NVreg_UsePageAttributeTable:int 2025-04-21T21:49:38.8272467Z parm: NVreg_EnablePCIeGen3:int 2025-04-21T21:49:38.8272783Z parm: NVreg_EnableMSI:int 2025-04-21T21:49:38.8273098Z parm: NVreg_EnableStreamMemOPs:int 2025-04-21T21:49:38.8273484Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-04-21T21:49:38.8273911Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-04-21T21:49:38.8274307Z parm: NVreg_EnableS0ixPowerManagement:int 2025-04-21T21:49:38.8274741Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-04-21T21:49:38.8275272Z parm: NVreg_DynamicPowerManagement:int 2025-04-21T21:49:38.8275761Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-04-21T21:49:38.8276183Z parm: NVreg_EnableGpuFirmware:int 2025-04-21T21:49:38.8276527Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-04-21T21:49:38.8276897Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-04-21T21:49:38.8277282Z parm: NVreg_EnableUserNUMAManagement:int 2025-04-21T21:49:38.8277635Z parm: NVreg_MemoryPoolSize:int 2025-04-21T21:49:38.8277965Z parm: NVreg_KMallocHeapMaxSize:int 2025-04-21T21:49:38.8278304Z parm: NVreg_VMallocHeapMaxSize:int 2025-04-21T21:49:38.8278640Z parm: NVreg_IgnoreMMIOCheck:int 2025-04-21T21:49:38.8278951Z parm: NVreg_NvLinkDisable:int 2025-04-21T21:49:38.8279309Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-04-21T21:49:38.8279680Z parm: NVreg_RegisterPCIDriver:int 2025-04-21T21:49:38.8280154Z parm: NVreg_EnableResizableBar:int 2025-04-21T21:49:38.8280490Z parm: NVreg_EnableDbgBreakpoint:int 2025-04-21T21:49:38.8280870Z parm: NVreg_EnableNonblockingOpen:int 2025-04-21T21:49:38.8281250Z parm: NVreg_RegistryDwords:charp 2025-04-21T21:49:38.8281590Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-04-21T21:49:38.8281929Z parm: NVreg_RmMsg:charp 2025-04-21T21:49:38.8282224Z parm: NVreg_GpuBlacklist:charp 2025-04-21T21:49:38.8282557Z parm: NVreg_TemporaryFilePath:charp 2025-04-21T21:49:38.8282883Z parm: NVreg_ExcludedGpus:charp 2025-04-21T21:49:38.8283212Z parm: NVreg_DmaRemapPeerMmio:int 2025-04-21T21:49:38.8283552Z parm: NVreg_RmNvlinkBandwidth:charp 2025-04-21T21:49:38.8283915Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-04-21T21:49:38.8284277Z parm: NVreg_ImexChannelCount:int 2025-04-21T21:49:38.8284619Z parm: NVreg_CreateImexChannel0:int 2025-04-21T21:49:38.8285069Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-04-21T21:49:38.8285422Z parm: rm_firmware_active:charp 2025-04-21T21:49:38.8285719Z + HAS_NVIDIA_DRIVER=0 2025-04-21T21:49:38.8285963Z ++ command -v nvidia-smi 2025-04-21T21:49:38.8286240Z + '[' -x /usr/bin/nvidia-smi ']' 2025-04-21T21:49:38.8286504Z + set +e 2025-04-21T21:49:38.8286823Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2025-04-21T21:49:38.8509395Z + INSTALLED_DRIVER_VERSION=570.133.07 2025-04-21T21:49:38.8509714Z + NVIDIA_SMI_STATUS=0 2025-04-21T21:49:38.8509961Z + '[' 0 -ne 0 ']' 2025-04-21T21:49:38.8510178Z + '[' 570.133.07 '!=' 570.133.07 ']' 2025-04-21T21:49:38.8510450Z + HAS_NVIDIA_DRIVER=1 2025-04-21T21:49:38.8510894Z + echo 'NVIDIA driver (570.133.07) has already been installed. Skipping NVIDIA driver installation' 2025-04-21T21:49:38.8511378Z + set -e 2025-04-21T21:49:38.8511578Z + '[' 1 -eq 0 ']' 2025-04-21T21:49:38.8511986Z NVIDIA driver (570.133.07) has already been installed. Skipping NVIDIA driver installation 2025-04-21T21:49:38.8512492Z + post_install_nvidia_driver_common 2025-04-21T21:49:38.8516775Z + sudo modprobe nvidia 2025-04-21T21:49:38.9595014Z + echo 'After installing NVIDIA driver' 2025-04-21T21:49:38.9595388Z + lspci 2025-04-21T21:49:38.9595677Z After installing NVIDIA driver 2025-04-21T21:49:38.9711594Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2025-04-21T21:49:38.9712128Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2025-04-21T21:49:38.9712682Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2025-04-21T21:49:38.9713217Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2025-04-21T21:49:38.9713710Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2025-04-21T21:49:38.9714247Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2025-04-21T21:49:38.9714772Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2025-04-21T21:49:38.9715280Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2025-04-21T21:49:38.9715689Z + lsmod 2025-04-21T21:49:38.9743474Z Module Size Used by 2025-04-21T21:49:38.9743764Z veth 36864 0 2025-04-21T21:49:38.9744043Z nvidia_modeset 1716224 0 2025-04-21T21:49:38.9744339Z video 65536 1 nvidia_modeset 2025-04-21T21:49:38.9744644Z wmi 36864 1 video 2025-04-21T21:49:38.9744922Z nvidia_uvm 1884160 0 2025-04-21T21:49:38.9745225Z nvidia 11583488 7 nvidia_uvm,nvidia_modeset 2025-04-21T21:49:38.9745549Z drm 602112 1 nvidia 2025-04-21T21:49:38.9745859Z drm_panel_orientation_quirks 32768 1 drm 2025-04-21T21:49:38.9746225Z backlight 24576 3 video,drm,nvidia_modeset 2025-04-21T21:49:38.9746571Z i2c_core 110592 2 nvidia,drm 2025-04-21T21:49:38.9746870Z xt_conntrack 16384 1 2025-04-21T21:49:38.9747393Z nft_chain_nat 16384 3 2025-04-21T21:49:38.9747653Z xt_MASQUERADE 20480 1 2025-04-21T21:49:38.9747960Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2025-04-21T21:49:38.9748299Z nf_conntrack_netlink 57344 0 2025-04-21T21:49:38.9748705Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2025-04-21T21:49:38.9749148Z nf_defrag_ipv6 24576 1 nf_conntrack 2025-04-21T21:49:38.9749465Z nf_defrag_ipv4 16384 1 nf_conntrack 2025-04-21T21:49:38.9749760Z xfrm_user 57344 1 2025-04-21T21:49:38.9750022Z xfrm_algo 16384 1 xfrm_user 2025-04-21T21:49:38.9750316Z xt_addrtype 16384 2 2025-04-21T21:49:38.9750576Z nft_compat 20480 4 2025-04-21T21:49:38.9750879Z nf_tables 311296 57 nft_compat,nft_chain_nat 2025-04-21T21:49:38.9751310Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2025-04-21T21:49:38.9751692Z br_netfilter 36864 0 2025-04-21T21:49:38.9752145Z bridge 323584 1 br_netfilter 2025-04-21T21:49:38.9752444Z stp 16384 1 bridge 2025-04-21T21:49:38.9752737Z llc 16384 2 bridge,stp 2025-04-21T21:49:38.9753026Z overlay 167936 0 2025-04-21T21:49:38.9753275Z tls 135168 0 2025-04-21T21:49:38.9753530Z nls_ascii 16384 1 2025-04-21T21:49:38.9753786Z nls_cp437 20480 1 2025-04-21T21:49:38.9754032Z vfat 24576 1 2025-04-21T21:49:38.9754284Z fat 86016 1 vfat 2025-04-21T21:49:38.9754556Z sunrpc 696320 1 2025-04-21T21:49:38.9754801Z ena 180224 0 2025-04-21T21:49:38.9755049Z i8042 45056 0 2025-04-21T21:49:38.9755306Z serio 28672 3 i8042 2025-04-21T21:49:38.9755583Z button 24576 0 2025-04-21T21:49:38.9755840Z ghash_clmulni_intel 16384 0 2025-04-21T21:49:38.9756112Z sch_fq_codel 20480 17 2025-04-21T21:49:38.9756382Z dm_mod 188416 0 2025-04-21T21:49:38.9756630Z loop 36864 0 2025-04-21T21:49:38.9756880Z fuse 163840 1 2025-04-21T21:49:38.9757136Z configfs 57344 1 2025-04-21T21:49:38.9757393Z dax 45056 1 dm_mod 2025-04-21T21:49:38.9757678Z dmi_sysfs 20480 0 2025-04-21T21:49:38.9757934Z crc32_pclmul 16384 0 2025-04-21T21:49:38.9758187Z crc32c_intel 24576 0 2025-04-21T21:49:38.9758445Z efivarfs 24576 1 2025-04-21T21:49:38.9758701Z + modinfo nvidia 2025-04-21T21:49:38.9760450Z filename: /lib/modules/6.1.130-139.222.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2025-04-21T21:49:38.9760928Z import_ns: DMA_BUF 2025-04-21T21:49:38.9761184Z alias: char-major-195-* 2025-04-21T21:49:38.9761458Z version: 570.133.07 2025-04-21T21:49:38.9761705Z supported: external 2025-04-21T21:49:38.9761963Z license: Dual MIT/GPL 2025-04-21T21:49:38.9762268Z firmware: nvidia/570.133.07/gsp_tu10x.bin 2025-04-21T21:49:38.9762611Z firmware: nvidia/570.133.07/gsp_ga10x.bin 2025-04-21T21:49:38.9762946Z srcversion: 49515739FD8F721A3F2F714 2025-04-21T21:49:38.9763275Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2025-04-21T21:49:38.9763620Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2025-04-21T21:49:38.9763957Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2025-04-21T21:49:38.9764283Z depends: i2c-core,drm 2025-04-21T21:49:38.9764544Z retpoline: Y 2025-04-21T21:49:38.9764767Z name: nvidia 2025-04-21T21:49:38.9765134Z vermagic: 6.1.130-139.222.amzn2023.x86_64 SMP preempt mod_unload modversions 2025-04-21T21:49:38.9765619Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2025-04-21T21:49:38.9766070Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2025-04-21T21:49:38.9766499Z parm: NVreg_ResmanDebugLevel:int 2025-04-21T21:49:38.9766822Z parm: NVreg_RmLogonRC:int 2025-04-21T21:49:38.9767234Z parm: NVreg_ModifyDeviceFiles:int 2025-04-21T21:49:38.9767548Z parm: NVreg_DeviceFileUID:int 2025-04-21T21:49:38.9767855Z parm: NVreg_DeviceFileGID:int 2025-04-21T21:49:38.9768166Z parm: NVreg_DeviceFileMode:int 2025-04-21T21:49:38.9768530Z parm: NVreg_InitializeSystemMemoryAllocations:int 2025-04-21T21:49:38.9768930Z parm: NVreg_UsePageAttributeTable:int 2025-04-21T21:49:38.9769269Z parm: NVreg_EnablePCIeGen3:int 2025-04-21T21:49:38.9769575Z parm: NVreg_EnableMSI:int 2025-04-21T21:49:38.9769887Z parm: NVreg_EnableStreamMemOPs:int 2025-04-21T21:49:38.9770255Z parm: NVreg_RestrictProfilingToAdminUsers:int 2025-04-21T21:49:38.9770662Z parm: NVreg_PreserveVideoMemoryAllocations:int 2025-04-21T21:49:38.9771050Z parm: NVreg_EnableS0ixPowerManagement:int 2025-04-21T21:49:38.9771473Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2025-04-21T21:49:38.9771993Z parm: NVreg_DynamicPowerManagement:int 2025-04-21T21:49:38.9772424Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2025-04-21T21:49:38.9772845Z parm: NVreg_EnableGpuFirmware:int 2025-04-21T21:49:38.9773192Z parm: NVreg_EnableGpuFirmwareLogs:int 2025-04-21T21:49:38.9773565Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2025-04-21T21:49:38.9773947Z parm: NVreg_EnableUserNUMAManagement:int 2025-04-21T21:49:38.9774301Z parm: NVreg_MemoryPoolSize:int 2025-04-21T21:49:38.9774632Z parm: NVreg_KMallocHeapMaxSize:int 2025-04-21T21:49:38.9774970Z parm: NVreg_VMallocHeapMaxSize:int 2025-04-21T21:49:38.9775299Z parm: NVreg_IgnoreMMIOCheck:int 2025-04-21T21:49:38.9775616Z parm: NVreg_NvLinkDisable:int 2025-04-21T21:49:38.9775963Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2025-04-21T21:49:38.9776336Z parm: NVreg_RegisterPCIDriver:int 2025-04-21T21:49:38.9776680Z parm: NVreg_EnableResizableBar:int 2025-04-21T21:49:38.9777023Z parm: NVreg_EnableDbgBreakpoint:int 2025-04-21T21:49:38.9777377Z parm: NVreg_EnableNonblockingOpen:int 2025-04-21T21:49:38.9777860Z parm: NVreg_RegistryDwords:charp 2025-04-21T21:49:38.9778206Z parm: NVreg_RegistryDwordsPerDevice:charp 2025-04-21T21:49:38.9778546Z parm: NVreg_RmMsg:charp 2025-04-21T21:49:38.9778840Z parm: NVreg_GpuBlacklist:charp 2025-04-21T21:49:38.9779177Z parm: NVreg_TemporaryFilePath:charp 2025-04-21T21:49:38.9779505Z parm: NVreg_ExcludedGpus:charp 2025-04-21T21:49:38.9779830Z parm: NVreg_DmaRemapPeerMmio:int 2025-04-21T21:49:38.9780171Z parm: NVreg_RmNvlinkBandwidth:charp 2025-04-21T21:49:38.9780574Z parm: NVreg_RmNvlinkBandwidthLinkCount:int 2025-04-21T21:49:38.9780944Z parm: NVreg_ImexChannelCount:int 2025-04-21T21:49:38.9781280Z parm: NVreg_CreateImexChannel0:int 2025-04-21T21:49:38.9781643Z parm: NVreg_GrdmaPciTopoCheckOverride:int 2025-04-21T21:49:38.9782001Z parm: rm_firmware_active:charp 2025-04-21T21:49:38.9782295Z + set +e 2025-04-21T21:49:38.9782486Z + nvidia-smi 2025-04-21T21:49:38.9949603Z Mon Apr 21 21:49:38 2025 2025-04-21T21:49:38.9950329Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:49:38.9951109Z | NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 | 2025-04-21T21:49:38.9951634Z |-----------------------------------------+------------------------+----------------------+ 2025-04-21T21:49:38.9952143Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-04-21T21:49:38.9952682Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-04-21T21:49:38.9953146Z | | | MIG M. | 2025-04-21T21:49:38.9953510Z |=========================================+========================+======================| 2025-04-21T21:49:39.0109587Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2025-04-21T21:49:39.0110049Z | 0% 28C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2025-04-21T21:49:39.0110441Z | | | N/A | 2025-04-21T21:49:39.0110848Z +-----------------------------------------+------------------------+----------------------+ 2025-04-21T21:49:39.0114290Z 2025-04-21T21:49:39.0114700Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:49:39.0115128Z | Processes: | 2025-04-21T21:49:39.0115755Z | GPU GI CI PID Type Process name GPU Memory | 2025-04-21T21:49:39.0116194Z | ID ID Usage | 2025-04-21T21:49:39.0116551Z |=========================================================================================| 2025-04-21T21:49:39.0119213Z | No running processes found | 2025-04-21T21:49:39.0119810Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:49:39.2623898Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-04-21T21:49:39.2785017Z NVIDIA A10G 2025-04-21T21:49:39.2829065Z + NVIDIA_SMI_STATUS=0 2025-04-21T21:49:39.2829325Z + '[' 0 -eq 0 ']' 2025-04-21T21:49:39.2829584Z + echo 'INFO: Ignoring allowed status 0' 2025-04-21T21:49:39.2829879Z + set -e 2025-04-21T21:49:39.2830096Z INFO: Ignoring allowed status 0 2025-04-21T21:49:39.2838980Z == Installing nvidia container toolkit for amzn2023 == 2025-04-21T21:49:39.2842698Z + sudo yum install -y yum-utils 2025-04-21T21:49:39.8480055Z Last metadata expiration check: 1:47:59 ago on Mon Apr 21 20:01:40 2025. 2025-04-21T21:49:39.8734736Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed. 2025-04-21T21:49:39.9136753Z Dependencies resolved. 2025-04-21T21:49:39.9318788Z Nothing to do. 2025-04-21T21:49:39.9319129Z Complete! 2025-04-21T21:49:40.0062421Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2025-04-21T21:49:40.0063217Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-04-21T21:49:40.0064241Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-04-21T21:49:40.3193730Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2025-04-21T21:49:40.3782322Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2025-04-21T21:49:40.8811005Z nvidia-container-toolkit 15 kB/s | 833 B 00:00 2025-04-21T21:49:40.9063070Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2025-04-21T21:49:40.9070121Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed. 2025-04-21T21:49:40.9465098Z Dependencies resolved. 2025-04-21T21:49:40.9649101Z Nothing to do. 2025-04-21T21:49:40.9649437Z Complete! 2025-04-21T21:49:41.0628142Z + sudo systemctl restart docker 2025-04-21T21:50:29.1765704Z nvidia-persistenced failed to initialize. Check syslog for more details. 2025-04-21T21:50:29.1962877Z Mon Apr 21 21:50:29 2025 2025-04-21T21:50:29.1965289Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:50:29.1965988Z | NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 | 2025-04-21T21:50:29.1966501Z |-----------------------------------------+------------------------+----------------------+ 2025-04-21T21:50:29.1967040Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-04-21T21:50:29.1967964Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-04-21T21:50:29.1968420Z | | | MIG M. | 2025-04-21T21:50:29.1968771Z |=========================================+========================+======================| 2025-04-21T21:50:29.2094757Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2025-04-21T21:50:29.2095360Z | 0% 28C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2025-04-21T21:50:29.2095752Z | | | N/A | 2025-04-21T21:50:29.2096172Z +-----------------------------------------+------------------------+----------------------+ 2025-04-21T21:50:29.2098440Z 2025-04-21T21:50:29.2099147Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:50:29.2099597Z | Processes: | 2025-04-21T21:50:29.2100048Z | GPU GI CI PID Type Process name GPU Memory | 2025-04-21T21:50:29.2100482Z | ID ID Usage | 2025-04-21T21:50:29.2100839Z |=========================================================================================| 2025-04-21T21:50:29.2104716Z | No running processes found | 2025-04-21T21:50:29.2105194Z +-----------------------------------------------------------------------------------------+ 2025-04-21T21:50:30.1049206Z Command completed after 1 attempt(s). 2025-04-21T21:50:30.1156140Z Prepare all required actions 2025-04-21T21:50:30.1189380Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-04-21T21:50:30.1189795Z with: 2025-04-21T21:50:30.1190291Z github-token: *** 2025-04-21T21:50:30.1190557Z env: 2025-04-21T21:50:30.1190809Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:30.1191194Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:30.1191607Z ##[endgroup] 2025-04-21T21:50:30.1214998Z ##[group]Run set -eux 2025-04-21T21:50:30.1215307Z set -eux 2025-04-21T21:50:30.1215815Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-04-21T21:50:30.1231465Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:30.1231895Z env: 2025-04-21T21:50:30.1232146Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:30.1232522Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:30.1233085Z GITHUB_TOKEN: *** 2025-04-21T21:50:30.1233359Z ##[endgroup] 2025-04-21T21:50:30.1267212Z + python3 .github/scripts/get_workflow_job_id.py 14580756816 i-02e84e4685011efd7 2025-04-21T21:50:31.1897838Z setting job-id=40900324323 2025-04-21T21:50:31.1898903Z setting job-name=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:31.2027415Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-04-21T21:50:31.2028262Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7 2025-04-21T21:50:31.2028937Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2025-04-21T21:50:31.2029478Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-04-21T21:50:31.2039195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:31.2039627Z env: 2025-04-21T21:50:31.2039873Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:31.2040272Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:31.2040674Z JOB_ID: 40900324323 2025-04-21T21:50:31.2041214Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:31.2042016Z WORKFLOW_NAME: inductor 2025-04-21T21:50:31.2042332Z WORKFLOW_RUN_ID: 14580756816 2025-04-21T21:50:31.2042679Z ##[endgroup] 2025-04-21T21:50:31.4579876Z Defaulting to user installation because normal site-packages is not writeable 2025-04-21T21:50:31.4767660Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2025-04-21T21:50:31.4771786Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2025-04-21T21:50:31.4775904Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-04-21T21:50:31.4876865Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-04-21T21:50:31.4880243Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-04-21T21:50:31.4964521Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-04-21T21:50:31.4990549Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.0.0) 2025-04-21T21:50:31.4998495Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.13.2) 2025-04-21T21:50:31.6931300Z Prepare all required actions 2025-04-21T21:50:31.6931673Z Getting action download info 2025-04-21T21:50:31.8406208Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-04-21T21:50:32.2555823Z Download action repository 'actions/download-artifact@v4' (SHA:95815c38cf2ff2164869cbab79da8d1f422bc89e) 2025-04-21T21:50:32.5294661Z ##[group]Run ./.github/actions/download-build-artifacts 2025-04-21T21:50:32.5295021Z with: 2025-04-21T21:50:32.5295268Z name: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-21T21:50:32.5295588Z s3-bucket: gha-artifacts 2025-04-21T21:50:32.5295833Z env: 2025-04-21T21:50:32.5296040Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:32.5296370Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:32.5296708Z ##[endgroup] 2025-04-21T21:50:32.5333763Z ##[group]Run seemethere/download-artifact-s3@v4 2025-04-21T21:50:32.5334098Z with: 2025-04-21T21:50:32.5334391Z name: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-21T21:50:32.5334711Z s3-bucket: gha-artifacts 2025-04-21T21:50:32.5334980Z region: us-east-1 2025-04-21T21:50:32.5335198Z env: 2025-04-21T21:50:32.5335404Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:32.5335729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:32.5336073Z ##[endgroup] 2025-04-21T21:50:33.0254540Z (node:289943) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-04-21T21:50:33.0255295Z 2025-04-21T21:50:33.0255601Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-04-21T21:50:33.0256409Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-04-21T21:50:33.0257302Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-04-21T21:50:33.1371087Z Found 1 objects with prefix pytorch/pytorch/14580756816/linux-focal-cuda12.6-py3.10-gcc9-sm86/ 2025-04-21T21:50:33.1372425Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-04-21T21:50:40.8868014Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-04-21T21:50:40.8873811Z Artifact download has finished successfully 2025-04-21T21:50:40.9262661Z ##[group]Run unzip -o artifacts.zip 2025-04-21T21:50:40.9262983Z unzip -o artifacts.zip 2025-04-21T21:50:40.9272103Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:40.9272456Z env: 2025-04-21T21:50:40.9272658Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:40.9272971Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:40.9273309Z ##[endgroup] 2025-04-21T21:50:40.9322558Z Archive: artifacts.zip 2025-04-21T21:50:40.9324415Z creating: dist/ 2025-04-21T21:50:43.1079585Z inflating: dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl 2025-04-21T21:50:43.1213301Z inflating: dist/.ninja_log 2025-04-21T21:50:43.1214045Z creating: build/custom_test_artifacts/ 2025-04-21T21:50:43.1214651Z creating: build/custom_test_artifacts/custom-op-build/ 2025-04-21T21:50:43.1215328Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-04-21T21:50:43.1216165Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-04-21T21:50:43.1224749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-04-21T21:50:43.1225735Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/ 2025-04-21T21:50:43.1226644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-04-21T21:50:43.1227652Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-04-21T21:50:43.1228710Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-04-21T21:50:43.1230731Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-04-21T21:50:43.1233601Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-04-21T21:50:43.1234668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-04-21T21:50:43.1235687Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-04-21T21:50:43.1236737Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-04-21T21:50:43.1239241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-04-21T21:50:43.1241131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-04-21T21:50:43.1242705Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-04-21T21:50:43.1245370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-04-21T21:50:43.1247946Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-04-21T21:50:43.1249053Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-04-21T21:50:43.1250100Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-04-21T21:50:43.1292156Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-04-21T21:50:43.1334183Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-04-21T21:50:43.1335691Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-04-21T21:50:43.1383736Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-04-21T21:50:43.1385193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-04-21T21:50:43.1386651Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-04-21T21:50:43.1388136Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-04-21T21:50:43.1389859Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-04-21T21:50:43.1391279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-04-21T21:50:43.1392713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-04-21T21:50:43.1394119Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-04-21T21:50:43.1395783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-04-21T21:50:43.1397080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-04-21T21:50:43.1398348Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-04-21T21:50:43.1399604Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-04-21T21:50:43.1400860Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-04-21T21:50:43.1402106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-04-21T21:50:43.1403964Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-04-21T21:50:43.1479649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-04-21T21:50:43.1480802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-04-21T21:50:43.1556378Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-04-21T21:50:43.1557485Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-04-21T21:50:43.1558371Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-04-21T21:50:43.1559301Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-04-21T21:50:43.1560224Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-04-21T21:50:43.1561255Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-04-21T21:50:43.1562428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-04-21T21:50:43.1563565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-04-21T21:50:43.1564628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-04-21T21:50:43.1565719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-04-21T21:50:43.1566819Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-04-21T21:50:43.1567905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-04-21T21:50:43.1569039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-04-21T21:50:43.1570166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-04-21T21:50:43.1591244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-04-21T21:50:43.1792503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-04-21T21:50:43.1793516Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-04-21T21:50:43.1794783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-04-21T21:50:43.1796238Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-04-21T21:50:43.1797416Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-04-21T21:50:43.1798590Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-04-21T21:50:43.1799720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-04-21T21:50:43.1800892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-04-21T21:50:43.1802026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-04-21T21:50:43.1803177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-04-21T21:50:43.1804369Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-04-21T21:50:43.1825832Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-04-21T21:50:43.1909942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-04-21T21:50:43.1911163Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-04-21T21:50:43.1912258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-04-21T21:50:43.1913470Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-04-21T21:50:43.1914402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-04-21T21:50:43.1916142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-04-21T21:50:43.1917210Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2025-04-21T21:50:43.1920147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-04-21T21:50:43.1921282Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-04-21T21:50:43.1922391Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-04-21T21:50:43.2087152Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-04-21T21:50:43.2144019Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-04-21T21:50:43.2144538Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-04-21T21:50:43.2144999Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-04-21T21:50:43.2145799Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-04-21T21:50:43.2154356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-04-21T21:50:43.2154989Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/ 2025-04-21T21:50:43.2155627Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-04-21T21:50:43.2156288Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-04-21T21:50:43.2157422Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-04-21T21:50:43.2160716Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-04-21T21:50:43.2162690Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-04-21T21:50:43.2163585Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-04-21T21:50:43.2164279Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-04-21T21:50:43.2164941Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-04-21T21:50:43.2168157Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-04-21T21:50:43.2170124Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-04-21T21:50:43.2171290Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-04-21T21:50:43.2173687Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-04-21T21:50:43.2175980Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-04-21T21:50:43.2176739Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-04-21T21:50:43.2177417Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-04-21T21:50:43.2219737Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-04-21T21:50:43.2261899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-04-21T21:50:43.2263876Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-04-21T21:50:43.2309934Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-04-21T21:50:43.2311375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-04-21T21:50:43.2312913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-04-21T21:50:43.2314415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-04-21T21:50:43.2315858Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-04-21T21:50:43.2317260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-04-21T21:50:43.2318683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-04-21T21:50:43.2320110Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-04-21T21:50:43.2321494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-04-21T21:50:43.2322791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-04-21T21:50:43.2324055Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-04-21T21:50:43.2325300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-04-21T21:50:43.2326562Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-04-21T21:50:43.2327799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-04-21T21:50:43.2330695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-04-21T21:50:43.2405824Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-04-21T21:50:43.2406949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-04-21T21:50:43.2482012Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-04-21T21:50:43.2483111Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-04-21T21:50:43.2484160Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-04-21T21:50:43.2485061Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-04-21T21:50:43.2486010Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-04-21T21:50:43.2487079Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-04-21T21:50:43.2488313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-04-21T21:50:43.2489477Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-04-21T21:50:43.2490564Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-04-21T21:50:43.2491688Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-04-21T21:50:43.2492823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-04-21T21:50:43.2493963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-04-21T21:50:43.2495338Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-04-21T21:50:43.2496481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-04-21T21:50:43.2517300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-04-21T21:50:43.2581787Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-04-21T21:50:43.2583003Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-04-21T21:50:43.2584115Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-04-21T21:50:43.2585127Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-04-21T21:50:43.2586015Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-04-21T21:50:43.2587939Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-04-21T21:50:43.2588843Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2025-04-21T21:50:43.2591947Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-04-21T21:50:43.2593114Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-04-21T21:50:43.2594463Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-04-21T21:50:43.2635237Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-04-21T21:50:43.2635759Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-04-21T21:50:43.2636280Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-04-21T21:50:43.2636886Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-04-21T21:50:43.2645310Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-04-21T21:50:43.2645996Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/ 2025-04-21T21:50:43.2646681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeSystem.cmake 2025-04-21T21:50:43.2647410Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/ 2025-04-21T21:50:43.2648116Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/tmp/ 2025-04-21T21:50:43.2651282Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c 2025-04-21T21:50:43.2653184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/a.out 2025-04-21T21:50:43.2654256Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake 2025-04-21T21:50:43.2654999Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/ 2025-04-21T21:50:43.2655728Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/ 2025-04-21T21:50:43.2658895Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-04-21T21:50:43.2660807Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out 2025-04-21T21:50:43.2662093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake 2025-04-21T21:50:43.2664305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin 2025-04-21T21:50:43.2666623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin 2025-04-21T21:50:43.2667454Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/ 2025-04-21T21:50:43.2668200Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/ 2025-04-21T21:50:43.2710240Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2025-04-21T21:50:43.2751945Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2025-04-21T21:50:43.2753003Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2025-04-21T21:50:43.2802469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2025-04-21T21:50:43.2803502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2025-04-21T21:50:43.2804682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2025-04-21T21:50:43.2805893Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2025-04-21T21:50:43.2806942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2025-04-21T21:50:43.2808158Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2025-04-21T21:50:43.2809498Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2025-04-21T21:50:43.2810861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2025-04-21T21:50:43.2813301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2025-04-21T21:50:43.2814599Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2025-04-21T21:50:43.2815961Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c 2025-04-21T21:50:43.2817296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin 2025-04-21T21:50:43.2818731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2025-04-21T21:50:43.2820025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o 2025-04-21T21:50:43.2823285Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu 2025-04-21T21:50:43.2898457Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out 2025-04-21T21:50:43.2899670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake 2025-04-21T21:50:43.2974878Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin 2025-04-21T21:50:43.2976041Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-04-21T21:50:43.2976979Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-04-21T21:50:43.2978047Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-04-21T21:50:43.2979112Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-04-21T21:50:43.2980263Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-04-21T21:50:43.2981569Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-04-21T21:50:43.2982836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-04-21T21:50:43.2984001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-04-21T21:50:43.2985198Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-04-21T21:50:43.2986782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-04-21T21:50:43.2988059Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-04-21T21:50:43.2989354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-04-21T21:50:43.2990542Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-04-21T21:50:43.2992573Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-04-21T21:50:43.3116143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-04-21T21:50:43.3117366Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-04-21T21:50:43.3118585Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-04-21T21:50:43.3119945Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-04-21T21:50:43.3121244Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-04-21T21:50:43.3122504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-04-21T21:50:43.3123770Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-04-21T21:50:43.3125055Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-04-21T21:50:43.3126345Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-04-21T21:50:43.3127700Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-04-21T21:50:43.3128973Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-04-21T21:50:43.3149235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-04-21T21:50:43.3205677Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-04-21T21:50:43.3207209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-04-21T21:50:43.3208421Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-04-21T21:50:43.3209519Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-04-21T21:50:43.3210505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-04-21T21:50:43.3211983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-04-21T21:50:43.3212962Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2025-04-21T21:50:43.3216481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-04-21T21:50:43.3217556Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-04-21T21:50:43.3218837Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-04-21T21:50:43.3320144Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-04-21T21:50:43.3359623Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-04-21T21:50:43.3360352Z creating: build/lib/ 2025-04-21T21:50:43.3449471Z inflating: build/lib/libprotobuf-lite.a 2025-04-21T21:50:43.3905887Z inflating: build/lib/libprotobuf.a 2025-04-21T21:50:43.4409785Z inflating: build/lib/libprotoc.a 2025-04-21T21:50:43.4420110Z inflating: build/lib/libpthreadpool.a 2025-04-21T21:50:43.4428446Z inflating: build/lib/libcpuinfo.a 2025-04-21T21:50:43.4436747Z inflating: build/lib/libcpuinfo_internals.a 2025-04-21T21:50:43.4437682Z inflating: build/lib/libclog.a 2025-04-21T21:50:43.4457775Z inflating: build/lib/libpytorch_qnnpack.a 2025-04-21T21:50:43.4460722Z inflating: build/lib/libnnpack_reference_layers.a 2025-04-21T21:50:43.4479260Z inflating: build/lib/libnnpack.a 2025-04-21T21:50:43.4666589Z inflating: build/lib/libmicrokernels-prod.a 2025-04-21T21:50:43.5471461Z inflating: build/lib/libmicrokernels-all.a 2025-04-21T21:50:43.5563529Z inflating: build/lib/libXNNPACK.a 2025-04-21T21:50:43.5632787Z inflating: build/lib/libgtest.a 2025-04-21T21:50:43.5650583Z inflating: build/lib/libgmock.a 2025-04-21T21:50:43.5651349Z inflating: build/lib/libgmock_main.a 2025-04-21T21:50:43.5652917Z inflating: build/lib/libgtest_main.a 2025-04-21T21:50:43.5728575Z inflating: build/lib/libbenchmark.a 2025-04-21T21:50:43.5729314Z inflating: build/lib/libbenchmark_main.a 2025-04-21T21:50:43.5794185Z inflating: build/lib/libasmjit.a 2025-04-21T21:50:43.7050942Z inflating: build/lib/libfbgemm.a 2025-04-21T21:50:43.7059021Z inflating: build/lib/libittnotify.a 2025-04-21T21:50:43.7087956Z inflating: build/lib/libtensorpipe_uv.a 2025-04-21T21:50:43.7664635Z inflating: build/lib/libtensorpipe.a 2025-04-21T21:50:43.7920295Z inflating: build/lib/libtensorpipe_cuda.a 2025-04-21T21:50:43.8056262Z inflating: build/lib/libgloo.a 2025-04-21T21:50:43.8447338Z inflating: build/lib/libgloo_cuda.a 2025-04-21T21:50:43.8490373Z inflating: build/lib/libonnx_proto.a 2025-04-21T21:50:43.9220212Z inflating: build/lib/libonnx.a 2025-04-21T21:50:44.9862449Z inflating: build/lib/libdnnl.a 2025-04-21T21:50:44.9882199Z inflating: build/lib/libfmt.a 2025-04-21T21:50:45.0351990Z inflating: build/lib/libkineto.a 2025-04-21T21:50:45.0463829Z inflating: build/lib/libc10.so 2025-04-21T21:50:45.0522359Z inflating: build/lib/libc10_cuda.so 2025-04-21T21:50:45.0524936Z inflating: build/lib/libcaffe2_nvrtc.so 2025-04-21T21:50:45.0526045Z inflating: build/lib/libtorch_global_deps.so 2025-04-21T21:50:47.6148507Z inflating: build/lib/libtorch_cpu.so 2025-04-21T21:50:49.6654017Z inflating: build/lib/libtorch_cuda.so 2025-04-21T21:50:49.6659088Z inflating: build/lib/libunbox_lib.a 2025-04-21T21:50:49.6660389Z inflating: build/lib/libtorch.so 2025-04-21T21:50:50.5502301Z inflating: build/lib/libtorch_cuda_linalg.so 2025-04-21T21:50:50.5572607Z inflating: build/lib/libtorchbind_test.so 2025-04-21T21:50:50.5593146Z inflating: build/lib/libjitbackend_test.so 2025-04-21T21:50:50.5618606Z inflating: build/lib/libbackend_with_compiler.so 2025-04-21T21:50:50.5643550Z inflating: build/lib/libaoti_custom_ops.so 2025-04-21T21:50:50.5647557Z inflating: build/lib/libc10d_cuda_test.so 2025-04-21T21:50:50.5650971Z inflating: build/lib/libshm.so 2025-04-21T21:50:50.7740259Z inflating: build/lib/libtorch_python.so 2025-04-21T21:50:50.7774747Z inflating: build/lib/libnnapi_backend.so 2025-04-21T21:50:50.7775084Z creating: build/bin/ 2025-04-21T21:50:50.8225844Z inflating: build/bin/protoc-3.13.0.0 2025-04-21T21:50:50.8676787Z inflating: build/bin/protoc 2025-04-21T21:50:50.8731375Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-04-21T21:50:50.8786284Z inflating: build/bin/c10_DeviceGuard_test 2025-04-21T21:50:50.8842043Z inflating: build/bin/c10_Device_test 2025-04-21T21:50:50.8905866Z inflating: build/bin/c10_DispatchKeySet_test 2025-04-21T21:50:50.8964326Z inflating: build/bin/c10_Scalar_test 2025-04-21T21:50:50.9016774Z inflating: build/bin/c10_StreamGuard_test 2025-04-21T21:50:50.9071631Z inflating: build/bin/c10_SymInt_test 2025-04-21T21:50:50.9129772Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-04-21T21:50:50.9189006Z inflating: build/bin/c10_InlineStreamGuard_test 2025-04-21T21:50:50.9250591Z inflating: build/bin/c10_SizesAndStrides_test 2025-04-21T21:50:50.9324995Z inflating: build/bin/c10_cow_test 2025-04-21T21:50:50.9378476Z inflating: build/bin/c10_ArrayRef_test 2025-04-21T21:50:50.9435234Z inflating: build/bin/c10_Bitset_test 2025-04-21T21:50:50.9488162Z inflating: build/bin/c10_ConstexprCrc_test 2025-04-21T21:50:50.9541831Z inflating: build/bin/c10_DeadlockDetection_test 2025-04-21T21:50:50.9596113Z inflating: build/bin/c10_Half_test 2025-04-21T21:50:50.9659084Z inflating: build/bin/c10_LeftRight_test 2025-04-21T21:50:50.9717999Z inflating: build/bin/c10_Metaprogramming_test 2025-04-21T21:50:50.9771588Z inflating: build/bin/c10_Synchronized_test 2025-04-21T21:50:50.9828534Z inflating: build/bin/c10_NetworkFlow_test 2025-04-21T21:50:50.9887668Z inflating: build/bin/c10_ThreadLocal_test 2025-04-21T21:50:50.9943697Z inflating: build/bin/c10_TypeIndex_test 2025-04-21T21:50:50.9998870Z inflating: build/bin/c10_TypeList_test 2025-04-21T21:50:51.0051404Z inflating: build/bin/c10_TypeTraits_test 2025-04-21T21:50:51.0107270Z inflating: build/bin/c10_accumulate_test 2025-04-21T21:50:51.0166719Z inflating: build/bin/c10_bfloat16_test 2025-04-21T21:50:51.0221457Z inflating: build/bin/c10_bit_cast_test 2025-04-21T21:50:51.0284202Z inflating: build/bin/c10_complex_math_test 2025-04-21T21:50:51.0343315Z inflating: build/bin/c10_complex_test 2025-04-21T21:50:51.0396346Z inflating: build/bin/c10_error_test 2025-04-21T21:50:51.0452735Z inflating: build/bin/c10_exception_test 2025-04-21T21:50:51.0507300Z inflating: build/bin/c10_flags_test 2025-04-21T21:50:51.0678960Z inflating: build/bin/c10_intrusive_ptr_test 2025-04-21T21:50:51.0733197Z inflating: build/bin/c10_generic_math_test 2025-04-21T21:50:51.0788431Z inflating: build/bin/c10_irange_test 2025-04-21T21:50:51.0845884Z inflating: build/bin/c10_lazy_test 2025-04-21T21:50:51.0907346Z inflating: build/bin/c10_logging_test 2025-04-21T21:50:51.0988933Z inflating: build/bin/c10_optional_test 2025-04-21T21:50:51.1055521Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-04-21T21:50:51.1112880Z inflating: build/bin/c10_registry_test 2025-04-21T21:50:51.1273626Z inflating: build/bin/c10_small_vector_test 2025-04-21T21:50:51.1331215Z inflating: build/bin/c10_ssize_test 2025-04-21T21:50:51.1387896Z inflating: build/bin/c10_string_util_test 2025-04-21T21:50:51.1440097Z inflating: build/bin/c10_string_view_test 2025-04-21T21:50:51.1494711Z inflating: build/bin/c10_tempfile_test 2025-04-21T21:50:51.1554603Z inflating: build/bin/c10_typeid_test 2025-04-21T21:50:51.1604013Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-04-21T21:50:51.1660457Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2025-04-21T21:50:51.1717131Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2025-04-21T21:50:51.1773884Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2025-04-21T21:50:51.1829087Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2025-04-21T21:50:51.1885684Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2025-04-21T21:50:51.1941918Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2025-04-21T21:50:51.1994480Z inflating: build/bin/c10_cuda_CUDATest 2025-04-21T21:50:51.2051557Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2025-04-21T21:50:51.2478371Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-04-21T21:50:51.2912107Z inflating: build/bin/vec_test_all_types_AVX512 2025-04-21T21:50:51.3362294Z inflating: build/bin/vec_test_all_types_AVX2 2025-04-21T21:50:51.3419111Z inflating: build/bin/test_edge_op_registration 2025-04-21T21:50:51.3498130Z inflating: build/bin/Dict_test 2025-04-21T21:50:51.3553495Z inflating: build/bin/Dimname_test 2025-04-21T21:50:51.3622394Z inflating: build/bin/MaybeOwned_test 2025-04-21T21:50:51.3684102Z inflating: build/bin/NamedTensor_test 2025-04-21T21:50:51.3746522Z inflating: build/bin/apply_utils_test 2025-04-21T21:50:51.3810475Z inflating: build/bin/atest 2025-04-21T21:50:51.3877460Z inflating: build/bin/basic 2025-04-21T21:50:51.3936382Z inflating: build/bin/broadcast_test 2025-04-21T21:50:51.3991352Z inflating: build/bin/cpu_allocator_test 2025-04-21T21:50:51.4054237Z inflating: build/bin/cpu_generator_test 2025-04-21T21:50:51.4111757Z inflating: build/bin/cpu_profiling_allocator_test 2025-04-21T21:50:51.4211786Z inflating: build/bin/cpu_rng_test 2025-04-21T21:50:51.4265663Z inflating: build/bin/dlconvertor_test 2025-04-21T21:50:51.4328231Z inflating: build/bin/extension_backend_test 2025-04-21T21:50:51.4388339Z inflating: build/bin/half_test 2025-04-21T21:50:51.4490582Z inflating: build/bin/ivalue_test 2025-04-21T21:50:51.4544324Z inflating: build/bin/lazy_tensor_test 2025-04-21T21:50:51.4602487Z inflating: build/bin/math_kernel_test 2025-04-21T21:50:51.4660350Z inflating: build/bin/memory_format_test 2025-04-21T21:50:51.4717657Z inflating: build/bin/memory_overlapping_test 2025-04-21T21:50:51.4774226Z inflating: build/bin/mobile_memory_cleanup 2025-04-21T21:50:51.4834476Z inflating: build/bin/native_test 2025-04-21T21:50:51.4888948Z inflating: build/bin/operator_name_test 2025-04-21T21:50:51.4944517Z inflating: build/bin/operators_test 2025-04-21T21:50:51.5000181Z inflating: build/bin/packedtensoraccessor_test 2025-04-21T21:50:51.5071273Z inflating: build/bin/pow_test 2025-04-21T21:50:51.5132604Z inflating: build/bin/quantized_test 2025-04-21T21:50:51.5185989Z inflating: build/bin/reduce_ops_test 2025-04-21T21:50:51.5241404Z inflating: build/bin/reportMemoryUsage_test 2025-04-21T21:50:51.5302342Z inflating: build/bin/scalar_tensor_test 2025-04-21T21:50:51.5365380Z inflating: build/bin/scalar_test 2025-04-21T21:50:51.5422524Z inflating: build/bin/StorageUtils_test 2025-04-21T21:50:51.5479866Z inflating: build/bin/stride_properties_test 2025-04-21T21:50:51.5563637Z inflating: build/bin/tensor_iterator_test 2025-04-21T21:50:51.5622003Z inflating: build/bin/test_parallel 2025-04-21T21:50:51.5625188Z inflating: build/bin/thread_init_test 2025-04-21T21:50:51.5684678Z inflating: build/bin/type_ptr_test 2025-04-21T21:50:51.5750428Z inflating: build/bin/type_test 2025-04-21T21:50:51.5806605Z inflating: build/bin/undefined_tensor_test 2025-04-21T21:50:51.5808496Z inflating: build/bin/verify_api_visibility 2025-04-21T21:50:51.5883132Z inflating: build/bin/legacy_vmap_test 2025-04-21T21:50:51.5938785Z inflating: build/bin/weakref_test 2025-04-21T21:50:51.5993965Z inflating: build/bin/wrapdim_test 2025-04-21T21:50:51.6050072Z inflating: build/bin/xla_tensor_test 2025-04-21T21:50:51.6114132Z inflating: build/bin/IListRef_test 2025-04-21T21:50:51.6226277Z inflating: build/bin/List_test 2025-04-21T21:50:51.6296891Z inflating: build/bin/KernelFunction_test 2025-04-21T21:50:51.6422913Z inflating: build/bin/kernel_function_legacy_test 2025-04-21T21:50:51.6526538Z inflating: build/bin/kernel_function_test 2025-04-21T21:50:51.6659710Z inflating: build/bin/kernel_lambda_legacy_test 2025-04-21T21:50:51.6767739Z inflating: build/bin/kernel_lambda_test 2025-04-21T21:50:51.6832803Z inflating: build/bin/kernel_stackbased_test 2025-04-21T21:50:51.6933553Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-04-21T21:50:51.6988635Z inflating: build/bin/CppSignature_test 2025-04-21T21:50:51.7048330Z inflating: build/bin/backend_fallback_test 2025-04-21T21:50:51.7101177Z inflating: build/bin/op_allowlist_test 2025-04-21T21:50:51.7411737Z inflating: build/bin/op_registration_test 2025-04-21T21:50:51.7481829Z inflating: build/bin/inline_container_test 2025-04-21T21:50:51.7540150Z inflating: build/bin/cuda_allocator_test 2025-04-21T21:50:51.7596545Z inflating: build/bin/cuda_apply_test 2025-04-21T21:50:51.7660836Z inflating: build/bin/cuda_atomic_ops_test 2025-04-21T21:50:51.7721908Z inflating: build/bin/cuda_caching_host_allocator_test 2025-04-21T21:50:51.7797458Z inflating: build/bin/cuda_complex_math_test 2025-04-21T21:50:51.7860744Z inflating: build/bin/cuda_complex_test 2025-04-21T21:50:51.7927858Z inflating: build/bin/cuda_cub_test 2025-04-21T21:50:51.7980968Z inflating: build/bin/cuda_device_test 2025-04-21T21:50:51.8050194Z inflating: build/bin/cuda_distributions_test 2025-04-21T21:50:51.8104598Z inflating: build/bin/cuda_dlconvertor_test 2025-04-21T21:50:51.8165664Z inflating: build/bin/cuda_generator_test 2025-04-21T21:50:51.8219946Z inflating: build/bin/cuda_half_test 2025-04-21T21:50:51.8275012Z inflating: build/bin/cuda_integer_divider_test 2025-04-21T21:50:51.8328507Z inflating: build/bin/cuda_optional_test 2025-04-21T21:50:51.8384401Z inflating: build/bin/cuda_packedtensoraccessor_test 2025-04-21T21:50:51.8440889Z inflating: build/bin/cuda_reportMemoryUsage_test 2025-04-21T21:50:51.8495116Z inflating: build/bin/cuda_allocatorTraceTracker_test 2025-04-21T21:50:51.8559666Z inflating: build/bin/cuda_stream_test 2025-04-21T21:50:51.8615989Z inflating: build/bin/cuda_vectorized_test 2025-04-21T21:50:51.8669352Z inflating: build/bin/cuda_cudnn_test 2025-04-21T21:50:51.9261106Z inflating: build/bin/test_jit 2025-04-21T21:50:52.0107921Z inflating: build/bin/test_tensorexpr 2025-04-21T21:50:52.0122653Z inflating: build/bin/tutorial_tensorexpr 2025-04-21T21:50:52.0179325Z inflating: build/bin/BackoffTest 2025-04-21T21:50:52.0236078Z inflating: build/bin/FileStoreTest 2025-04-21T21:50:52.0295946Z inflating: build/bin/TCPStoreTest 2025-04-21T21:50:52.0352939Z inflating: build/bin/HashStoreTest 2025-04-21T21:50:52.0423730Z inflating: build/bin/ProcessGroupGlooTest 2025-04-21T21:50:52.0485967Z inflating: build/bin/ProcessGroupGlooAsyncTest 2025-04-21T21:50:52.0554037Z inflating: build/bin/ProcessGroupNCCLTest 2025-04-21T21:50:52.0568432Z inflating: build/bin/ProcessGroupMPITest 2025-04-21T21:50:52.0634480Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2025-04-21T21:50:52.0637530Z inflating: build/bin/example_allreduce 2025-04-21T21:50:52.0696913Z inflating: build/bin/test_dist_autograd 2025-04-21T21:50:52.0769365Z inflating: build/bin/test_cpp_rpc 2025-04-21T21:50:52.1985988Z inflating: build/bin/test_api 2025-04-21T21:50:52.1988919Z inflating: build/bin/parallel_benchmark 2025-04-21T21:50:52.2059530Z inflating: build/bin/test_mobile_nnc 2025-04-21T21:50:52.2068748Z inflating: build/bin/aot_model_compiler_test 2025-04-21T21:50:52.2427093Z inflating: build/bin/test_lazy 2025-04-21T21:50:52.2431147Z inflating: build/bin/torch_shm_manager 2025-04-21T21:50:52.2431489Z creating: .additional_ci_files/ 2025-04-21T21:50:52.2548039Z inflating: .additional_ci_files/test-times.json 2025-04-21T21:50:52.3003052Z inflating: .additional_ci_files/test-class-times.json 2025-04-21T21:50:52.3037732Z ##[group]Run rm artifacts.zip 2025-04-21T21:50:52.3038041Z rm artifacts.zip 2025-04-21T21:50:52.3046809Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:52.3047192Z env: 2025-04-21T21:50:52.3047402Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:52.3047756Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:52.3048102Z ##[endgroup] 2025-04-21T21:50:52.4791892Z ##[group]Run df -H 2025-04-21T21:50:52.4792140Z df -H 2025-04-21T21:50:52.4802367Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:52.4802730Z env: 2025-04-21T21:50:52.4802937Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:52.4803247Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:52.4803591Z ##[endgroup] 2025-04-21T21:50:52.5113781Z Filesystem Size Used Avail Use% Mounted on 2025-04-21T21:50:52.5114168Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-04-21T21:50:52.5114497Z tmpfs 34G 0 34G 0% /dev/shm 2025-04-21T21:50:52.5114829Z tmpfs 14G 553k 14G 1% /run 2025-04-21T21:50:52.5115143Z /dev/nvme0n1p1 161G 59G 103G 37% / 2025-04-21T21:50:52.5115471Z tmpfs 34G 25k 34G 1% /tmp 2025-04-21T21:50:52.5115811Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-04-21T21:50:52.5116161Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2025-04-21T21:50:52.5150573Z Prepare all required actions 2025-04-21T21:50:52.5151591Z Getting action download info 2025-04-21T21:50:52.6619560Z ##[group]Run ./.github/actions/download-td-artifacts 2025-04-21T21:50:52.6619892Z with: 2025-04-21T21:50:52.6620085Z env: 2025-04-21T21:50:52.6620292Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:52.6620609Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:52.6620954Z ##[endgroup] 2025-04-21T21:50:52.6721507Z ##[group]Run seemethere/download-artifact-s3@v4 2025-04-21T21:50:52.6721835Z with: 2025-04-21T21:50:52.6722033Z name: td_results 2025-04-21T21:50:52.6722271Z s3-bucket: gha-artifacts 2025-04-21T21:50:52.6722533Z region: us-east-1 2025-04-21T21:50:52.6722749Z env: 2025-04-21T21:50:52.6722952Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:52.6723288Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:52.6723635Z ##[endgroup] 2025-04-21T21:50:53.1368076Z (node:289966) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-04-21T21:50:53.1368567Z 2025-04-21T21:50:53.1368763Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-04-21T21:50:53.1369277Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-04-21T21:50:53.1369821Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-04-21T21:50:53.2252394Z Found 0 objects with prefix pytorch/pytorch/14580756816/td_results/ 2025-04-21T21:50:53.2262169Z Artifact download has finished successfully 2025-04-21T21:50:53.2684898Z ##[group]Run mkdir -p .additional_ci_files 2025-04-21T21:50:53.2685255Z mkdir -p .additional_ci_files 2025-04-21T21:50:53.2685671Z mv td_results.json .additional_ci_files/td_results.json || true 2025-04-21T21:50:53.2694967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:53.2695321Z env: 2025-04-21T21:50:53.2695528Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:53.2695849Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:53.2696210Z ##[endgroup] 2025-04-21T21:50:53.3202315Z mv: cannot stat 'td_results.json': No such file or directory 2025-04-21T21:50:53.3281638Z ##[group]Run .github/scripts/parse_ref.py 2025-04-21T21:50:53.3282006Z .github/scripts/parse_ref.py 2025-04-21T21:50:53.3290190Z shell: /usr/bin/bash -e {0} 2025-04-21T21:50:53.3290442Z env: 2025-04-21T21:50:53.3290647Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:53.3290962Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:53.3291304Z ##[endgroup] 2025-04-21T21:50:53.3630166Z Prepare all required actions 2025-04-21T21:50:53.3630541Z Getting action download info 2025-04-21T21:50:53.4778557Z ##[group]Run ./.github/actions/filter-test-configs 2025-04-21T21:50:53.4778883Z with: 2025-04-21T21:50:53.4779261Z github-token: *** 2025-04-21T21:50:53.4780890Z test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2025-04-21T21:50:53.4782838Z job-name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:53.4783330Z env: 2025-04-21T21:50:53.4783573Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:53.4783889Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:53.4784241Z ##[endgroup] 2025-04-21T21:50:53.4874728Z ##[group]Run nick-fields/retry@v3.0.0 2025-04-21T21:50:53.4875032Z with: 2025-04-21T21:50:53.4875227Z shell: bash 2025-04-21T21:50:53.4875440Z timeout_minutes: 10 2025-04-21T21:50:53.4875678Z max_attempts: 5 2025-04-21T21:50:53.4875906Z retry_wait_seconds: 30 2025-04-21T21:50:53.4876653Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-04-21T21:50:53.4877433Z polling_interval_seconds: 1 2025-04-21T21:50:53.4877709Z warning_on_retry: true 2025-04-21T21:50:53.4877964Z continue_on_error: false 2025-04-21T21:50:53.4878201Z env: 2025-04-21T21:50:53.4878409Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:53.4878734Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:53.4879232Z GITHUB_TOKEN: *** 2025-04-21T21:50:53.4879449Z ##[endgroup] 2025-04-21T21:50:53.5878321Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2025-04-21T21:50:53.8182593Z Defaulting to user installation because normal site-packages is not writeable 2025-04-21T21:50:53.8357863Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2025-04-21T21:50:53.8361462Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2025-04-21T21:50:53.8469736Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-04-21T21:50:53.8477936Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2025-04-21T21:50:53.8487217Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-04-21T21:50:53.8491034Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2025.1.31) 2025-04-21T21:50:54.5655106Z Command completed after 1 attempt(s). 2025-04-21T21:50:54.5728945Z ##[group]Run set -x 2025-04-21T21:50:54.5729272Z set -x 2025-04-21T21:50:54.5729494Z  2025-04-21T21:50:54.5729891Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-04-21T21:50:54.5730380Z # in runner workspace 2025-04-21T21:50:54.5730753Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-04-21T21:50:54.5739748Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:54.5740103Z env: 2025-04-21T21:50:54.5740314Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:54.5740635Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:54.5740973Z ##[endgroup] 2025-04-21T21:50:54.5769733Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-04-21T21:50:54.6006702Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-04-21T21:50:54.6007094Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-04-21T21:50:54.6007434Z echo "Job name: ${JOB_NAME}" 2025-04-21T21:50:54.6007718Z  2025-04-21T21:50:54.6008092Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-04-21T21:50:54.6008554Z # in runner workspace 2025-04-21T21:50:54.6008974Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-04-21T21:50:54.6009442Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-04-21T21:50:54.6009777Z  --job-name "${JOB_NAME}" \ 2025-04-21T21:50:54.6011556Z  --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2025-04-21T21:50:54.6013531Z  --selected-test-configs "" \ 2025-04-21T21:50:54.6013867Z  --pr-number "${PR_NUMBER}" \ 2025-04-21T21:50:54.6014179Z  --tag "${TAG}" \ 2025-04-21T21:50:54.6014470Z  --event-name "${EVENT_NAME}" \ 2025-04-21T21:50:54.6014786Z  --schedule "${SCHEDULE}" \ 2025-04-21T21:50:54.6015112Z  --branch "${HEAD_BRANCH}" 2025-04-21T21:50:54.6023275Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:54.6023633Z env: 2025-04-21T21:50:54.6023851Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:54.6024186Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:54.6024720Z GITHUB_TOKEN: *** 2025-04-21T21:50:54.6025171Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:54.6025688Z PR_NUMBER: 2025-04-21T21:50:54.6036801Z TAG: ciflow/inductor/148893 2025-04-21T21:50:54.6037120Z EVENT_NAME: push 2025-04-21T21:50:54.6037354Z SCHEDULE: 2025-04-21T21:50:54.6037581Z HEAD_BRANCH: 2025-04-21T21:50:54.6037811Z ##[endgroup] 2025-04-21T21:50:54.6069355Z Workflow: inductor 2025-04-21T21:50:54.6069989Z Job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:54.8709600Z INFO:root:Found no test-config label on the PR, so all test configs are included 2025-04-21T21:50:55.0408036Z ##[group]Run echo "Filtered matrix:" 2025-04-21T21:50:55.0408377Z echo "Filtered matrix:" 2025-04-21T21:50:55.0410108Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2025-04-21T21:50:55.0412008Z  2025-04-21T21:50:55.0412213Z echo 2025-04-21T21:50:55.0412485Z echo "Is the current job unstable? False" 2025-04-21T21:50:55.0412810Z  2025-04-21T21:50:55.0413005Z echo 2025-04-21T21:50:55.0413264Z echo "Is keep-going label set? False" 2025-04-21T21:50:55.0413581Z  2025-04-21T21:50:55.0413777Z echo 2025-04-21T21:50:55.0414015Z echo "Renabled issues? " 2025-04-21T21:50:55.0422962Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:55.0423514Z env: 2025-04-21T21:50:55.0423729Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:55.0424055Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:55.0424403Z ##[endgroup] 2025-04-21T21:50:55.0451400Z Filtered matrix: 2025-04-21T21:50:55.0453736Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2025-04-21T21:50:55.0455388Z 2025-04-21T21:50:55.0455504Z Is the current job unstable? False 2025-04-21T21:50:55.0455703Z 2025-04-21T21:50:55.0455819Z Is keep-going label set? False 2025-04-21T21:50:55.0456000Z 2025-04-21T21:50:55.0456087Z Renabled issues? 2025-04-21T21:50:55.0503718Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-04-21T21:50:55.0504232Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-04-21T21:50:55.0514588Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-21T21:50:55.0514966Z env: 2025-04-21T21:50:55.0515176Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:55.0515494Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:55.0515840Z JOB_TIMEOUT: 240 2025-04-21T21:50:55.0516069Z ##[endgroup] 2025-04-21T21:50:55.0597374Z ##[group]Run set -x 2025-04-21T21:50:55.0597701Z set -x 2025-04-21T21:50:55.0597926Z  2025-04-21T21:50:55.0598165Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-04-21T21:50:55.0598558Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-04-21T21:50:55.0598946Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-04-21T21:50:55.0599300Z  TEST_COMMAND=.ci/onnx/test.sh 2025-04-21T21:50:55.0599584Z else 2025-04-21T21:50:55.0599828Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-04-21T21:50:55.0600123Z fi 2025-04-21T21:50:55.0600319Z  2025-04-21T21:50:55.0600575Z # Leaving 1GB for the runner and other things 2025-04-21T21:50:55.0601133Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-04-21T21:50:55.0601995Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-04-21T21:50:55.0602676Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-04-21T21:50:55.0603184Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-04-21T21:50:55.0603579Z  2025-04-21T21:50:55.0603838Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-04-21T21:50:55.0604167Z  SHM_OPTS= 2025-04-21T21:50:55.0604411Z  JENKINS_USER= 2025-04-21T21:50:55.0604750Z  # ensure that docker container cleanly exits in 12 hours 2025-04-21T21:50:55.0605206Z  # if for some reason cleanup action doesn't stop container 2025-04-21T21:50:55.0605585Z  # when job is cancelled 2025-04-21T21:50:55.0606035Z  DOCKER_SHELL_CMD="sleep 12h" 2025-04-21T21:50:55.0606325Z  2025-04-21T21:50:55.0606668Z  # since some steps are skipped on s390x, if they are necessary, run them here 2025-04-21T21:50:55.0607185Z  env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-04-21T21:50:55.0607612Z  env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-04-21T21:50:55.0607953Z else 2025-04-21T21:50:55.0608194Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-04-21T21:50:55.0608522Z  JENKINS_USER="--user jenkins" 2025-04-21T21:50:55.0608823Z  DOCKER_SHELL_CMD= 2025-04-21T21:50:55.0609068Z fi 2025-04-21T21:50:55.0609268Z  2025-04-21T21:50:55.0609587Z # detached container should get cleaned up by teardown_ec2_linux 2025-04-21T21:50:55.0610097Z # TODO: Stop building test binaries as part of the build phase 2025-04-21T21:50:55.0610668Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-04-21T21:50:55.0611182Z # shellcheck disable=SC2086,SC2090 2025-04-21T21:50:55.0611508Z container_name=$(docker run \ 2025-04-21T21:50:55.0611801Z  ${GPU_FLAG:-} \ 2025-04-21T21:50:55.0612100Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-04-21T21:50:55.0612437Z  -e BUILD_ENVIRONMENT \ 2025-04-21T21:50:55.0612717Z  -e PR_NUMBER \ 2025-04-21T21:50:55.0612985Z  -e GITHUB_ACTIONS \ 2025-04-21T21:50:55.0613263Z  -e GITHUB_REPOSITORY \ 2025-04-21T21:50:55.0613550Z  -e GITHUB_WORKFLOW \ 2025-04-21T21:50:55.0613823Z  -e GITHUB_JOB \ 2025-04-21T21:50:55.0614086Z  -e GITHUB_RUN_ID \ 2025-04-21T21:50:55.0614358Z  -e GITHUB_RUN_NUMBER \ 2025-04-21T21:50:55.0614641Z  -e GITHUB_RUN_ATTEMPT \ 2025-04-21T21:50:55.0614932Z  -e JOB_ID \ 2025-04-21T21:50:55.0615181Z  -e JOB_NAME \ 2025-04-21T21:50:55.0615426Z  -e BASE_SHA \ 2025-04-21T21:50:55.0615676Z  -e BRANCH \ 2025-04-21T21:50:55.0615914Z  -e SHA1 \ 2025-04-21T21:50:55.0616151Z  -e AWS_DEFAULT_REGION \ 2025-04-21T21:50:55.0616556Z  -e IN_WHEEL_TEST \ 2025-04-21T21:50:55.0616828Z  -e SHARD_NUMBER \ 2025-04-21T21:50:55.0617100Z  -e TEST_CONFIG \ 2025-04-21T21:50:55.0617364Z  -e NUM_TEST_SHARDS \ 2025-04-21T21:50:55.0617723Z  -e REENABLED_ISSUES \ 2025-04-21T21:50:55.0618017Z  -e CONTINUE_THROUGH_ERROR \ 2025-04-21T21:50:55.0618310Z  -e VERBOSE_TEST_LOGS \ 2025-04-21T21:50:55.0618599Z  -e TEST_SHOWLOCALS \ 2025-04-21T21:50:55.0618882Z  -e NO_TEST_TIMEOUT \ 2025-04-21T21:50:55.0619138Z  -e NO_TD \ 2025-04-21T21:50:55.0619385Z  -e TD_DISTRIBUTED \ 2025-04-21T21:50:55.0619650Z  -e PR_LABELS \ 2025-04-21T21:50:55.0619946Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-04-21T21:50:55.0620257Z  -e SCCACHE_BUCKET \ 2025-04-21T21:50:55.0620530Z  -e SCCACHE_REGION \ 2025-04-21T21:50:55.0620796Z  -e XLA_CUDA \ 2025-04-21T21:50:55.0621071Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-04-21T21:50:55.0621430Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-04-21T21:50:55.0621791Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-04-21T21:50:55.0622138Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-04-21T21:50:55.0622466Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-04-21T21:50:55.0622787Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-04-21T21:50:55.0623106Z  -e DASHBOARD_TAG \ 2025-04-21T21:50:55.0623372Z  -e IS_A100_RUNNER \ 2025-04-21T21:50:55.0623650Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-04-21T21:50:55.0624003Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-04-21T21:50:55.0624399Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-04-21T21:50:55.0624884Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-04-21T21:50:55.0625265Z  --security-opt seccomp=unconfined \ 2025-04-21T21:50:55.0625599Z  --cap-add=SYS_PTRACE \ 2025-04-21T21:50:55.0625885Z  --ipc=host \ 2025-04-21T21:50:55.0626138Z  ${SHM_OPTS} \ 2025-04-21T21:50:55.0626380Z  --tty \ 2025-04-21T21:50:55.0626604Z  --detach \ 2025-04-21T21:50:55.0626868Z  --name="${container_name}" \ 2025-04-21T21:50:55.0627173Z  ${JENKINS_USER} \ 2025-04-21T21:50:55.0627503Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-04-21T21:50:55.0627891Z  -w /var/lib/jenkins/workspace \ 2025-04-21T21:50:55.0628197Z  "${DOCKER_IMAGE}" \ 2025-04-21T21:50:55.0628460Z  ${DOCKER_SHELL_CMD} 2025-04-21T21:50:55.0628718Z ) 2025-04-21T21:50:55.0628999Z # Propagate download.pytorch.org IP to container 2025-04-21T21:50:55.0629648Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-04-21T21:50:55.0630318Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-04-21T21:50:55.0630711Z  2025-04-21T21:50:55.0630971Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-04-21T21:50:55.0631530Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-04-21T21:50:55.0632063Z fi 2025-04-21T21:50:55.0632269Z  2025-04-21T21:50:55.0632735Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-04-21T21:50:55.0641193Z shell: /usr/bin/bash -e {0} 2025-04-21T21:50:55.0641447Z env: 2025-04-21T21:50:55.0641655Z GIT_DEFAULT_BRANCH: main 2025-04-21T21:50:55.0642023Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:50:55.0642461Z BUILD_ENVIRONMENT: linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-21T21:50:55.0642822Z PR_NUMBER: 2025-04-21T21:50:55.0643056Z GITHUB_REPOSITORY: pytorch/pytorch 2025-04-21T21:50:55.0643346Z GITHUB_WORKFLOW: inductor 2025-04-21T21:50:55.0643599Z GITHUB_JOB: test 2025-04-21T21:50:55.0643935Z GITHUB_RUN_ID: 14580756816 2025-04-21T21:50:55.0644194Z GITHUB_RUN_NUMBER: 128203 2025-04-21T21:50:55.0644450Z GITHUB_RUN_ATTEMPT: 1 2025-04-21T21:50:55.0644683Z JOB_ID: 40900324323 2025-04-21T21:50:55.0645124Z JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:50:55.0645620Z BRANCH: 2025-04-21T21:50:55.0645863Z SHA1: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:50:55.0646224Z BASE_SHA: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:50:55.0646570Z TEST_CONFIG: inductor_torchbench 2025-04-21T21:50:55.0646843Z SHARD_NUMBER: 1 2025-04-21T21:50:55.0647062Z NUM_TEST_SHARDS: 2 2025-04-21T21:50:55.0647288Z REENABLED_ISSUES: 2025-04-21T21:50:55.0647530Z CONTINUE_THROUGH_ERROR: False 2025-04-21T21:50:55.0647802Z VERBOSE_TEST_LOGS: False 2025-04-21T21:50:55.0648051Z TEST_SHOWLOCALS: False 2025-04-21T21:50:55.0648299Z NO_TEST_TIMEOUT: False 2025-04-21T21:50:55.0648535Z NO_TD: False 2025-04-21T21:50:55.0648754Z TD_DISTRIBUTED: False 2025-04-21T21:50:55.0649049Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-04-21T21:50:55.0649407Z SCCACHE_REGION: us-east-1 2025-04-21T21:50:55.0649652Z SHM_SIZE: 2g 2025-04-21T21:50:55.0650439Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:50:55.0651288Z XLA_CUDA: 2025-04-21T21:50:55.0651633Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-04-21T21:50:55.0652065Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-04-21T21:50:55.0652380Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-04-21T21:50:55.0652757Z DASHBOARD_TAG: 2025-04-21T21:50:55.0653144Z HUGGING_FACE_HUB_TOKEN: *** 2025-04-21T21:50:55.0653534Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-04-21T21:50:55.0653823Z IS_A100_RUNNER: 0 2025-04-21T21:50:55.0654269Z ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-21T21:50:55.0654766Z ##[endgroup] 2025-04-21T21:50:55.0694069Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2025-04-21T21:50:55.0694759Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *onnx* ]] 2025-04-21T21:50:55.0695276Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-04-21T21:50:55.0698998Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-04-21T21:50:55.0722942Z + TOTAL_AVAILABLE_MEMORY_IN_GB='61.094 ' 2025-04-21T21:50:55.0723268Z + TOTAL_MEMORY_WITH_SWAP=64 2025-04-21T21:50:55.0723612Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]] 2025-04-21T21:50:55.0723997Z + SHM_OPTS=--shm-size=2g 2025-04-21T21:50:55.0724268Z + JENKINS_USER='--user jenkins' 2025-04-21T21:50:55.0724550Z + DOCKER_SHELL_CMD= 2025-04-21T21:50:55.0733768Z +++ nproc --ignore=2 2025-04-21T21:50:55.0751042Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --memory=61g --memory-swap=64g --env-file=/tmp/github_env_14580756816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-21T21:51:02.1111871Z + container_name=a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-21T21:51:02.1115278Z + grep download.pytorch.org /etc/hosts 2025-04-21T21:51:02.1115990Z + docker exec -i a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee sudo bash -c '/bin/cat >> /etc/hosts' 2025-04-21T21:51:02.2765350Z + echo DOCKER_CONTAINER_ID=a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-21T21:51:02.2765996Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]] 2025-04-21T21:51:02.2770108Z ++ echo dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl 2025-04-21T21:51:02.2773593Z + docker exec -t a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee sh -c 'python3 -m pip install dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-04-21T21:51:02.6930740Z Processing ./dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl (from torch==2.8.0a0+gitb68c0ef) 2025-04-21T21:51:03.0299417Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.16.1) 2025-04-21T21:51:03.0302328Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (4.13.2) 2025-04-21T21:51:03.0627925Z Collecting sympy>=1.13.3 (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) 2025-04-21T21:51:03.0641045Z Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB) 2025-04-21T21:51:03.0658628Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (2.8.8) 2025-04-21T21:51:03.0662075Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.1.6) 2025-04-21T21:51:03.0665594Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (2024.10.0) 2025-04-21T21:51:03.0680487Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.3.0) 2025-04-21T21:51:03.0699667Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (1.22.4) 2025-04-21T21:51:03.0707463Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (1.3.0) 2025-04-21T21:51:03.1089968Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.0.2) 2025-04-21T21:51:03.1173829Z Using cached sympy-1.13.3-py3-none-any.whl (6.2 MB) 2025-04-21T21:51:03.7558544Z Installing collected packages: sympy, torch 2025-04-21T21:51:03.7558914Z Attempting uninstall: sympy 2025-04-21T21:51:03.7570375Z Found existing installation: sympy 1.13.1 2025-04-21T21:51:03.9631940Z Uninstalling sympy-1.13.1: 2025-04-21T21:51:05.2752850Z Successfully uninstalled sympy-1.13.1 2025-04-21T21:51:19.8971012Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-04-21T21:51:19.8972061Z timm 1.0.14 requires torchvision, which is not installed. 2025-04-21T21:51:19.8972627Z Successfully installed sympy-1.13.3 torch-2.8.0a0+gitb68c0ef 2025-04-21T21:51:19.9842371Z + export TERM=vt100 2025-04-21T21:51:19.9842619Z + TERM=vt100 2025-04-21T21:51:19.9846486Z ++ dirname .ci/pytorch/test.sh 2025-04-21T21:51:19.9857025Z + source .ci/pytorch/common.sh 2025-04-21T21:51:19.9861161Z +++ dirname .ci/pytorch/common.sh 2025-04-21T21:51:19.9871479Z ++ source .ci/pytorch/common_utils.sh 2025-04-21T21:51:19.9873545Z +++ declare -f -t trap_add 2025-04-21T21:51:19.9880628Z ++ set -ex -o pipefail 2025-04-21T21:51:19.9881077Z ++ [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-04-21T21:51:19.9881589Z ++ BUILD_TEST_LIBTORCH=0 2025-04-21T21:51:19.9882041Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *rocm* ]] 2025-04-21T21:51:19.9882497Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *s390x* ]] 2025-04-21T21:51:19.9882863Z + [[ -d /var/lib/jenkins/workspace ]] 2025-04-21T21:51:19.9886586Z ++ stat -c %u /var/lib/jenkins/workspace 2025-04-21T21:51:19.9908189Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-04-21T21:51:19.9908497Z + trap_add cleanup_workspace EXIT 2025-04-21T21:51:19.9908787Z + trap_add_cmd=cleanup_workspace 2025-04-21T21:51:19.9909078Z + shift 2025-04-21T21:51:19.9909316Z + for trap_add_name in "$@" 2025-04-21T21:51:19.9916524Z +++ trap -p EXIT 2025-04-21T21:51:19.9920240Z ++ eval 'extract_trap_cmd ' 2025-04-21T21:51:19.9920498Z +++ extract_trap_cmd 2025-04-21T21:51:19.9920724Z +++ printf '%s\n' '' 2025-04-21T21:51:19.9920974Z ++ printf '%s\n' cleanup_workspace 2025-04-21T21:51:19.9924256Z + trap -- ' 2025-04-21T21:51:19.9924589Z cleanup_workspace' EXIT 2025-04-21T21:51:19.9924917Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-04-21T21:51:20.7859294Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-04-21T21:51:20.7882360Z + echo 'Environment variables:' 2025-04-21T21:51:20.7882816Z Environment variables: 2025-04-21T21:51:20.7883062Z + env 2025-04-21T21:51:20.7896236Z NV_LIBCUBLAS_VERSION=12.6.4.1-1 2025-04-21T21:51:20.7896759Z NVIDIA_VISIBLE_DEVICES=all 2025-04-21T21:51:20.7897162Z NV_NVML_DEV_VERSION=12.6.77-1 2025-04-21T21:51:20.7897858Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-04-21T21:51:20.7898401Z CONTINUE_THROUGH_ERROR=False 2025-04-21T21:51:20.7898824Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6 2025-04-21T21:51:20.7899333Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1 2025-04-21T21:51:20.7899850Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-21T21:51:20.7900351Z HOSTNAME=a89da563ed2a 2025-04-21T21:51:20.7900992Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.7901603Z GITHUB_ACTION=__self 2025-04-21T21:51:20.7901874Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-04-21T21:51:20.7906984Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 2025-04-21T21:51:20.7912007Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1 2025-04-21T21:51:20.7912615Z NV_NVTX_VERSION=12.6.77-1 2025-04-21T21:51:20.7912902Z GITHUB_RUN_NUMBER=128203 2025-04-21T21:51:20.7913161Z TEST_CONFIG=inductor_torchbench 2025-04-21T21:51:20.7913451Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-04-21T21:51:20.7913776Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-04-21T21:51:20.7914066Z IS_A100_RUNNER=0 2025-04-21T21:51:20.7914308Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1 2025-04-21T21:51:20.7914609Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1 2025-04-21T21:51:20.7915113Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-04-21T21:51:20.7915391Z NV_LIBNPP_VERSION=12.3.1.54-1 2025-04-21T21:51:20.7915688Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2025-04-21T21:51:20.7916044Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-04-21T21:51:20.7916384Z GITHUB_REF_TYPE=tag 2025-04-21T21:51:20.7916623Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-04-21T21:51:20.7916888Z NCCL_VERSION=2.23.4-1 2025-04-21T21:51:20.7917165Z BASE_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.7917486Z XLA_CUDA= 2025-04-21T21:51:20.7917713Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-04-21T21:51:20.7918111Z HUGGING_FACE_HUB_TOKEN=*** 2025-04-21T21:51:20.7918620Z *** 2025-04-21T21:51:20.7918834Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-04-21T21:51:20.7919142Z GITHUB_REPOSITORY_ID=65600975 2025-04-21T21:51:20.7919412Z GITHUB_ACTIONS=true 2025-04-21T21:51:20.7919653Z NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:51:20.7919972Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1 2025-04-21T21:51:20.7920328Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1 2025-04-21T21:51:20.7920667Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.7921004Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-04-21T21:51:20.7921354Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.7922110Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/148893 2025-04-21T21:51:20.7922634Z UCC_HOME=/usr 2025-04-21T21:51:20.7922868Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1 2025-04-21T21:51:20.7923158Z VERBOSE_TEST_LOGS=False 2025-04-21T21:51:20.7923407Z NVIDIA_PRODUCT_NAME=CUDA 2025-04-21T21:51:20.7923706Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6 2025-04-21T21:51:20.7924073Z GITHUB_REF=refs/tags/ciflow/inductor/148893 2025-04-21T21:51:20.7924387Z NV_CUDA_CUDART_VERSION=12.6.77-1 2025-04-21T21:51:20.7924654Z SHARD_NUMBER=1 2025-04-21T21:51:20.7924882Z GITHUB_REF_PROTECTED=false 2025-04-21T21:51:20.7925128Z HOME=/var/lib/jenkins 2025-04-21T21:51:20.7925400Z GITHUB_API_URL=https://api.github.com 2025-04-21T21:51:20.7925720Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-04-21T21:51:20.7926060Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-04-21T21:51:20.7926385Z CUDA_VERSION=12.6.3 2025-04-21T21:51:20.7926649Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1 2025-04-21T21:51:20.7926962Z USE_SYSTEM_NCCL=1 2025-04-21T21:51:20.7927173Z NUM_TEST_SHARDS=2 2025-04-21T21:51:20.7927391Z UCX_HOME=/usr 2025-04-21T21:51:20.7927722Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1 2025-04-21T21:51:20.7928440Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.7929312Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:51:20.7930110Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.7930874Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-04-21T21:51:20.7931343Z GITHUB_EVENT_NAME=push 2025-04-21T21:51:20.7931590Z DASHBOARD_TAG= 2025-04-21T21:51:20.7931814Z GITHUB_RUN_ID=14580756816 2025-04-21T21:51:20.7932102Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1 2025-04-21T21:51:20.7932462Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6 2025-04-21T21:51:20.7933108Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.7933753Z GITHUB_ACTOR=pytorch-bot[bot] 2025-04-21T21:51:20.7934132Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1 2025-04-21T21:51:20.7934402Z PR_NUMBER= 2025-04-21T21:51:20.7934612Z GITHUB_RUN_ATTEMPT=1 2025-04-21T21:51:20.7934853Z ANACONDA_PYTHON_VERSION=3.10 2025-04-21T21:51:20.7935174Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-04-21T21:51:20.7935506Z TERM=vt100 2025-04-21T21:51:20.7935723Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1 2025-04-21T21:51:20.7936014Z INSTALLED_VISION=yes 2025-04-21T21:51:20.7936243Z BRANCH= 2025-04-21T21:51:20.7936445Z SCCACHE_REGION=us-east-1 2025-04-21T21:51:20.7936707Z OPENSSL_ROOT_DIR=/opt/openssl 2025-04-21T21:51:20.7936993Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-04-21T21:51:20.7937293Z CUDA_PATH=/usr/local/cuda 2025-04-21T21:51:20.7937890Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-04-21T21:51:20.7938450Z GITHUB_SERVER_URL=https://github.com 2025-04-21T21:51:20.7938805Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-04-21T21:51:20.7939178Z REENABLED_ISSUES= 2025-04-21T21:51:20.7939391Z SHLVL=1 2025-04-21T21:51:20.7939584Z MAX_JOBS=14 2025-04-21T21:51:20.7939797Z NV_CUDA_LIB_VERSION=12.6.3-1 2025-04-21T21:51:20.7940050Z NVARCH=x86_64 2025-04-21T21:51:20.7940269Z GITHUB_ACTOR_ID=54816060 2025-04-21T21:51:20.7940588Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.7940976Z GITHUB_REF_NAME=ciflow/inductor/148893 2025-04-21T21:51:20.7941395Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-04-21T21:51:20.7941795Z GITHUB_JOB=test 2025-04-21T21:51:20.7942046Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6 2025-04-21T21:51:20.7942435Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-04-21T21:51:20.7942886Z NO_TEST_TIMEOUT=False 2025-04-21T21:51:20.7943117Z TD_DISTRIBUTED=False 2025-04-21T21:51:20.7943372Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1 2025-04-21T21:51:20.7943682Z GITHUB_REPOSITORY=pytorch/pytorch 2025-04-21T21:51:20.7943969Z NV_NVPROF_VERSION=12.6.80-1 2025-04-21T21:51:20.7944231Z GITHUB_RETENTION_DAYS=90 2025-04-21T21:51:20.7944483Z OPENSSL_DIR=/opt/openssl 2025-04-21T21:51:20.7944729Z GITHUB_ACTION_REPOSITORY= 2025-04-21T21:51:20.7945463Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-04-21T21:51:20.7946213Z GITHUB_BASE_REF= 2025-04-21T21:51:20.7946650Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-21T21:51:20.7947172Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-04-21T21:51:20.7947439Z CI=true 2025-04-21T21:51:20.7947655Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1 2025-04-21T21:51:20.7947947Z GITHUB_REPOSITORY_OWNER=pytorch 2025-04-21T21:51:20.7948215Z JOB_ID=40900324323 2025-04-21T21:51:20.7948439Z INSTALLED_PROTOBUF=yes 2025-04-21T21:51:20.7948669Z GITHUB_HEAD_REF= 2025-04-21T21:51:20.7948890Z GITHUB_ACTION_REF= 2025-04-21T21:51:20.7949169Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-04-21T21:51:20.7949505Z TEST_SHOWLOCALS=False 2025-04-21T21:51:20.7949749Z GITHUB_WORKFLOW=inductor 2025-04-21T21:51:20.7950013Z DEBIAN_FRONTEND=noninteractive 2025-04-21T21:51:20.7950595Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.7951193Z NO_TD=False 2025-04-21T21:51:20.7951417Z SKIP_SCCACHE_INITIALIZATION=1 2025-04-21T21:51:20.7951704Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-04-21T21:51:20.7951999Z _=/usr/bin/env 2025-04-21T21:51:20.7952301Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-04-21T21:51:20.8084650Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-04-21T21:51:20.8085363Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-04-21T21:51:20.8085939Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-04-21T21:51:20.8086792Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-04-21T21:51:20.8087384Z + BUILD_DIR=build 2025-04-21T21:51:20.8087702Z + BUILD_RENAMED_DIR=build_renamed 2025-04-21T21:51:20.8088032Z + BUILD_BIN_DIR=build/bin 2025-04-21T21:51:20.8088282Z + SHARD_NUMBER=1 2025-04-21T21:51:20.8088498Z + NUM_TEST_SHARDS=2 2025-04-21T21:51:20.8088750Z + export TORCH_SERIALIZATION_DEBUG=1 2025-04-21T21:51:20.8089058Z + TORCH_SERIALIZATION_DEBUG=1 2025-04-21T21:51:20.8089332Z + export VALGRIND=ON 2025-04-21T21:51:20.8089564Z + VALGRIND=ON 2025-04-21T21:51:20.8089852Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *clang9* ]] 2025-04-21T21:51:20.8090269Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]] 2025-04-21T21:51:20.8090673Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *s390x* ]] 2025-04-21T21:51:20.8091014Z + [[ 0 == \1 ]] 2025-04-21T21:51:20.8091228Z + [[ False == \1 ]] 2025-04-21T21:51:20.8091519Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *bazel* ]] 2025-04-21T21:51:20.8091933Z ++ realpath build/custom_test_artifacts 2025-04-21T21:51:20.8105133Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-04-21T21:51:20.8105772Z + [[ -n '' ]] 2025-04-21T21:51:20.8106019Z + echo 'Environment variables' 2025-04-21T21:51:20.8106382Z Environment variables 2025-04-21T21:51:20.8106614Z + env 2025-04-21T21:51:20.8114708Z NV_LIBCUBLAS_VERSION=12.6.4.1-1 2025-04-21T21:51:20.8115180Z NVIDIA_VISIBLE_DEVICES=all 2025-04-21T21:51:20.8115559Z NV_NVML_DEV_VERSION=12.6.77-1 2025-04-21T21:51:20.8116071Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-04-21T21:51:20.8116650Z CONTINUE_THROUGH_ERROR=False 2025-04-21T21:51:20.8117099Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6 2025-04-21T21:51:20.8117899Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1 2025-04-21T21:51:20.8118428Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-21T21:51:20.8118975Z HOSTNAME=a89da563ed2a 2025-04-21T21:51:20.8119673Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.8120389Z GITHUB_ACTION=__self 2025-04-21T21:51:20.8120703Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-04-21T21:51:20.8125996Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 2025-04-21T21:51:20.8131831Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1 2025-04-21T21:51:20.8132195Z NV_NVTX_VERSION=12.6.77-1 2025-04-21T21:51:20.8132448Z GITHUB_RUN_NUMBER=128203 2025-04-21T21:51:20.8132715Z TEST_CONFIG=inductor_torchbench 2025-04-21T21:51:20.8133010Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-04-21T21:51:20.8133321Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-04-21T21:51:20.8133664Z IS_A100_RUNNER=0 2025-04-21T21:51:20.8133908Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1 2025-04-21T21:51:20.8134394Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1 2025-04-21T21:51:20.8134849Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-04-21T21:51:20.8135136Z NV_LIBNPP_VERSION=12.3.1.54-1 2025-04-21T21:51:20.8135423Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2025-04-21T21:51:20.8135783Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2025-04-21T21:51:20.8136123Z GITHUB_REF_TYPE=tag 2025-04-21T21:51:20.8136357Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-04-21T21:51:20.8136621Z NCCL_VERSION=2.23.4-1 2025-04-21T21:51:20.8136908Z BASE_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.8137220Z XLA_CUDA= 2025-04-21T21:51:20.8137444Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-04-21T21:51:20.8138013Z HUGGING_FACE_HUB_TOKEN=*** 2025-04-21T21:51:20.8138302Z *** 2025-04-21T21:51:20.8138512Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2025-04-21T21:51:20.8138807Z GITHUB_REPOSITORY_ID=65600975 2025-04-21T21:51:20.8139074Z GITHUB_ACTIONS=true 2025-04-21T21:51:20.8139311Z NVIDIA_DRIVER_CAPABILITIES=all 2025-04-21T21:51:20.8139628Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1 2025-04-21T21:51:20.8139986Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1 2025-04-21T21:51:20.8140320Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.8140657Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2025-04-21T21:51:20.8141003Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.8141592Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/148893 2025-04-21T21:51:20.8142116Z UCC_HOME=/usr 2025-04-21T21:51:20.8142344Z TORCH_SERIALIZATION_DEBUG=1 2025-04-21T21:51:20.8142617Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1 2025-04-21T21:51:20.8142896Z VERBOSE_TEST_LOGS=False 2025-04-21T21:51:20.8143144Z NVIDIA_PRODUCT_NAME=CUDA 2025-04-21T21:51:20.8143535Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6 2025-04-21T21:51:20.8143887Z GITHUB_REF=refs/tags/ciflow/inductor/148893 2025-04-21T21:51:20.8144207Z NV_CUDA_CUDART_VERSION=12.6.77-1 2025-04-21T21:51:20.8144477Z SHARD_NUMBER=1 2025-04-21T21:51:20.8144696Z GITHUB_REF_PROTECTED=false 2025-04-21T21:51:20.8144952Z HOME=/var/lib/jenkins 2025-04-21T21:51:20.8145221Z GITHUB_API_URL=https://api.github.com 2025-04-21T21:51:20.8145533Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-04-21T21:51:20.8145868Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2025-04-21T21:51:20.8146202Z CUDA_VERSION=12.6.3 2025-04-21T21:51:20.8146466Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1 2025-04-21T21:51:20.8146775Z USE_SYSTEM_NCCL=1 2025-04-21T21:51:20.8146997Z NUM_TEST_SHARDS=2 2025-04-21T21:51:20.8147212Z UCX_HOME=/usr 2025-04-21T21:51:20.8147537Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1 2025-04-21T21:51:20.8148259Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.8149157Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-21T21:51:20.8149954Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.8150727Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-04-21T21:51:20.8151208Z GITHUB_EVENT_NAME=push 2025-04-21T21:51:20.8151449Z DASHBOARD_TAG= 2025-04-21T21:51:20.8151663Z GITHUB_RUN_ID=14580756816 2025-04-21T21:51:20.8151960Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1 2025-04-21T21:51:20.8152322Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6 2025-04-21T21:51:20.8152970Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.8153630Z GITHUB_ACTOR=pytorch-bot[bot] 2025-04-21T21:51:20.8153915Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1 2025-04-21T21:51:20.8154185Z PR_NUMBER= 2025-04-21T21:51:20.8154395Z GITHUB_RUN_ATTEMPT=1 2025-04-21T21:51:20.8165863Z VALGRIND=ON 2025-04-21T21:51:20.8166114Z ANACONDA_PYTHON_VERSION=3.10 2025-04-21T21:51:20.8166458Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-04-21T21:51:20.8166797Z TERM=vt100 2025-04-21T21:51:20.8167158Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1 2025-04-21T21:51:20.8167465Z INSTALLED_VISION=yes 2025-04-21T21:51:20.8167693Z BRANCH= 2025-04-21T21:51:20.8167902Z SCCACHE_REGION=us-east-1 2025-04-21T21:51:20.8168170Z OPENSSL_ROOT_DIR=/opt/openssl 2025-04-21T21:51:20.8168455Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2025-04-21T21:51:20.8168765Z CUDA_PATH=/usr/local/cuda 2025-04-21T21:51:20.8169266Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-04-21T21:51:20.8169826Z GITHUB_SERVER_URL=https://github.com 2025-04-21T21:51:20.8170179Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2025-04-21T21:51:20.8170517Z REENABLED_ISSUES= 2025-04-21T21:51:20.8170737Z SHLVL=1 2025-04-21T21:51:20.8170936Z MAX_JOBS=14 2025-04-21T21:51:20.8171160Z NV_CUDA_LIB_VERSION=12.6.3-1 2025-04-21T21:51:20.8171412Z NVARCH=x86_64 2025-04-21T21:51:20.8171634Z GITHUB_ACTOR_ID=54816060 2025-04-21T21:51:20.8171972Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-21T21:51:20.8172359Z GITHUB_REF_NAME=ciflow/inductor/148893 2025-04-21T21:51:20.8172787Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-04-21T21:51:20.8173197Z GITHUB_JOB=test 2025-04-21T21:51:20.8173454Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6 2025-04-21T21:51:20.8173843Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2025-04-21T21:51:20.8174211Z NO_TEST_TIMEOUT=False 2025-04-21T21:51:20.8174454Z TD_DISTRIBUTED=False 2025-04-21T21:51:20.8174709Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1 2025-04-21T21:51:20.8175021Z GITHUB_REPOSITORY=pytorch/pytorch 2025-04-21T21:51:20.8175303Z NV_NVPROF_VERSION=12.6.80-1 2025-04-21T21:51:20.8175564Z GITHUB_RETENTION_DAYS=90 2025-04-21T21:51:20.8175933Z OPENSSL_DIR=/opt/openssl 2025-04-21T21:51:20.8176192Z GITHUB_ACTION_REPOSITORY= 2025-04-21T21:51:20.8176921Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-04-21T21:51:20.8177856Z GITHUB_BASE_REF= 2025-04-21T21:51:20.8178304Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-21T21:51:20.8178852Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2025-04-21T21:51:20.8179138Z CI=true 2025-04-21T21:51:20.8179354Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1 2025-04-21T21:51:20.8179654Z GITHUB_REPOSITORY_OWNER=pytorch 2025-04-21T21:51:20.8179916Z JOB_ID=40900324323 2025-04-21T21:51:20.8180146Z INSTALLED_PROTOBUF=yes 2025-04-21T21:51:20.8180385Z GITHUB_HEAD_REF= 2025-04-21T21:51:20.8180601Z GITHUB_ACTION_REF= 2025-04-21T21:51:20.8180885Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-04-21T21:51:20.8181226Z TEST_SHOWLOCALS=False 2025-04-21T21:51:20.8181477Z GITHUB_WORKFLOW=inductor 2025-04-21T21:51:20.8181738Z DEBIAN_FRONTEND=noninteractive 2025-04-21T21:51:20.8182336Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_a262bbad-2833-4220-9e40-8c5c5f40b041 2025-04-21T21:51:20.8182954Z NO_TD=False 2025-04-21T21:51:20.8183177Z SKIP_SCCACHE_INITIALIZATION=1 2025-04-21T21:51:20.8183476Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-04-21T21:51:20.8183772Z _=/usr/bin/env 2025-04-21T21:51:20.8183997Z + echo 'Testing pytorch' 2025-04-21T21:51:20.8184245Z Testing pytorch 2025-04-21T21:51:20.8184478Z + export LANG=C.UTF-8 2025-04-21T21:51:20.8184713Z + LANG=C.UTF-8 2025-04-21T21:51:20.8184925Z + PR_NUMBER= 2025-04-21T21:51:20.8185167Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2025-04-21T21:51:20.8185518Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-04-21T21:51:20.8185867Z + [[ inductor_torchbench == \s\l\o\w ]] 2025-04-21T21:51:20.8186272Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2025-04-21T21:51:20.8186722Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *cuda* ]] 2025-04-21T21:51:20.8187103Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-04-21T21:51:20.8187441Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-04-21T21:51:20.8187866Z + [[ inductor_torchbench == *crossref* ]] 2025-04-21T21:51:20.8188226Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-04-21T21:51:20.8188631Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]] 2025-04-21T21:51:20.8189043Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]] 2025-04-21T21:51:20.8189410Z + pip_install --user ninja==1.10.2 2025-04-21T21:51:20.8189788Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T21:51:20.8190265Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2025-04-21T21:51:21.2845521Z Collecting ninja==1.10.2 2025-04-21T21:51:21.3027997Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-04-21T21:51:21.3149060Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-04-21T21:51:21.9137833Z Installing collected packages: ninja 2025-04-21T21:51:21.9220443Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2025-04-21T21:51:21.9221563Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2025-04-21T21:51:21.9277905Z Successfully installed ninja-1.10.2 2025-04-21T21:51:22.0068132Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-04-21T21:51:22.0069678Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-04-21T21:51:22.0070863Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]] 2025-04-21T21:51:22.0071223Z + install_tlparse 2025-04-21T21:51:22.0071466Z + pip_install --user tlparse==0.3.30 2025-04-21T21:51:22.0071847Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T21:51:22.0072337Z + python3 -m pip install --progress-bar off --user tlparse==0.3.30 2025-04-21T21:51:22.4094505Z Collecting tlparse==0.3.30 2025-04-21T21:51:22.4256959Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB) 2025-04-21T21:51:22.4394039Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB) 2025-04-21T21:51:23.0602824Z Installing collected packages: tlparse 2025-04-21T21:51:23.0954639Z Successfully installed tlparse-0.3.30 2025-04-21T21:51:23.1757602Z ++ python -m site --user-base 2025-04-21T21:51:23.1950054Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-04-21T21:51:23.1951533Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *asan* ]] 2025-04-21T21:51:23.1952070Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-debug* ]] 2025-04-21T21:51:23.1952510Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]] 2025-04-21T21:51:23.1953089Z + echo 'We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass' 2025-04-21T21:51:23.1953980Z We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass 2025-04-21T21:51:23.1954979Z + cd test 2025-04-21T21:51:23.1955326Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-04-21T21:51:24.9334294Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-04-21T21:51:24.9334704Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-04-21T21:51:24.9341646Z + DYNAMO_BENCHMARK_FLAGS=() 2025-04-21T21:51:24.9342165Z + [[ inductor_torchbench == *pr_time_benchmarks* ]] 2025-04-21T21:51:24.9342564Z + [[ inductor_torchbench == *dynamo_eager* ]] 2025-04-21T21:51:24.9342976Z + [[ inductor_torchbench == *aot_eager* ]] 2025-04-21T21:51:24.9343331Z + [[ inductor_torchbench == *aot_inductor* ]] 2025-04-21T21:51:24.9344391Z + [[ inductor_torchbench == *max_autotune_inductor* ]] 2025-04-21T21:51:24.9344854Z + [[ inductor_torchbench == *inductor* ]] 2025-04-21T21:51:24.9345194Z + [[ inductor_torchbench != *perf* ]] 2025-04-21T21:51:24.9345595Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-04-21T21:51:24.9345939Z + [[ inductor_torchbench == *dynamic* ]] 2025-04-21T21:51:24.9346290Z + [[ inductor_torchbench == *cpu* ]] 2025-04-21T21:51:24.9346648Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-04-21T21:51:24.9375673Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]] 2025-04-21T21:51:24.9376229Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-bazel-* ]] 2025-04-21T21:51:24.9378703Z + cd test 2025-04-21T21:51:24.9379055Z + python -c 'import torch; print(torch.__config__.show())' 2025-04-21T21:51:26.4585947Z PyTorch built with: 2025-04-21T21:51:26.4586346Z - GCC 9.4 2025-04-21T21:51:26.4586632Z - C++ Version: 201703 2025-04-21T21:51:26.4587369Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-04-21T21:51:26.4588068Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-04-21T21:51:26.4588506Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-04-21T21:51:26.4588836Z - LAPACK is enabled (usually provided by MKL) 2025-04-21T21:51:26.4589155Z - NNPACK is enabled 2025-04-21T21:51:26.4589409Z - CPU capability usage: AVX2 2025-04-21T21:51:26.4589677Z - CUDA Runtime 12.6 2025-04-21T21:51:26.4590020Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2025-04-21T21:51:26.4590391Z - CuDNN 90.5.1 2025-04-21T21:51:26.4590611Z - Magma 2.6.1 2025-04-21T21:51:26.4595152Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2025-04-21T21:51:26.4600006Z 2025-04-21T21:51:26.8693005Z + cd test 2025-04-21T21:51:26.8693505Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-04-21T21:51:28.2807117Z ATen/Parallel: 2025-04-21T21:51:28.2807573Z at::get_num_threads() : 8 2025-04-21T21:51:28.2807961Z at::get_num_interop_threads() : 16 2025-04-21T21:51:28.2808278Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-04-21T21:51:28.2808569Z omp_get_max_threads() : 8 2025-04-21T21:51:28.2809112Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2025-04-21T21:51:28.2809832Z mkl_get_max_threads() : 8 2025-04-21T21:51:28.2810333Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-04-21T21:51:28.2810865Z std::thread::hardware_concurrency() : 16 2025-04-21T21:51:28.2811177Z Environment variables: 2025-04-21T21:51:28.2811435Z OMP_NUM_THREADS : [not set] 2025-04-21T21:51:28.2811721Z MKL_NUM_THREADS : [not set] 2025-04-21T21:51:28.2811984Z ATen parallel backend: OpenMP 2025-04-21T21:51:28.2812171Z 2025-04-21T21:51:28.5855866Z + [[ inductor_torchbench == *numpy_2* ]] 2025-04-21T21:51:28.5856486Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]] 2025-04-21T21:51:28.5857448Z + [[ inductor_torchbench == *backward* ]] 2025-04-21T21:51:28.5857925Z + [[ inductor_torchbench == *xla* ]] 2025-04-21T21:51:28.5858231Z + [[ inductor_torchbench == *executorch* ]] 2025-04-21T21:51:28.5858572Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2025-04-21T21:51:28.5858970Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]] 2025-04-21T21:51:28.5859343Z + [[ inductor_torchbench == distributed ]] 2025-04-21T21:51:28.5859694Z + [[ inductor_torchbench == *operator_benchmark* ]] 2025-04-21T21:51:28.5860081Z + [[ inductor_torchbench == *inductor_distributed* ]] 2025-04-21T21:51:28.5860458Z + [[ inductor_torchbench == *inductor-halide* ]] 2025-04-21T21:51:28.5860826Z + [[ inductor_torchbench == *inductor-triton-cpu* ]] 2025-04-21T21:51:28.5861242Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2025-04-21T21:51:28.5861617Z + [[ inductor_torchbench == *huggingface* ]] 2025-04-21T21:51:28.5861935Z + [[ inductor_torchbench == *timm* ]] 2025-04-21T21:51:28.5862250Z + [[ inductor_torchbench == cachebench ]] 2025-04-21T21:51:28.5862598Z + [[ inductor_torchbench == verify_cachebench ]] 2025-04-21T21:51:28.5863052Z + [[ inductor_torchbench == *torchbench* ]] 2025-04-21T21:51:28.5863490Z + [[ inductor_torchbench == *cpu* ]] 2025-04-21T21:51:28.5863888Z + install_torchaudio cuda 2025-04-21T21:51:28.5864198Z + local commit 2025-04-21T21:51:28.5864434Z ++ get_pinned_commit audio 2025-04-21T21:51:28.5864713Z ++ cat .github/ci_commit_pins/audio.txt 2025-04-21T21:51:28.5881930Z + commit=bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:28.5882398Z + [[ cuda == \c\u\d\a ]] 2025-04-21T21:51:28.5882737Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2025-04-21T21:51:28.5883429Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:28.5884326Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T21:51:28.5885112Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:28.9383921Z Collecting git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:28.9387913Z Cloning https://github.com/pytorch/audio.git (to revision bccaa454a54c3c648697cc2f46a4fb0500b1f01b) to /tmp/pip-req-build-u2t606xn 2025-04-21T21:51:28.9419738Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-u2t606xn 2025-04-21T21:51:29.7212517Z Running command git rev-parse -q --verify 'sha^bccaa454a54c3c648697cc2f46a4fb0500b1f01b' 2025-04-21T21:51:29.7241105Z Running command git fetch -q https://github.com/pytorch/audio.git bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:30.1410931Z Running command git checkout -q bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:30.3143514Z Resolved https://github.com/pytorch/audio.git to commit bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T21:51:30.3144246Z Running command git submodule update --init --recursive -q 2025-04-21T21:51:32.9872316Z Preparing metadata (setup.py) ... [?25l- done 2025-04-21T21:51:32.9903275Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.6.0a0+bccaa45) (2.8.0a0+gitb68c0ef) 2025-04-21T21:51:32.9927300Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (3.16.1) 2025-04-21T21:51:32.9933466Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (4.13.2) 2025-04-21T21:51:32.9934775Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (1.13.3) 2025-04-21T21:51:32.9938092Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (2.8.8) 2025-04-21T21:51:32.9941444Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (3.1.6) 2025-04-21T21:51:32.9943370Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (2024.10.0) 2025-04-21T21:51:32.9958588Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchaudio==2.6.0a0+bccaa45) (1.3.0) 2025-04-21T21:51:33.0440829Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.6.0a0+bccaa45) (3.0.2) 2025-04-21T21:51:33.0506865Z Building wheels for collected packages: torchaudio 2025-04-21T21:54:11.4373451Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2025-04-21T21:54:11.4409945Z [?25h Created wheel for torchaudio: filename=torchaudio-2.6.0a0+bccaa45-cp310-cp310-linux_x86_64.whl size=2396807 sha256=2f5242ab7931325c94f680039bfdd7bd638e7b67f32a5ebb468f1f95c8c28b53 2025-04-21T21:54:11.4411142Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/a2/00/0f/924e514cbe6637d9ca623106f058b6c50bd5c3ca92b0f3facc 2025-04-21T21:54:11.4442690Z Successfully built torchaudio 2025-04-21T21:54:11.9722253Z Installing collected packages: torchaudio 2025-04-21T21:54:12.1878004Z Successfully installed torchaudio-2.6.0a0+bccaa45 2025-04-21T21:54:12.3804695Z + install_torchvision 2025-04-21T21:54:12.3805099Z + local orig_preload 2025-04-21T21:54:12.3805424Z + local commit 2025-04-21T21:54:12.3809557Z ++ get_pinned_commit vision 2025-04-21T21:54:12.3809974Z ++ cat .github/ci_commit_pins/vision.txt 2025-04-21T21:54:12.3836542Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:12.3836925Z + orig_preload= 2025-04-21T21:54:12.3837160Z + '[' -n '' ']' 2025-04-21T21:54:12.3837704Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:12.3838376Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T21:54:12.3839148Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:12.7361361Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:12.7365159Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-4vgrnber 2025-04-21T21:54:12.7408173Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-4vgrnber 2025-04-21T21:54:14.3395189Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2025-04-21T21:54:14.3426048Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:15.7922768Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:16.1899465Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T21:54:19.0783572Z Preparing metadata (setup.py) ... [?25l- \ done 2025-04-21T21:54:19.0826294Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2025-04-21T21:54:19.0830397Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.8.0a0+gitb68c0ef) 2025-04-21T21:54:19.0834599Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0) 2025-04-21T21:54:19.0903124Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1) 2025-04-21T21:54:19.0906869Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.13.2) 2025-04-21T21:54:19.0910676Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.3) 2025-04-21T21:54:19.0913402Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2025-04-21T21:54:19.0916332Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.6) 2025-04-21T21:54:19.0919154Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0) 2025-04-21T21:54:19.0933566Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2025-04-21T21:54:19.1416365Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2) 2025-04-21T21:54:19.1481172Z Building wheels for collected packages: torchvision 2025-04-21T21:55:38.8411523Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2025-04-21T21:55:38.8443799Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2076940 sha256=e186a87c5ebb512da8fc28457978b7c338fe279f0aaef39724e45be807bfb8c8 2025-04-21T21:55:38.8445285Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2025-04-21T21:55:38.8479844Z Successfully built torchvision 2025-04-21T21:55:39.3744716Z Installing collected packages: torchvision 2025-04-21T21:55:39.7921015Z Successfully installed torchvision-0.19.0a0+d23a6e1 2025-04-21T21:55:39.9380643Z + '[' -n '' ']' 2025-04-21T21:55:39.9380891Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2025-04-21T21:55:39.9381301Z + pip_install git+https://github.com/pytorch/ao.git 2025-04-21T21:55:39.9381774Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T21:55:39.9382328Z + python3 -m pip install --progress-bar off git+https://github.com/pytorch/ao.git 2025-04-21T21:55:40.2930268Z Collecting git+https://github.com/pytorch/ao.git 2025-04-21T21:55:40.2934914Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-5cmg94cm 2025-04-21T21:55:40.2965892Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-5cmg94cm 2025-04-21T21:55:41.4005093Z Resolved https://github.com/pytorch/ao.git to commit 4805efd03b1aea9b5b9fe84df5c5a7171b5a7f2c 2025-04-21T21:55:41.4006243Z Running command git submodule update --init --recursive -q 2025-04-21T21:55:48.8855136Z Preparing metadata (setup.py) ... [?25l- done 2025-04-21T21:55:48.8904585Z [?25hBuilding wheels for collected packages: torchao 2025-04-21T22:01:12.1202702Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - done 2025-04-21T22:01:12.1248633Z [?25h Created wheel for torchao: filename=torchao-0.11.0+git4805efd0-cp39-abi3-linux_x86_64.whl size=5609497 sha256=8430fad4f0c2396c331c6c07c4e16fd30d8f73ba4415fa3872af42121b21e7a9 2025-04-21T22:01:12.1251587Z Stored in directory: /tmp/pip-ephem-wheel-cache-p70alirm/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2025-04-21T22:01:12.1290834Z Successfully built torchao 2025-04-21T22:01:12.7217508Z Installing collected packages: torchao 2025-04-21T22:01:13.5097182Z Successfully installed torchao-0.11.0+git4805efd0 2025-04-21T22:01:13.9653882Z + id=0 2025-04-21T22:01:13.9654277Z + pip_install opencv-python==4.8.0.74 2025-04-21T22:01:13.9654822Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T22:01:13.9655316Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74 2025-04-21T22:01:14.4936106Z Collecting opencv-python==4.8.0.74 2025-04-21T22:01:14.5117577Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2025-04-21T22:01:14.5217042Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2025-04-21T22:01:14.5304953Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2025-04-21T22:01:15.5832701Z Installing collected packages: opencv-python 2025-04-21T22:01:16.4522628Z Successfully installed opencv-python-4.8.0.74 2025-04-21T22:01:16.5473697Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2025-04-21T22:01:16.5474238Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2025-04-21T22:01:16.5474703Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2025-04-21T22:01:16.5475076Z + checkout_install_torchbench 2025-04-21T22:01:16.5475345Z + local commit 2025-04-21T22:01:16.5477004Z ++ get_pinned_commit torchbench 2025-04-21T22:01:16.5477771Z ++ cat .github/ci_commit_pins/torchbench.txt 2025-04-21T22:01:16.5504729Z + commit=373ffb19dc470f4423a3176a4133f8f4b3cdb5bd 2025-04-21T22:01:16.5505181Z + git clone https://github.com/pytorch/benchmark torchbench 2025-04-21T22:01:16.5548094Z Cloning into 'torchbench'... 2025-04-21T22:01:16.7221127Z remote: Enumerating objects: 35746, done. 2025-04-21T22:01:16.7221959Z remote: Counting objects: 7% (1/14) 2025-04-21T22:01:16.7222678Z remote: Counting objects: 14% (2/14) 2025-04-21T22:01:16.7223381Z remote: Counting objects: 21% (3/14) 2025-04-21T22:01:16.7224016Z remote: Counting objects: 28% (4/14) 2025-04-21T22:01:16.7224423Z remote: Counting objects: 35% (5/14) 2025-04-21T22:01:16.7224777Z remote: Counting objects: 42% (6/14) 2025-04-21T22:01:16.7225134Z remote: Counting objects: 50% (7/14) 2025-04-21T22:01:16.7225494Z remote: Counting objects: 57% (8/14) 2025-04-21T22:01:16.7225848Z remote: Counting objects: 64% (9/14) 2025-04-21T22:01:16.7226211Z remote: Counting objects: 71% (10/14) 2025-04-21T22:01:16.7226588Z remote: Counting objects: 78% (11/14) 2025-04-21T22:01:16.7226954Z remote: Counting objects: 85% (12/14) 2025-04-21T22:01:16.7227309Z remote: Counting objects: 92% (13/14) 2025-04-21T22:01:16.7227680Z remote: Counting objects: 100% (14/14) 2025-04-21T22:01:16.7228072Z remote: Counting objects: 100% (14/14), done. 2025-04-21T22:01:16.7228469Z remote: Compressing objects: 9% (1/11) 2025-04-21T22:01:16.7234550Z remote: Compressing objects: 18% (2/11) 2025-04-21T22:01:16.7235015Z remote: Compressing objects: 27% (3/11) 2025-04-21T22:01:16.7235396Z remote: Compressing objects: 36% (4/11) 2025-04-21T22:01:16.7235767Z remote: Compressing objects: 45% (5/11) 2025-04-21T22:01:16.7236142Z remote: Compressing objects: 54% (6/11) 2025-04-21T22:01:16.7236520Z remote: Compressing objects: 63% (7/11) 2025-04-21T22:01:16.7236889Z remote: Compressing objects: 72% (8/11) 2025-04-21T22:01:16.7237262Z remote: Compressing objects: 81% (9/11) 2025-04-21T22:01:16.7237692Z remote: Compressing objects: 90% (10/11) 2025-04-21T22:01:16.7238109Z remote: Compressing objects: 100% (11/11) 2025-04-21T22:01:16.7238556Z remote: Compressing objects: 100% (11/11), done. 2025-04-21T22:01:16.7317977Z Receiving objects: 0% (1/35746) 2025-04-21T22:01:16.7362925Z Receiving objects: 1% (358/35746) 2025-04-21T22:01:16.7418280Z Receiving objects: 2% (715/35746) 2025-04-21T22:01:16.7463905Z Receiving objects: 3% (1073/35746) 2025-04-21T22:01:16.7507855Z Receiving objects: 4% (1430/35746) 2025-04-21T22:01:16.7548159Z Receiving objects: 5% (1788/35746) 2025-04-21T22:01:16.7584564Z Receiving objects: 6% (2145/35746) 2025-04-21T22:01:16.7622929Z Receiving objects: 7% (2503/35746) 2025-04-21T22:01:16.7670478Z Receiving objects: 8% (2860/35746) 2025-04-21T22:01:16.7705965Z Receiving objects: 9% (3218/35746) 2025-04-21T22:01:16.7743073Z Receiving objects: 10% (3575/35746) 2025-04-21T22:01:16.7812084Z Receiving objects: 11% (3933/35746) 2025-04-21T22:01:16.7877568Z Receiving objects: 12% (4290/35746) 2025-04-21T22:01:16.7956338Z Receiving objects: 13% (4647/35746) 2025-04-21T22:01:16.8155659Z Receiving objects: 14% (5005/35746) 2025-04-21T22:01:16.8298334Z Receiving objects: 15% (5362/35746) 2025-04-21T22:01:16.8372792Z Receiving objects: 16% (5720/35746) 2025-04-21T22:01:16.8422554Z Receiving objects: 17% (6077/35746) 2025-04-21T22:01:16.8499985Z Receiving objects: 18% (6435/35746) 2025-04-21T22:01:16.8552582Z Receiving objects: 19% (6792/35746) 2025-04-21T22:01:16.8590640Z Receiving objects: 20% (7150/35746) 2025-04-21T22:01:16.8658023Z Receiving objects: 21% (7507/35746) 2025-04-21T22:01:16.8700136Z Receiving objects: 22% (7865/35746) 2025-04-21T22:01:16.8805810Z Receiving objects: 23% (8222/35746) 2025-04-21T22:01:16.8820585Z Receiving objects: 24% (8580/35746) 2025-04-21T22:01:16.8839238Z Receiving objects: 25% (8937/35746) 2025-04-21T22:01:16.8853628Z Receiving objects: 26% (9294/35746) 2025-04-21T22:01:16.8871679Z Receiving objects: 27% (9652/35746) 2025-04-21T22:01:16.8896164Z Receiving objects: 28% (10009/35746) 2025-04-21T22:01:16.8916596Z Receiving objects: 29% (10367/35746) 2025-04-21T22:01:16.8932244Z Receiving objects: 30% (10724/35746) 2025-04-21T22:01:16.8973498Z Receiving objects: 31% (11082/35746) 2025-04-21T22:01:16.8989783Z Receiving objects: 32% (11439/35746) 2025-04-21T22:01:16.9019882Z Receiving objects: 33% (11797/35746) 2025-04-21T22:01:16.9040128Z Receiving objects: 34% (12154/35746) 2025-04-21T22:01:16.9071378Z Receiving objects: 35% (12512/35746) 2025-04-21T22:01:16.9089083Z Receiving objects: 36% (12869/35746) 2025-04-21T22:01:16.9130189Z Receiving objects: 37% (13227/35746) 2025-04-21T22:01:17.7259416Z Receiving objects: 38% (13584/35746) 2025-04-21T22:01:17.8469969Z Receiving objects: 38% (13930/35746), 73.78 MiB | 73.77 MiB/s 2025-04-21T22:01:18.4476030Z Receiving objects: 39% (13941/35746), 73.78 MiB | 73.77 MiB/s 2025-04-21T22:01:18.5303125Z Receiving objects: 40% (14299/35746), 112.21 MiB | 74.80 MiB/s 2025-04-21T22:01:18.6128410Z Receiving objects: 41% (14656/35746), 112.21 MiB | 74.80 MiB/s 2025-04-21T22:01:18.6939118Z Receiving objects: 42% (15014/35746), 112.21 MiB | 74.80 MiB/s 2025-04-21T22:01:18.7258081Z Receiving objects: 43% (15371/35746), 112.21 MiB | 74.80 MiB/s 2025-04-21T22:01:18.7759242Z Receiving objects: 43% (15508/35746), 112.21 MiB | 74.80 MiB/s 2025-04-21T22:01:18.8589601Z Receiving objects: 44% (15729/35746), 149.81 MiB | 74.90 MiB/s 2025-04-21T22:01:18.9415954Z Receiving objects: 45% (16086/35746), 149.81 MiB | 74.90 MiB/s 2025-04-21T22:01:19.1856690Z Receiving objects: 46% (16444/35746), 149.81 MiB | 74.90 MiB/s 2025-04-21T22:01:19.2619770Z Receiving objects: 47% (16801/35746), 149.81 MiB | 74.90 MiB/s 2025-04-21T22:01:19.4365480Z Receiving objects: 48% (17159/35746), 185.59 MiB | 74.24 MiB/s 2025-04-21T22:01:19.7260362Z Receiving objects: 49% (17516/35746), 185.59 MiB | 74.24 MiB/s 2025-04-21T22:01:19.7556000Z Receiving objects: 49% (17842/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7762605Z Receiving objects: 50% (17873/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7773003Z Receiving objects: 51% (18231/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7797177Z Receiving objects: 52% (18588/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7813082Z Receiving objects: 53% (18946/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7831635Z Receiving objects: 54% (19303/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7838117Z Receiving objects: 55% (19661/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7844963Z Receiving objects: 56% (20018/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7859097Z Receiving objects: 57% (20376/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7868876Z Receiving objects: 58% (20733/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7877101Z Receiving objects: 59% (21091/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7882909Z Receiving objects: 60% (21448/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7887717Z Receiving objects: 61% (21806/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.7904144Z Receiving objects: 62% (22163/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8102980Z Receiving objects: 63% (22520/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8108891Z Receiving objects: 64% (22878/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8117351Z Receiving objects: 65% (23235/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8124522Z Receiving objects: 66% (23593/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8137798Z Receiving objects: 67% (23950/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8143833Z Receiving objects: 68% (24308/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8163062Z Receiving objects: 69% (24665/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8169657Z Receiving objects: 70% (25023/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8181489Z Receiving objects: 71% (25380/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8318471Z Receiving objects: 72% (25738/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8325523Z Receiving objects: 73% (26095/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8334313Z Receiving objects: 74% (26453/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8343015Z Receiving objects: 75% (26810/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8589659Z Receiving objects: 76% (27167/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.8607610Z Receiving objects: 77% (27525/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9003951Z Receiving objects: 78% (27882/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9016308Z Receiving objects: 79% (28240/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9033613Z Receiving objects: 80% (28597/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9046398Z Receiving objects: 81% (28955/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9056156Z Receiving objects: 82% (29312/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9077430Z Receiving objects: 83% (29670/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9085977Z Receiving objects: 84% (30027/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9095221Z Receiving objects: 85% (30385/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9312572Z Receiving objects: 86% (30742/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9345403Z Receiving objects: 87% (31100/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9357845Z Receiving objects: 88% (31457/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9368667Z Receiving objects: 89% (31814/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9422340Z Receiving objects: 90% (32172/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9559605Z Receiving objects: 91% (32529/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9566867Z Receiving objects: 92% (32887/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9574939Z Receiving objects: 93% (33244/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9583365Z Receiving objects: 94% (33602/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9589645Z Receiving objects: 95% (33959/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9700978Z Receiving objects: 96% (34317/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9717112Z Receiving objects: 97% (34674/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9731812Z Receiving objects: 98% (35032/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9747546Z Receiving objects: 99% (35389/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9748327Z remote: Total 35746 (delta 3), reused 11 (delta 3), pack-reused 35732 (from 1) 2025-04-21T22:01:19.9765882Z Receiving objects: 100% (35746/35746), 220.36 MiB | 73.45 MiB/s 2025-04-21T22:01:19.9766353Z Receiving objects: 100% (35746/35746), 234.19 MiB | 72.04 MiB/s, done. 2025-04-21T22:01:19.9804673Z Resolving deltas: 0% (0/19886) 2025-04-21T22:01:19.9819711Z Resolving deltas: 1% (199/19886) 2025-04-21T22:01:19.9830326Z Resolving deltas: 2% (420/19886) 2025-04-21T22:01:19.9850633Z Resolving deltas: 3% (600/19886) 2025-04-21T22:01:19.9857245Z Resolving deltas: 4% (930/19886) 2025-04-21T22:01:19.9871600Z Resolving deltas: 5% (1045/19886) 2025-04-21T22:01:19.9879377Z Resolving deltas: 6% (1256/19886) 2025-04-21T22:01:19.9892388Z Resolving deltas: 7% (1402/19886) 2025-04-21T22:01:19.9902372Z Resolving deltas: 8% (1609/19886) 2025-04-21T22:01:19.9911380Z Resolving deltas: 9% (1835/19886) 2025-04-21T22:01:19.9928890Z Resolving deltas: 10% (1993/19886) 2025-04-21T22:01:19.9946982Z Resolving deltas: 11% (2189/19886) 2025-04-21T22:01:19.9970471Z Resolving deltas: 12% (2399/19886) 2025-04-21T22:01:19.9982405Z Resolving deltas: 13% (2597/19886) 2025-04-21T22:01:19.9996645Z Resolving deltas: 14% (2789/19886) 2025-04-21T22:01:20.0018672Z Resolving deltas: 15% (2984/19886) 2025-04-21T22:01:20.0039813Z Resolving deltas: 16% (3195/19886) 2025-04-21T22:01:20.0052218Z Resolving deltas: 17% (3391/19886) 2025-04-21T22:01:20.0085725Z Resolving deltas: 18% (3648/19886) 2025-04-21T22:01:20.0122809Z Resolving deltas: 19% (3779/19886) 2025-04-21T22:01:20.0152508Z Resolving deltas: 20% (3992/19886) 2025-04-21T22:01:20.0229380Z Resolving deltas: 21% (4187/19886) 2025-04-21T22:01:20.0267121Z Resolving deltas: 22% (4383/19886) 2025-04-21T22:01:20.0315765Z Resolving deltas: 23% (4583/19886) 2025-04-21T22:01:20.0344225Z Resolving deltas: 24% (4775/19886) 2025-04-21T22:01:20.0375605Z Resolving deltas: 25% (4972/19886) 2025-04-21T22:01:20.0405532Z Resolving deltas: 26% (5175/19886) 2025-04-21T22:01:20.0445281Z Resolving deltas: 27% (5370/19886) 2025-04-21T22:01:20.0508480Z Resolving deltas: 28% (5570/19886) 2025-04-21T22:01:20.0529593Z Resolving deltas: 29% (5816/19886) 2025-04-21T22:01:20.0603071Z Resolving deltas: 30% (5978/19886) 2025-04-21T22:01:20.0614264Z Resolving deltas: 32% (6526/19886) 2025-04-21T22:01:20.0637148Z Resolving deltas: 33% (6654/19886) 2025-04-21T22:01:20.0669007Z Resolving deltas: 34% (6784/19886) 2025-04-21T22:01:20.0691396Z Resolving deltas: 35% (6961/19886) 2025-04-21T22:01:20.0718049Z Resolving deltas: 36% (7185/19886) 2025-04-21T22:01:20.0758032Z Resolving deltas: 37% (7358/19886) 2025-04-21T22:01:20.0769410Z Resolving deltas: 39% (7788/19886) 2025-04-21T22:01:20.0819163Z Resolving deltas: 40% (7962/19886) 2025-04-21T22:01:20.0871851Z Resolving deltas: 41% (8158/19886) 2025-04-21T22:01:20.0905806Z Resolving deltas: 42% (8354/19886) 2025-04-21T22:01:20.0919792Z Resolving deltas: 43% (8551/19886) 2025-04-21T22:01:20.0942382Z Resolving deltas: 44% (8759/19886) 2025-04-21T22:01:20.0957230Z Resolving deltas: 45% (8968/19886) 2025-04-21T22:01:20.0973067Z Resolving deltas: 46% (9149/19886) 2025-04-21T22:01:20.0989027Z Resolving deltas: 47% (9347/19886) 2025-04-21T22:01:20.1009086Z Resolving deltas: 48% (9547/19886) 2025-04-21T22:01:20.1021101Z Resolving deltas: 49% (9753/19886) 2025-04-21T22:01:20.1033405Z Resolving deltas: 50% (9945/19886) 2025-04-21T22:01:20.1047425Z Resolving deltas: 51% (10143/19886) 2025-04-21T22:01:20.1058119Z Resolving deltas: 52% (10381/19886) 2025-04-21T22:01:20.1065958Z Resolving deltas: 53% (10569/19886) 2025-04-21T22:01:20.1081007Z Resolving deltas: 54% (10749/19886) 2025-04-21T22:01:20.1109019Z Resolving deltas: 55% (10940/19886) 2025-04-21T22:01:20.1116047Z Resolving deltas: 56% (11255/19886) 2025-04-21T22:01:20.1130790Z Resolving deltas: 57% (11336/19886) 2025-04-21T22:01:20.1139973Z Resolving deltas: 58% (11555/19886) 2025-04-21T22:01:20.1154368Z Resolving deltas: 59% (11756/19886) 2025-04-21T22:01:20.1162234Z Resolving deltas: 60% (11955/19886) 2025-04-21T22:01:20.1193470Z Resolving deltas: 61% (12134/19886) 2025-04-21T22:01:20.1208439Z Resolving deltas: 62% (12494/19886) 2025-04-21T22:01:20.1213269Z Resolving deltas: 63% (12650/19886) 2025-04-21T22:01:20.1230995Z Resolving deltas: 64% (12747/19886) 2025-04-21T22:01:20.1243529Z Resolving deltas: 65% (12927/19886) 2025-04-21T22:01:20.1267074Z Resolving deltas: 66% (13125/19886) 2025-04-21T22:01:20.1277369Z Resolving deltas: 67% (13327/19886) 2025-04-21T22:01:20.1287048Z Resolving deltas: 68% (13523/19886) 2025-04-21T22:01:20.1300960Z Resolving deltas: 69% (13725/19886) 2025-04-21T22:01:20.1308575Z Resolving deltas: 70% (13972/19886) 2025-04-21T22:01:20.1317926Z Resolving deltas: 71% (14137/19886) 2025-04-21T22:01:20.1326267Z Resolving deltas: 72% (14355/19886) 2025-04-21T22:01:20.1341442Z Resolving deltas: 73% (14522/19886) 2025-04-21T22:01:20.1348926Z Resolving deltas: 74% (14776/19886) 2025-04-21T22:01:20.1356623Z Resolving deltas: 75% (14922/19886) 2025-04-21T22:01:20.1377101Z Resolving deltas: 76% (15141/19886) 2025-04-21T22:01:20.1392404Z Resolving deltas: 77% (15448/19886) 2025-04-21T22:01:20.1413177Z Resolving deltas: 78% (15647/19886) 2025-04-21T22:01:20.1415687Z Resolving deltas: 79% (15887/19886) 2025-04-21T22:01:20.1427054Z Resolving deltas: 80% (15917/19886) 2025-04-21T22:01:20.1435153Z Resolving deltas: 81% (16110/19886) 2025-04-21T22:01:20.1444035Z Resolving deltas: 82% (16311/19886) 2025-04-21T22:01:20.1460850Z Resolving deltas: 83% (16506/19886) 2025-04-21T22:01:20.1480108Z Resolving deltas: 84% (16706/19886) 2025-04-21T22:01:20.1487057Z Resolving deltas: 85% (16969/19886) 2025-04-21T22:01:20.1500733Z Resolving deltas: 86% (17105/19886) 2025-04-21T22:01:20.1519914Z Resolving deltas: 87% (17314/19886) 2025-04-21T22:01:20.1529893Z Resolving deltas: 88% (17500/19886) 2025-04-21T22:01:20.1544374Z Resolving deltas: 89% (17700/19886) 2025-04-21T22:01:20.1557141Z Resolving deltas: 90% (17898/19886) 2025-04-21T22:01:20.1568725Z Resolving deltas: 91% (18098/19886) 2025-04-21T22:01:20.1579262Z Resolving deltas: 92% (18339/19886) 2025-04-21T22:01:20.1589180Z Resolving deltas: 93% (18537/19886) 2025-04-21T22:01:20.1600590Z Resolving deltas: 94% (18695/19886) 2025-04-21T22:01:20.1613383Z Resolving deltas: 95% (18899/19886) 2025-04-21T22:01:20.1624156Z Resolving deltas: 96% (19103/19886) 2025-04-21T22:01:20.1636330Z Resolving deltas: 97% (19291/19886) 2025-04-21T22:01:20.1646215Z Resolving deltas: 98% (19494/19886) 2025-04-21T22:01:20.1668212Z Resolving deltas: 99% (19693/19886) 2025-04-21T22:01:20.1668511Z Resolving deltas: 100% (19886/19886) 2025-04-21T22:01:20.1669026Z Resolving deltas: 100% (19886/19886), done. 2025-04-21T22:01:21.1836312Z + pushd torchbench 2025-04-21T22:01:21.1836593Z ~/workspace/torchbench ~/workspace 2025-04-21T22:01:21.1836943Z + git checkout 373ffb19dc470f4423a3176a4133f8f4b3cdb5bd 2025-04-21T22:01:21.2299010Z Note: switching to '373ffb19dc470f4423a3176a4133f8f4b3cdb5bd'. 2025-04-21T22:01:21.2299304Z 2025-04-21T22:01:21.2299995Z You are in 'detached HEAD' state. You can look around, make experimental 2025-04-21T22:01:21.2301075Z changes and commit them, and you can discard any commits you make in this 2025-04-21T22:01:21.2302097Z state without impacting any branches by switching back to a branch. 2025-04-21T22:01:21.2302702Z 2025-04-21T22:01:21.2303102Z If you want to create a new branch to retain commits you create, you may 2025-04-21T22:01:21.2304069Z do so (now or later) by using -c with the switch command. Example: 2025-04-21T22:01:21.2304620Z 2025-04-21T22:01:21.2304837Z git switch -c 2025-04-21T22:01:21.2305219Z 2025-04-21T22:01:21.2305414Z Or undo this operation with: 2025-04-21T22:01:21.2305801Z 2025-04-21T22:01:21.2305899Z git switch - 2025-04-21T22:01:21.2306027Z 2025-04-21T22:01:21.2306257Z Turn off this advice by setting config variable advice.detachedHead to false 2025-04-21T22:01:21.2306599Z 2025-04-21T22:01:21.2306803Z HEAD is now at 373ffb19 Copy model before benchmark warmup runs (#145858) 2025-04-21T22:01:21.2309284Z + '[' '' ']' 2025-04-21T22:01:21.2309632Z + python install.py --continue_on_fail 2025-04-21T22:01:28.3847911Z checking packages numpy, torch, torchvision, torchaudio are installed, generating constaints...OK 2025-04-21T22:02:00.9282557Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2025-04-21T22:02:16.0614121Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2025-04-21T22:02:31.4158200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2025-04-21T22:02:46.3836896Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2025-04-21T22:02:59.7174252Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2025-04-21T22:03:21.0637532Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2025-04-21T22:03:37.2991984Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2025-04-21T22:03:53.5678367Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2025-04-21T22:04:09.7784512Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2025-04-21T22:04:09.7787200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2025-04-21T22:04:24.1992603Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2025-04-21T22:04:40.3354368Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2025-04-21T22:04:53.6696934Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2025-04-21T22:06:05.9718175Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2025-04-21T22:06:29.6104445Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2025-04-21T22:06:51.9249474Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2025-04-21T22:07:13.9981832Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2025-04-21T22:07:37.8571468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2025-04-21T22:08:00.0572723Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2025-04-21T22:08:21.5880533Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2025-04-21T22:08:43.8506000Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2025-04-21T22:09:06.1089938Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2025-04-21T22:09:28.4584724Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2025-04-21T22:09:50.3598603Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2025-04-21T22:10:12.4347311Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2025-04-21T22:10:27.2687849Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2025-04-21T22:11:02.7254054Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2025-04-21T22:11:23.3217519Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2025-04-21T22:11:42.2809904Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2025-04-21T22:12:06.7574725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2025-04-21T22:12:21.6635785Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2025-04-21T22:12:36.8628778Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2025-04-21T22:12:58.6456129Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2025-04-21T22:13:20.8650321Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2025-04-21T22:13:41.5016616Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2025-04-21T22:14:04.3442522Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2025-04-21T22:14:25.3934281Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2025-04-21T22:14:45.6449854Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2025-04-21T22:15:07.0188852Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2025-04-21T22:15:38.1140460Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2025-04-21T22:15:59.5440175Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2025-04-21T22:16:18.7837607Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2025-04-21T22:16:43.6881750Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Roberta_base...OK 2025-04-21T22:17:03.8081142Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2025-04-21T22:17:25.8483275Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2025-04-21T22:17:25.8484700Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2025-04-21T22:17:53.9994190Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2025-04-21T22:18:11.9646582Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2025-04-21T22:18:11.9649845Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2025-04-21T22:18:34.8155645Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2025-04-21T22:18:49.7943381Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2025-04-21T22:19:04.7878838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2025-04-21T22:20:03.5752830Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2025-04-21T22:22:06.3343371Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2025-04-21T22:22:20.0817409Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2025-04-21T22:22:35.3583046Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2025-04-21T22:22:35.3584759Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/microbench_unbacked_tolist_sum...SKIP - No install.py is found 2025-04-21T22:22:49.0597976Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2025-04-21T22:23:02.7691568Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2025-04-21T22:23:16.4788542Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2025-04-21T22:23:30.2074072Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2025-04-21T22:23:43.9284860Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2025-04-21T22:24:19.6804813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2025-04-21T22:24:19.6807186Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2025-04-21T22:24:34.6405530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2025-04-21T22:24:50.8814908Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2025-04-21T22:25:04.5933383Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2025-04-21T22:25:18.3182520Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2025-04-21T22:25:32.0334815Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2025-04-21T22:25:45.7587716Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2025-04-21T22:25:59.4745968Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2025-04-21T22:26:20.2118997Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2025-04-21T22:26:35.4672946Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2025-04-21T22:26:52.9177790Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2025-04-21T22:27:06.6189025Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2025-04-21T22:27:20.3043775Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2025-04-21T22:27:34.0294674Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2025-04-21T22:27:47.7121722Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2025-04-21T22:28:01.4188657Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2025-04-21T22:28:24.1781810Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2025-04-21T22:28:48.8569164Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2025-04-21T22:29:02.4443766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2025-04-21T22:29:02.4446691Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2025-04-21T22:29:02.4449935Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2025-04-21T22:29:18.7843135Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2025-04-21T22:29:34.2006372Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2025-04-21T22:29:47.8051312Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2025-04-21T22:30:17.7336563Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2025-04-21T22:30:38.8427971Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2025-04-21T22:30:57.0536731Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2025-04-21T22:31:19.0399296Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2025-04-21T22:31:32.6902969Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2025-04-21T22:31:46.3018957Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2025-04-21T22:31:59.9079349Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2025-04-21T22:32:13.5076679Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2025-04-21T22:32:27.1432437Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2025-04-21T22:32:40.7984165Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2025-04-21T22:32:54.3964118Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2025-04-21T22:33:14.3508985Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2025-04-21T22:33:33.9182969Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2025-04-21T22:33:47.5464050Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2025-04-21T22:34:02.8996570Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2025-04-21T22:34:18.5644420Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2025-04-21T22:34:25.3839248Z installed torchbench with package constraints: {'numpy': '1.22.4', 'torch': '2.8.0a0+gitb68c0ef', 'torchvision': '0.19.0a0+d23a6e1', 'torchaudio': '2.6.0a0+bccaa45'} 2025-04-21T22:34:25.7035117Z + pip install transformers==4.38.1 2025-04-21T22:34:26.1639589Z Collecting transformers==4.38.1 2025-04-21T22:34:26.1834325Z Downloading transformers-4.38.1-py3-none-any.whl.metadata (131 kB) 2025-04-21T22:34:26.3837049Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (3.16.1) 2025-04-21T22:34:26.3839998Z Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.30.2) 2025-04-21T22:34:26.3843796Z Requirement already satisfied: numpy>=1.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (1.22.4) 2025-04-21T22:34:26.3847796Z Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (24.2) 2025-04-21T22:34:26.3851661Z Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (6.0.2) 2025-04-21T22:34:26.3855583Z Requirement already satisfied: regex!=2019.12.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2024.11.6) 2025-04-21T22:34:26.3859464Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2.32.3) 2025-04-21T22:34:26.5265714Z Collecting tokenizers<0.19,>=0.14 (from transformers==4.38.1) 2025-04-21T22:34:26.5286915Z Using cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB) 2025-04-21T22:34:26.5318883Z Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.5.3) 2025-04-21T22:34:26.5322602Z Requirement already satisfied: tqdm>=4.27 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (4.67.1) 2025-04-21T22:34:26.5547772Z Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (2024.10.0) 2025-04-21T22:34:26.5554413Z Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (4.13.2) 2025-04-21T22:34:26.5731661Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.4.1) 2025-04-21T22:34:26.5734960Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.10) 2025-04-21T22:34:26.5738848Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2.4.0) 2025-04-21T22:34:26.5742453Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2025.1.31) 2025-04-21T22:34:26.6184856Z Downloading transformers-4.38.1-py3-none-any.whl (8.5 MB) 2025-04-21T22:34:26.6981946Z [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.5 MB ? eta -:--:-- 2025-04-21T22:34:26.6982723Z  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 MB 114.7 MB/s eta 0:00:00 2025-04-21T22:34:26.6999082Z [?25hUsing cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB) 2025-04-21T22:34:27.7092788Z Installing collected packages: tokenizers, transformers 2025-04-21T22:34:27.7093334Z Attempting uninstall: tokenizers 2025-04-21T22:34:27.7108729Z Found existing installation: tokenizers 0.19.1 2025-04-21T22:34:27.7151888Z Uninstalling tokenizers-0.19.1: 2025-04-21T22:34:27.7166716Z Successfully uninstalled tokenizers-0.19.1 2025-04-21T22:34:27.7923754Z Attempting uninstall: transformers 2025-04-21T22:34:27.7948253Z Found existing installation: transformers 4.44.2 2025-04-21T22:34:28.0066380Z Uninstalling transformers-4.44.2: 2025-04-21T22:34:28.0385946Z Successfully uninstalled transformers-4.44.2 2025-04-21T22:34:32.1714438Z Successfully installed tokenizers-0.15.2 transformers-4.38.1 2025-04-21T22:34:32.3072494Z + echo 'Print all dependencies after TorchBench is installed' 2025-04-21T22:34:32.3082639Z Print all dependencies after TorchBench is installed 2025-04-21T22:34:32.3083116Z + python -mpip freeze 2025-04-21T22:34:32.7563116Z absl-py==2.2.2 2025-04-21T22:34:32.7563494Z accelerate==1.6.0 2025-04-21T22:34:32.7563865Z aiohappyeyeballs==2.6.1 2025-04-21T22:34:32.7564231Z aiohttp==3.11.16 2025-04-21T22:34:32.7564565Z aiosignal==1.3.2 2025-04-21T22:34:32.7564901Z annotated-types==0.7.0 2025-04-21T22:34:32.7565280Z antlr4-python3-runtime==4.9.3 2025-04-21T22:34:32.7565669Z anyascii==0.3.2 2025-04-21T22:34:32.7566023Z astroid==3.3.9 2025-04-21T22:34:32.7566332Z astunparse==1.6.3 2025-04-21T22:34:32.7566636Z async-timeout==5.0.1 2025-04-21T22:34:32.7566961Z attrs==23.1.0 2025-04-21T22:34:32.7567259Z audioread==3.0.1 2025-04-21T22:34:32.7567555Z babel==2.17.0 2025-04-21T22:34:32.7567895Z beautifulsoup4==4.13.4 2025-04-21T22:34:32.7568892Z -e git+https://github.com/pytorch/benchmark@373ffb19dc470f4423a3176a4133f8f4b3cdb5bd#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2025-04-21T22:34:32.7569930Z black==25.1.0 2025-04-21T22:34:32.7570228Z blinker==1.9.0 2025-04-21T22:34:32.7570518Z blis==1.2.1 2025-04-21T22:34:32.7570793Z blobfile==3.0.0 2025-04-21T22:34:32.7571075Z bokeh==3.7.2 2025-04-21T22:34:32.7571301Z boto3==1.35.42 2025-04-21T22:34:32.7571592Z botocore==1.35.99 2025-04-21T22:34:32.7571820Z bs4==0.0.2 2025-04-21T22:34:32.7572049Z cachetools==5.5.2 2025-04-21T22:34:32.7572375Z cardboardlint==1.3.1 2025-04-21T22:34:32.7572699Z catalogue==2.0.10 2025-04-21T22:34:32.7572933Z certifi==2025.1.31 2025-04-21T22:34:32.7573238Z cffi==1.17.1 2025-04-21T22:34:32.7573496Z charset-normalizer==3.4.1 2025-04-21T22:34:32.7573802Z click==8.1.8 2025-04-21T22:34:32.7574040Z cloudpathlib==0.21.0 2025-04-21T22:34:32.7574391Z cloudpickle==3.1.1 2025-04-21T22:34:32.7574714Z colorama==0.4.6 2025-04-21T22:34:32.7575343Z confection==0.1.5 2025-04-21T22:34:32.7575691Z contourpy==1.2.1 2025-04-21T22:34:32.7576015Z coremltools==5.0b5 2025-04-21T22:34:32.7576372Z cryptography==44.0.2 2025-04-21T22:34:32.7576680Z csvw==3.5.1 2025-04-21T22:34:32.7576957Z cycler==0.12.1 2025-04-21T22:34:32.7577259Z cymem==2.0.11 2025-04-21T22:34:32.7577730Z Cython==3.0.12 2025-04-21T22:34:32.7578052Z DALL-E==0.1 2025-04-21T22:34:32.7578369Z dataclasses-json==0.6.7 2025-04-21T22:34:32.7578738Z datasets==3.5.0 2025-04-21T22:34:32.7579041Z decorator==5.2.1 2025-04-21T22:34:32.7579371Z defusedxml==0.7.1 2025-04-21T22:34:32.7579674Z Deprecated==1.2.18 2025-04-21T22:34:32.7580412Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@0df2d73d0013db7de629602c23cc120219b4f2b8 2025-04-21T22:34:32.7581141Z diffusers==0.30.3 2025-04-21T22:34:32.7581405Z dill==0.3.8 2025-04-21T22:34:32.7581692Z diskcache==5.6.3 2025-04-21T22:34:32.7581956Z distro==1.9.0 2025-04-21T22:34:32.7582163Z dlinfo==2.0.0 2025-04-21T22:34:32.7582404Z docker-pycreds==0.4.0 2025-04-21T22:34:32.7582730Z dominate==2.9.1 2025-04-21T22:34:32.7583352Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2025-04-21T22:34:32.7584072Z einops==0.8.1 2025-04-21T22:34:32.7584385Z exceptiongroup==1.2.2 2025-04-21T22:34:32.7584726Z execnet==2.1.1 2025-04-21T22:34:32.7584947Z expecttest==0.3.0 2025-04-21T22:34:32.7585258Z FastNLP==0.6.0 2025-04-21T22:34:32.7585528Z fbscribelogger==0.1.7 2025-04-21T22:34:32.7585775Z ffmpeg-python==0.2.0 2025-04-21T22:34:32.7586008Z filelock==3.16.1 2025-04-21T22:34:32.7586229Z Flask==3.1.0 2025-04-21T22:34:32.7586438Z flatbuffers==2.0 2025-04-21T22:34:32.7586686Z fonttools==4.57.0 2025-04-21T22:34:32.7587191Z frozenlist==1.6.0 2025-04-21T22:34:32.7587456Z fsspec==2024.10.0 2025-04-21T22:34:32.7587664Z ftfy==6.3.1 2025-04-21T22:34:32.7588135Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2025-04-21T22:34:32.7588679Z future==1.0.0 2025-04-21T22:34:32.7588908Z fvcore==0.1.5.post20221221 2025-04-21T22:34:32.7589148Z gdown==5.2.0 2025-04-21T22:34:32.7589352Z ghstack==0.8.0 2025-04-21T22:34:32.7589573Z gitdb==4.0.12 2025-04-21T22:34:32.7589786Z GitPython==3.1.44 2025-04-21T22:34:32.7590011Z google-auth==2.39.0 2025-04-21T22:34:32.7590255Z google-auth-oauthlib==1.0.0 2025-04-21T22:34:32.7590510Z grpcio==1.71.0 2025-04-21T22:34:32.7590719Z gym==0.26.2 2025-04-21T22:34:32.7590928Z gym-notices==0.0.8 2025-04-21T22:34:32.7591143Z h5py==3.13.0 2025-04-21T22:34:32.7591345Z higher==0.2.1 2025-04-21T22:34:32.7591567Z huggingface-hub==0.30.2 2025-04-21T22:34:32.7591806Z hydra-core==1.3.2 2025-04-21T22:34:32.7592029Z hypothesis==5.35.1 2025-04-21T22:34:32.7592263Z idna==3.10 2025-04-21T22:34:32.7592461Z imageio==2.37.0 2025-04-21T22:34:32.7592691Z importlib_metadata==8.6.1 2025-04-21T22:34:32.7592943Z inflect==7.5.0 2025-04-21T22:34:32.7593150Z iniconfig==2.1.0 2025-04-21T22:34:32.7593370Z iopath==0.1.9 2025-04-21T22:34:32.7593581Z isodate==0.7.2 2025-04-21T22:34:32.7593789Z isort==6.0.1 2025-04-21T22:34:32.7594002Z itsdangerous==2.2.0 2025-04-21T22:34:32.7594462Z Jinja2==3.1.6 2025-04-21T22:34:32.7594689Z jmespath==1.0.1 2025-04-21T22:34:32.7594913Z joblib==1.4.2 2025-04-21T22:34:32.7595123Z jsonpatch==1.33 2025-04-21T22:34:32.7595335Z jsonpointer==3.0.0 2025-04-21T22:34:32.7595564Z jsonschema==4.23.0 2025-04-21T22:34:32.7595818Z jsonschema-specifications==2024.10.1 2025-04-21T22:34:32.7596111Z junitparser==2.1.1 2025-04-21T22:34:32.7596336Z kaldi-io==0.9.8 2025-04-21T22:34:32.7596554Z kiwisolver==1.4.8 2025-04-21T22:34:32.7596773Z kornia==0.8.0 2025-04-21T22:34:32.7596979Z kornia_rs==0.1.8 2025-04-21T22:34:32.7597192Z lameenc==1.8.1 2025-04-21T22:34:32.7597412Z langcodes==3.5.0 2025-04-21T22:34:32.7597630Z langdetect==1.0.9 2025-04-21T22:34:32.7597856Z language-tags==1.2.0 2025-04-21T22:34:32.7598125Z language_data==1.3.0 2025-04-21T22:34:32.7598372Z lark==0.12.0 2025-04-21T22:34:32.7598584Z lazy_loader==0.4 2025-04-21T22:34:32.7598802Z libcst==1.7.0 2025-04-21T22:34:32.7599152Z librosa==0.9.2 2025-04-21T22:34:32.7599380Z lintrunner==0.12.7 2025-04-21T22:34:32.7599604Z llvmlite==0.38.1 2025-04-21T22:34:32.7599813Z lxml==5.3.0 2025-04-21T22:34:32.7600020Z marisa-trie==1.2.1 2025-04-21T22:34:32.7600238Z Markdown==3.8 2025-04-21T22:34:32.7600456Z markdown-it-py==3.0.0 2025-04-21T22:34:32.7600697Z MarkupSafe==3.0.2 2025-04-21T22:34:32.7600921Z marshmallow==3.26.1 2025-04-21T22:34:32.7601147Z matplotlib==3.8.4 2025-04-21T22:34:32.7601367Z mccabe==0.7.0 2025-04-21T22:34:32.7601575Z mdurl==0.1.2 2025-04-21T22:34:32.7601775Z ml_dtypes==0.5.1 2025-04-21T22:34:32.7601994Z MonkeyType==23.3.0 2025-04-21T22:34:32.7602223Z more-itertools==10.6.0 2025-04-21T22:34:32.7602463Z mpmath==1.3.0 2025-04-21T22:34:32.7602678Z msgpack==1.1.0 2025-04-21T22:34:32.7602891Z multidict==6.4.3 2025-04-21T22:34:32.7603107Z multiprocess==0.70.16 2025-04-21T22:34:32.7603346Z murmurhash==1.0.12 2025-04-21T22:34:32.7603570Z musdb==0.4.2 2025-04-21T22:34:32.7603770Z museval==0.4.1 2025-04-21T22:34:32.7603986Z mypy==1.14.0 2025-04-21T22:34:32.7604208Z mypy-extensions==1.0.0 2025-04-21T22:34:32.7604443Z narwhals==1.35.0 2025-04-21T22:34:32.7604662Z networkx==2.8.8 2025-04-21T22:34:32.7604877Z ninja==1.10.2 2025-04-21T22:34:32.7605137Z nose==1.3.7 2025-04-21T22:34:32.7605334Z numba==0.55.2 2025-04-21T22:34:32.7605530Z numpy==1.22.4 2025-04-21T22:34:32.7605758Z nvidia-cublas-cu12==12.4.5.8 2025-04-21T22:34:32.7606048Z nvidia-cuda-cupti-cu12==12.4.127 2025-04-21T22:34:32.7606334Z nvidia-cuda-nvrtc-cu12==12.4.127 2025-04-21T22:34:32.7606630Z nvidia-cuda-runtime-cu12==12.4.127 2025-04-21T22:34:32.7606927Z nvidia-cudnn-cu12==9.1.0.70 2025-04-21T22:34:32.7607193Z nvidia-cufft-cu12==11.2.1.3 2025-04-21T22:34:32.7607594Z nvidia-curand-cu12==10.3.5.147 2025-04-21T22:34:32.7607880Z nvidia-cusolver-cu12==11.6.1.9 2025-04-21T22:34:32.7608165Z nvidia-cusparse-cu12==12.3.1.170 2025-04-21T22:34:32.7608453Z nvidia-cusparselt-cu12==0.6.2 2025-04-21T22:34:32.7608733Z nvidia-ml-py==12.570.86 2025-04-21T22:34:32.7609001Z nvidia-nccl-cu12==2.21.5 2025-04-21T22:34:32.7609262Z nvidia-nvjitlink-cu12==12.4.127 2025-04-21T22:34:32.7609544Z nvidia-nvtx-cu12==12.4.127 2025-04-21T22:34:32.7609794Z oauthlib==3.2.2 2025-04-21T22:34:32.7610008Z omegaconf==2.3.0 2025-04-21T22:34:32.7610229Z onnx==1.17.0 2025-04-21T22:34:32.7610442Z onnxscript==0.2.2 2025-04-21T22:34:32.7610659Z opacus==1.5.3 2025-04-21T22:34:32.7610880Z opencv-python==4.8.0.74 2025-04-21T22:34:32.7611124Z opt-einsum==3.3.0 2025-04-21T22:34:32.7611362Z optree==0.13.0 2025-04-21T22:34:32.7611589Z packaging==24.2 2025-04-21T22:34:32.7611816Z pandas==2.0.3 2025-04-21T22:34:32.7612043Z parameterized==0.8.1 2025-04-21T22:34:32.7612280Z patch==1.16 2025-04-21T22:34:32.7612487Z pathspec==0.12.1 2025-04-21T22:34:32.7612723Z phonemizer==3.3.0 2025-04-21T22:34:32.7612945Z pillow==11.0.0 2025-04-21T22:34:32.7613165Z platformdirs==4.3.7 2025-04-21T22:34:32.7613395Z pluggy==1.5.0 2025-04-21T22:34:32.7613605Z ply==3.11 2025-04-21T22:34:32.7613798Z pooch==1.8.2 2025-04-21T22:34:32.7614015Z portalocker==3.1.1 2025-04-21T22:34:32.7614237Z preshed==3.0.9 2025-04-21T22:34:32.7614450Z prettytable==3.16.0 2025-04-21T22:34:32.7614680Z propcache==0.3.1 2025-04-21T22:34:32.7614905Z protobuf==3.20.2 2025-04-21T22:34:32.7615118Z psutil==7.0.0 2025-04-21T22:34:32.7615335Z PuLP==2.9.0 2025-04-21T22:34:32.7615540Z pwlf==2.2.1 2025-04-21T22:34:32.7615739Z py-cpuinfo==9.0.0 2025-04-21T22:34:32.7615959Z pyaml==25.1.0 2025-04-21T22:34:32.7616168Z pyarrow==19.0.1 2025-04-21T22:34:32.7616375Z pyasn1==0.6.1 2025-04-21T22:34:32.7616590Z pyasn1_modules==0.4.2 2025-04-21T22:34:32.7616836Z pyclipper==1.3.0.post6 2025-04-21T22:34:32.7617078Z pycocotools==2.0.8 2025-04-21T22:34:32.7617300Z pycparser==2.22 2025-04-21T22:34:32.7617713Z pycryptodomex==3.22.0 2025-04-21T22:34:32.7617967Z pydantic==2.11.3 2025-04-21T22:34:32.7618186Z pydantic_core==2.33.1 2025-04-21T22:34:32.7618424Z pyDOE==0.3.8 2025-04-21T22:34:32.7618631Z pydot==3.0.4 2025-04-21T22:34:32.7618833Z pygame==2.6.1 2025-04-21T22:34:32.7619047Z PyGithub==2.3.0 2025-04-21T22:34:32.7619352Z Pygments==2.15.0 2025-04-21T22:34:32.7619565Z PyJWT==2.10.1 2025-04-21T22:34:32.7619773Z pylint==3.3.6 2025-04-21T22:34:32.7619982Z PyNaCl==1.5.0 2025-04-21T22:34:32.7620184Z pynvml==12.0.0 2025-04-21T22:34:32.7620401Z pyparsing==3.2.3 2025-04-21T22:34:32.7620624Z pypdfium2==4.30.1 2025-04-21T22:34:32.7620839Z pysbd==0.3.4 2025-04-21T22:34:32.7621046Z PySocks==1.7.1 2025-04-21T22:34:32.7621261Z pytest==8.3.5 2025-04-21T22:34:32.7621481Z pytest-benchmark==5.1.0 2025-04-21T22:34:32.7621732Z pytest-cpp==2.3.0 2025-04-21T22:34:32.7621975Z pytest-flakefinder==1.1.0 2025-04-21T22:34:32.7622244Z pytest-rerunfailures==14.0 2025-04-21T22:34:32.7622516Z pytest-subtests==0.13.1 2025-04-21T22:34:32.7622769Z pytest-xdist==3.3.1 2025-04-21T22:34:32.7623006Z python-dateutil==2.9.0.post0 2025-04-21T22:34:32.7623274Z python-doctr==0.11.0 2025-04-21T22:34:32.7623958Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@e6aadeb86f3ae1f58c3f98e2a91e251716e0f2aa 2025-04-21T22:34:32.7624679Z pytz==2025.2 2025-04-21T22:34:32.7624888Z PyWavelets==1.4.1 2025-04-21T22:34:32.7625113Z PyYAML==6.0.2 2025-04-21T22:34:32.7625325Z pyzstd==0.16.2 2025-04-21T22:34:32.7625533Z RapidFuzz==3.13.0 2025-04-21T22:34:32.7625750Z rdflib==7.1.4 2025-04-21T22:34:32.7625964Z redis==5.2.1 2025-04-21T22:34:32.7626173Z referencing==0.36.2 2025-04-21T22:34:32.7626397Z regex==2024.11.6 2025-04-21T22:34:32.7626614Z requests==2.32.3 2025-04-21T22:34:32.7626839Z requests-oauthlib==2.0.0 2025-04-21T22:34:32.7627089Z resampy==0.4.3 2025-04-21T22:34:32.7627300Z rfc3986==1.5.0 2025-04-21T22:34:32.7627500Z rich==14.0.0 2025-04-21T22:34:32.7627706Z rpds-py==0.24.0 2025-04-21T22:34:32.7627914Z rsa==4.9.1 2025-04-21T22:34:32.7629050Z s3transfer==0.10.4 2025-04-21T22:34:32.7629277Z safetensors==0.5.3 2025-04-21T22:34:32.7629505Z scikit-image==0.22.0 2025-04-21T22:34:32.7629737Z scikit-learn==1.6.1 2025-04-21T22:34:32.7629963Z scipy==1.10.1 2025-04-21T22:34:32.7630173Z segments==2.3.0 2025-04-21T22:34:32.7630394Z sentencepiece==0.2.0 2025-04-21T22:34:32.7630625Z sentry-sdk==2.26.1 2025-04-21T22:34:32.7630847Z setproctitle==1.3.5 2025-04-21T22:34:32.7631064Z shapely==2.1.0 2025-04-21T22:34:32.7631280Z shellingham==1.5.4 2025-04-21T22:34:32.7631505Z simplejson==3.20.1 2025-04-21T22:34:32.7631717Z six==1.17.0 2025-04-21T22:34:32.7631922Z smart-open==7.1.0 2025-04-21T22:34:32.7632138Z smmap==5.0.2 2025-04-21T22:34:32.7632352Z sortedcontainers==2.4.0 2025-04-21T22:34:32.7632597Z soundfile==0.13.1 2025-04-21T22:34:32.7632820Z soupsieve==2.7 2025-04-21T22:34:32.7633030Z soxr==0.5.0.post1 2025-04-21T22:34:32.7633242Z spacy==3.8.5 2025-04-21T22:34:32.7633457Z spacy-legacy==3.0.12 2025-04-21T22:34:32.7633690Z spacy-loggers==1.0.5 2025-04-21T22:34:32.7633923Z srsly==2.5.1 2025-04-21T22:34:32.7634132Z stempeg==0.2.3 2025-04-21T22:34:32.7634344Z submitit==1.5.2 2025-04-21T22:34:32.7634555Z sympy==1.13.3 2025-04-21T22:34:32.7634766Z tabulate==0.9.0 2025-04-21T22:34:32.7634990Z tb-nightly==2.13.0a20230426 2025-04-21T22:34:32.7635243Z tensorboard==2.13.0 2025-04-21T22:34:32.7635496Z tensorboard-data-server==0.7.2 2025-04-21T22:34:32.7635776Z tensorboardX==2.6.2.2 2025-04-21T22:34:32.7636008Z termcolor==3.0.1 2025-04-21T22:34:32.7636230Z thinc==8.3.4 2025-04-21T22:34:32.7636442Z threadpoolctl==3.6.0 2025-04-21T22:34:32.7636668Z thriftpy2==0.5.2 2025-04-21T22:34:32.7636892Z tifffile==2025.3.30 2025-04-21T22:34:32.7637401Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2025-04-21T22:34:32.7637955Z tlparse==0.3.30 2025-04-21T22:34:32.7638199Z tokenizers==0.15.2 2025-04-21T22:34:32.7638446Z tomli==2.2.1 2025-04-21T22:34:32.7638650Z tomlkit==0.13.2 2025-04-21T22:34:32.7639416Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.8.0a0%2Bgitb68c0ef-cp310-cp310-linux_x86_64.whl#sha256=9bbbdfa20e86b1248aa24ec871651c6a0e5290f9fa886ae1219a5942117eb26f 2025-04-21T22:34:32.7640611Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2025-04-21T22:34:32.7641383Z torchao @ git+https://github.com/pytorch/ao.git@4805efd03b1aea9b5b9fe84df5c5a7171b5a7f2c 2025-04-21T22:34:32.7642077Z torchaudio @ git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b 2025-04-21T22:34:32.7642887Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@6569fcc03450c2360b50d772bf9b18ec3487fcf4 2025-04-21T22:34:32.7643699Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2025-04-21T22:34:32.7644202Z tornado==6.4.2 2025-04-21T22:34:32.7644411Z tqdm==4.67.1 2025-04-21T22:34:32.7644632Z transformers==4.38.1 2025-04-21T22:34:32.7644871Z treetable==0.2.5 2025-04-21T22:34:32.7645520Z triton @ file:///opt/triton/triton-3.3.0%2Bgit96316ce5-cp310-cp310-linux_x86_64.whl#sha256=417d95e3b977e7a78a0979b36cd88613615b5045352cbac12a89503f9febf540 2025-04-21T22:34:32.7646231Z typeguard==4.4.2 2025-04-21T22:34:32.7646450Z typer==0.15.2 2025-04-21T22:34:32.7646675Z typing-inspect==0.9.0 2025-04-21T22:34:32.7646931Z typing-inspection==0.4.0 2025-04-21T22:34:32.7647202Z typing_extensions==4.13.2 2025-04-21T22:34:32.7647447Z tzdata==2025.2 2025-04-21T22:34:32.7647664Z Unidecode==1.3.8 2025-04-21T22:34:32.7647911Z unittest-xml-reporting==3.2.0 2025-04-21T22:34:32.7648178Z uritemplate==4.1.1 2025-04-21T22:34:32.7648401Z urllib3==2.4.0 2025-04-21T22:34:32.7648613Z visdom==0.2.4 2025-04-21T22:34:32.7648818Z wandb==0.19.9 2025-04-21T22:34:32.7649023Z wasabi==1.1.3 2025-04-21T22:34:32.7649234Z wcwidth==0.2.13 2025-04-21T22:34:32.7649449Z weasel==0.4.1 2025-04-21T22:34:32.7649667Z websocket-client==1.8.0 2025-04-21T22:34:32.7649910Z Werkzeug==3.1.3 2025-04-21T22:34:32.7650127Z wrapt==1.17.2 2025-04-21T22:34:32.7650439Z xdoctest==1.1.0 2025-04-21T22:34:32.7650656Z xxhash==3.5.0 2025-04-21T22:34:32.7650873Z xyzservices==2025.1.0 2025-04-21T22:34:32.7651097Z yacs==0.1.8 2025-04-21T22:34:32.7651300Z yarl==1.20.0 2025-04-21T22:34:32.7651510Z z3-solver==4.12.6.0 2025-04-21T22:34:32.7651725Z zipp==3.21.0 2025-04-21T22:34:32.7882706Z + popd 2025-04-21T22:34:32.7883758Z ~/workspace 2025-04-21T22:34:32.7884050Z + [[ inductor_torchbench != *cpu* ]] 2025-04-21T22:34:32.7884390Z + install_torchrec_and_fbgemm 2025-04-21T22:34:32.7884676Z + local torchrec_commit 2025-04-21T22:34:32.7887991Z ++ get_pinned_commit torchrec 2025-04-21T22:34:32.7888358Z ++ cat .github/ci_commit_pins/torchrec.txt 2025-04-21T22:34:32.7915253Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T22:34:32.7915636Z + local fbgemm_commit 2025-04-21T22:34:32.7919139Z ++ get_pinned_commit fbgemm 2025-04-21T22:34:32.7919443Z ++ cat .github/ci_commit_pins/fbgemm.txt 2025-04-21T22:34:32.7944515Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2025-04-21T22:34:32.7944982Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-04-21T22:34:32.7945350Z + pip_uninstall torchrec-nightly 2025-04-21T22:34:32.7945673Z + pip3 uninstall -y torchrec-nightly 2025-04-21T22:34:33.1868440Z WARNING: Skipping torchrec-nightly as it is not installed. 2025-04-21T22:34:33.2236956Z + pip_uninstall fbgemm-gpu-nightly 2025-04-21T22:34:33.2237354Z + pip3 uninstall -y fbgemm-gpu-nightly 2025-04-21T22:34:33.6156085Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2025-04-21T22:34:33.6527664Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2025-04-21T22:34:33.6528200Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T22:34:33.6528808Z + python3 -m pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2025-04-21T22:34:34.0942638Z Collecting setuptools-git-versioning 2025-04-21T22:34:34.1120405Z Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl.metadata (6.1 kB) 2025-04-21T22:34:34.1344187Z Collecting scikit-build 2025-04-21T22:34:34.1384239Z Downloading scikit_build-0.18.1-py3-none-any.whl.metadata (18 kB) 2025-04-21T22:34:34.1732530Z Collecting pyre-extensions 2025-04-21T22:34:34.1764160Z Downloading pyre_extensions-0.0.32-py3-none-any.whl.metadata (4.0 kB) 2025-04-21T22:34:34.1830950Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.2) 2025-04-21T22:34:34.1833934Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (78.1.0) 2025-04-21T22:34:34.1839713Z Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (2.2.1) 2025-04-21T22:34:34.1850647Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2025-04-21T22:34:34.1856519Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.45.1) 2025-04-21T22:34:34.1863548Z Requirement already satisfied: typing-inspect in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (0.9.0) 2025-04-21T22:34:34.1866704Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.13.2) 2025-04-21T22:34:34.1955587Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2025-04-21T22:34:34.1994805Z Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl (10 kB) 2025-04-21T22:34:34.2092297Z Downloading scikit_build-0.18.1-py3-none-any.whl (85 kB) 2025-04-21T22:34:34.2221280Z Downloading pyre_extensions-0.0.32-py3-none-any.whl (12 kB) 2025-04-21T22:34:35.1158368Z Installing collected packages: setuptools-git-versioning, scikit-build, pyre-extensions 2025-04-21T22:34:35.2350188Z Successfully installed pyre-extensions-0.0.32 scikit-build-0.18.1 setuptools-git-versioning-2.1.0 2025-04-21T22:34:35.3483947Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]] 2025-04-21T22:34:35.3484333Z + CUDA_PATH=/usr/local/cuda-12.1 2025-04-21T22:34:35.3485126Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2025-04-21T22:34:35.3485971Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T22:34:35.3486916Z + python3 -m pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2025-04-21T22:34:35.7341936Z Collecting fbgemm-gpu 2025-04-21T22:34:35.7345694Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-fobks01o/fbgemm-gpu_ef26e9b04f094b6a92e59f8012d03786 2025-04-21T22:34:35.7379146Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-fobks01o/fbgemm-gpu_ef26e9b04f094b6a92e59f8012d03786 2025-04-21T22:34:36.9384326Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2025-04-21T22:34:36.9422364Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2025-04-21T22:34:37.7240844Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2025-04-21T22:34:38.2667686Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2025-04-21T22:34:38.2668354Z Running command git submodule update --init --recursive -q 2025-04-21T22:34:46.8027064Z Preparing metadata (setup.py) ... [?25l- done 2025-04-21T22:34:46.8046799Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.22.4) 2025-04-21T22:34:46.8054151Z Building wheels for collected packages: fbgemm-gpu 2025-04-21T23:20:22.2850598Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2025-04-21T23:20:22.4703466Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253793463 sha256=7d0663d2a5529ecb6102d648e9b57c031a9b740ca4aff6ea03325234cba59ca7 2025-04-21T23:20:22.4704839Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2025-04-21T23:20:22.4725613Z Successfully built fbgemm-gpu 2025-04-21T23:20:23.3657995Z Installing collected packages: fbgemm-gpu 2025-04-21T23:20:26.6192508Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2025-04-21T23:20:27.5734043Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:27.5734766Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-04-21T23:20:27.5735573Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:27.9589414Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:27.9593176Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-qmtyp4pr 2025-04-21T23:20:27.9626589Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-qmtyp4pr 2025-04-21T23:20:28.8350399Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2025-04-21T23:20:28.8377841Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:29.2491607Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:29.6255043Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2025-04-21T23:20:29.9086688Z Preparing metadata (setup.py) ... [?25l- done 2025-04-21T23:20:29.9120932Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2025-04-21T23:20:29.9123803Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2025-04-21T23:20:29.9126547Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2025-04-21T23:20:29.9579302Z Collecting torchmetrics (from torchrec==0.3.2) 2025-04-21T23:20:29.9772673Z Downloading torchmetrics-1.7.1-py3-none-any.whl.metadata (21 kB) 2025-04-21T23:20:30.0262427Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.67.1) 2025-04-21T23:20:30.0263863Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2025-04-21T23:20:30.0275022Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.22.4) 2025-04-21T23:20:30.0284814Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (3.1.1) 2025-04-21T23:20:30.0570865Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2025-04-21T23:20:30.0575482Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.2) 2025-04-21T23:20:30.0578465Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.2) 2025-04-21T23:20:30.0603479Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.2) 2025-04-21T23:20:30.0607489Z Requirement already satisfied: torch>=2.0.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.8.0a0+gitb68c0ef) 2025-04-21T23:20:30.0756756Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2025-04-21T23:20:30.0791512Z Downloading lightning_utilities-0.14.3-py3-none-any.whl.metadata (5.6 kB) 2025-04-21T23:20:30.0902092Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (78.1.0) 2025-04-21T23:20:30.0905073Z Requirement already satisfied: typing_extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.13.2) 2025-04-21T23:20:30.0925128Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.17.0) 2025-04-21T23:20:30.0957308Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.16.1) 2025-04-21T23:20:30.0962303Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.13.3) 2025-04-21T23:20:30.0965665Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2025-04-21T23:20:30.0969013Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.1.6) 2025-04-21T23:20:30.0972707Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2024.10.0) 2025-04-21T23:20:30.1021680Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2025-04-21T23:20:30.1406589Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.0.2) 2025-04-21T23:20:30.1558359Z Downloading torchmetrics-1.7.1-py3-none-any.whl (961 kB) 2025-04-21T23:20:30.1764417Z Downloading lightning_utilities-0.14.3-py3-none-any.whl (28 kB) 2025-04-21T23:20:30.1876377Z Building wheels for collected packages: torchrec 2025-04-21T23:20:30.7590344Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2025-04-21T23:20:30.7601007Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374580 sha256=96d85b77ef08a6c63297add2970a612abad3cb8ecfa75c7d30bb356a34fefe21 2025-04-21T23:20:30.7602435Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2025-04-21T23:20:30.7626110Z Successfully built torchrec 2025-04-21T23:20:31.6021980Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2025-04-21T23:20:32.2592724Z Successfully installed lightning-utilities-0.14.3 torchmetrics-1.7.1 torchrec-0.3.2 2025-04-21T23:20:32.4083581Z ++ pwd 2025-04-21T23:20:32.4085590Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2025-04-21T23:20:32.4086047Z + test_dynamo_benchmark torchbench 0 2025-04-21T23:20:32.4091098Z ++ pwd 2025-04-21T23:20:32.4095094Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-04-21T23:20:32.4095658Z + local suite=torchbench 2025-04-21T23:20:32.4095912Z + shift 2025-04-21T23:20:32.4096109Z + local shard_id=0 2025-04-21T23:20:32.4096326Z + shift 2025-04-21T23:20:32.4096574Z + [[ inductor_torchbench == *perf_compare* ]] 2025-04-21T23:20:32.4096905Z + [[ inductor_torchbench == *perf* ]] 2025-04-21T23:20:32.4097215Z + [[ inductor_torchbench == *cpu* ]] 2025-04-21T23:20:32.4097531Z + [[ inductor_torchbench == *aot_inductor* ]] 2025-04-21T23:20:32.4098006Z + [[ inductor_torchbench == *max_autotune_inductor* ]] 2025-04-21T23:20:32.4098489Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2025-04-21T23:20:32.4100848Z ++ pwd 2025-04-21T23:20:32.4103625Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-04-21T23:20:32.4104087Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-04-21T23:20:32.4168843Z + local name=inference 2025-04-21T23:20:32.4169653Z + shift 2025-04-21T23:20:32.4169950Z + local suite=torchbench 2025-04-21T23:20:32.4170298Z + shift 2025-04-21T23:20:32.4170576Z + local shard_id=0 2025-04-21T23:20:32.4170820Z + shift 2025-04-21T23:20:32.4171022Z + partition_flags=() 2025-04-21T23:20:32.4171281Z + local partition_flags 2025-04-21T23:20:32.4171519Z + [[ -n 2 ]] 2025-04-21T23:20:32.4171731Z + [[ -n 0 ]] 2025-04-21T23:20:32.4172115Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-04-21T23:20:32.4172617Z + [[ inductor_torchbench == *perf_compare* ]] 2025-04-21T23:20:32.4172947Z + [[ inductor_torchbench == *perf* ]] 2025-04-21T23:20:32.4173255Z + [[ inductor_torchbench == *_avx2* ]] 2025-04-21T23:20:32.4173567Z + [[ inductor_torchbench == *_avx512* ]] 2025-04-21T23:20:32.4174733Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2025-04-21T23:20:39.3659220Z 2025-04-21T23:20:40.5398414Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:20:40.5398775Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:20:40.5399080Z cuda eval torchrec_dlrm 2025-04-21T23:20:40.5425960Z Traceback (most recent call last): 2025-04-21T23:20:40.5426527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model 2025-04-21T23:20:40.5427088Z self.model_iter_fn(model, example_inputs) 2025-04-21T23:20:40.5427625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in forward_pass 2025-04-21T23:20:40.5428148Z return mod(*inputs) 2025-04-21T23:20:40.5428740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:20:40.5429383Z return self._call_impl(*args, **kwargs) 2025-04-21T23:20:40.5429965Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:20:40.5430568Z return forward_call(*args, **kwargs) 2025-04-21T23:20:40.5431127Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2025-04-21T23:20:40.5432046Z logits = self.model(batch.dense_features, batch.sparse_features) 2025-04-21T23:20:40.5432769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:20:40.5433397Z return self._call_impl(*args, **kwargs) 2025-04-21T23:20:40.5433972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:20:40.5434553Z return forward_call(*args, **kwargs) 2025-04-21T23:20:40.5435105Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2025-04-21T23:20:40.5435673Z concatenated_dense = self.inter_arch( 2025-04-21T23:20:40.5436289Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:20:40.5436915Z return self._call_impl(*args, **kwargs) 2025-04-21T23:20:40.5437498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:20:40.5438081Z return forward_call(*args, **kwargs) 2025-04-21T23:20:40.5438623Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2025-04-21T23:20:40.5439264Z return self.crossnet(combined_values.reshape([B, -1])) 2025-04-21T23:20:40.5439943Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:20:40.5440573Z return self._call_impl(*args, **kwargs) 2025-04-21T23:20:40.5441146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:20:40.5441892Z return forward_call(*args, **kwargs) 2025-04-21T23:20:40.5442479Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2025-04-21T23:20:40.5443144Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2025-04-21T23:20:40.5443741Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2025-04-21T23:20:40.5444136Z 2025-04-21T23:20:40.5444348Z The above exception was the direct cause of the following exception: 2025-04-21T23:20:40.5444659Z 2025-04-21T23:20:40.5444780Z Traceback (most recent call last): 2025-04-21T23:20:40.5445241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run 2025-04-21T23:20:40.5445716Z ) = runner.load_model( 2025-04-21T23:20:40.5446191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model 2025-04-21T23:20:40.5446733Z self.validate_model(model, example_inputs) 2025-04-21T23:20:40.5447277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model 2025-04-21T23:20:40.5447830Z raise RuntimeError("Eager run failed") from e 2025-04-21T23:20:40.5448186Z RuntimeError: Eager run failed 2025-04-21T23:20:40.5448369Z 2025-04-21T23:20:40.5448473Z eager_fail_to_run 2025-04-21T23:20:41.0622705Z [rank0]:[W421 23:20:41.052119220 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-04-21T23:20:44.7982843Z 2025-04-21T23:20:47.0971825Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:20:47.0972242Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:20:47.1028933Z cuda eval BERT_pytorch 2025-04-21T23:21:06.6464712Z Compilation time (from dynamo_timed): 16.92863384 2025-04-21T23:21:06.6465132Z pass 2025-04-21T23:21:06.6506982Z TIMING: _recursive_pre_grad_passes:0.01443 pad_mm_benchmark:0.45809 _recursive_joint_graph_passes:1.04159 _recursive_post_grad_passes:0.74215 async_compile.wait:1.95821 code_gen:4.29595 inductor_compile:7.53043 backend_compile:13.92712 entire_frame_compile:16.92863 gc:0.0001 cudagraphify.get_container:0.2744 CachingAutotuner.benchmark_all_configs:0.02836 CUDAGraphNode.record:0.26201 total_wall_time:16.92863 2025-04-21T23:21:06.6508950Z STATS: call_* op count: 543 | FakeTensor.__torch_dispatch__:6407 | FakeTensorMode.__torch_dispatch__:19121 | ProxyTorchDispatchMode.__torch_dispatch__:7466 2025-04-21T23:21:06.6509770Z Dynamo produced 1 graphs covering 543 ops with 0 graph breaks (0 unique) 2025-04-21T23:21:12.3350448Z 2025-04-21T23:21:15.0543970Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:21:15.0544457Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:21:15.0592335Z cuda eval Background_Matting 2025-04-21T23:21:15.0826919Z Compilation time (from dynamo_timed): 0 2025-04-21T23:21:15.0827313Z pass_due_to_skip 2025-04-21T23:21:15.1334483Z TIMING: total_wall_time:0 2025-04-21T23:21:15.1334826Z STATS: call_* op count: 0 2025-04-21T23:21:15.1335203Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-04-21T23:21:19.4134992Z 2025-04-21T23:21:22.7605388Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:21:22.7605938Z loading model: 0it [00:03, ?it/s] 2025-04-21T23:21:22.7632261Z cuda eval LearningToPaint 2025-04-21T23:21:31.0268350Z Compilation time (from dynamo_timed): 5.988433256 2025-04-21T23:21:31.0268753Z pass 2025-04-21T23:21:31.0277640Z TIMING: _recursive_pre_grad_passes:0.00596 pad_mm_benchmark:0.0613 _recursive_joint_graph_passes:0.22599 _recursive_post_grad_passes:0.09109 async_compile.wait:0.54015 code_gen:1.47912 inductor_compile:2.49497 backend_compile:4.59608 entire_frame_compile:5.98843 gc:9e-05 cudagraphify.get_container:0.22499 CUDAGraphNode.record:0.21761 total_wall_time:5.98843 2025-04-21T23:21:31.0279513Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4774 | ProxyTorchDispatchMode.__torch_dispatch__:1728 | FakeTensor.__torch_dispatch__:2713 2025-04-21T23:21:31.0280743Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique) 2025-04-21T23:21:36.6241123Z 2025-04-21T23:21:38.1496166Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2025-04-21T23:21:38.1652968Z 2025-04-21T23:21:38.1653168Z 2025-04-21T23:21:38.2655252Z 0% 0.00/528M [00:00 will be ignored 2025-04-21T23:48:05.9810546Z Compilation time (from dynamo_timed): 62.633208453 2025-04-21T23:48:05.9825117Z pass 2025-04-21T23:48:06.0086366Z TIMING: entire_frame_compile:47.6218 gc:0.00487 _recursive_pre_grad_passes:0.02246 _recursive_joint_graph_passes:0.72859 _recursive_post_grad_passes:0.54364 async_compile.wait:5.52985 code_gen:30.49206 inductor_compile:40.35053 backend_compile:38.36845 cudagraphify.get_container:0.39008 CachingAutotuner.benchmark_all_configs:3.74979 entire_backward_compile:15.01141 CUDAGraphNode.record:1.90368 total_wall_time:62.63321 2025-04-21T23:48:06.0088401Z STATS: call_* op count: 857 | FakeTensorMode.__torch_dispatch__:41342 | ProxyTorchDispatchMode.__torch_dispatch__:17943 | FakeTensor.__torch_dispatch__:24736 2025-04-21T23:48:06.0089241Z Dynamo produced 3 graphs covering 857 ops with 7 graph breaks (5 unique) 2025-04-21T23:48:13.2903855Z 2025-04-21T23:48:14.9296318Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:48:14.9296670Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:48:14.9297003Z cuda train alexnet 2025-04-21T23:48:31.0519667Z Compilation time (from dynamo_timed): 12.284947098 2025-04-21T23:48:31.0520153Z pass 2025-04-21T23:48:31.0724632Z TIMING: entire_frame_compile:4.69162 gc:0.00213 _recursive_pre_grad_passes:0.00427 pad_mm_benchmark:0.53853 _recursive_joint_graph_passes:0.70243 _recursive_post_grad_passes:0.05459 async_compile.wait:1.97496 code_gen:8.88549 inductor_compile:9.74588 backend_compile:4.12896 cudagraphify.get_container:0.18554 CachingAutotuner.benchmark_all_configs:0.74918 entire_backward_compile:7.59333 CUDAGraphNode.record:0.39342 total_wall_time:12.28495 2025-04-21T23:48:31.0726890Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:2803 | ProxyTorchDispatchMode.__torch_dispatch__:954 | FakeTensor.__torch_dispatch__:704 2025-04-21T23:48:31.0729546Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2025-04-21T23:48:36.5612315Z 2025-04-21T23:48:38.6312324Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:48:38.6312714Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:48:38.6313027Z cuda train basic_gnn_edgecnn 2025-04-21T23:48:42.1526508Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_reduce.two Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 98, in torch_dynamo_resume_in_scatter_at_93 2025-04-21T23:48:42.1527685Z return src.new_zeros(size).scatter_reduce_( 2025-04-21T23:48:42.1527931Z 2025-04-21T23:48:42.1527935Z 2025-04-21T23:48:44.1682823Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:48:44.1683394Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 190, in propagate 2025-04-21T23:48:44.1683871Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2025-04-21T23:48:44.1684313Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 128, in _collect 2025-04-21T23:48:44.1684724Z x_j = self._lift(x_j, edge_def, j) 2025-04-21T23:48:44.1685096Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 88, in _lift 2025-04-21T23:48:44.1685500Z return src.index_select(self.node_dim, index) 2025-04-21T23:48:44.1685749Z 2025-04-21T23:48:44.2166204Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:48:44.2174874Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 190, in propagate 2025-04-21T23:48:44.2175331Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2025-04-21T23:48:44.2175765Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 128, in _collect 2025-04-21T23:48:44.2176158Z x_j = self._lift(x_j, edge_def, j) 2025-04-21T23:48:44.2176523Z File "/tmp/jenkins_pyg/tmp17nb2c5b.py", line 88, in _lift 2025-04-21T23:48:44.2176916Z return src.index_select(self.node_dim, index) 2025-04-21T23:48:44.2177188Z 2025-04-21T23:48:44.4607126Z W0421 23:48:44.459000 61380 site-packages/torch/_logging/_internal.py:1130] [24/0] Profiler function will be ignored 2025-04-21T23:48:48.9784204Z Compilation time (from dynamo_timed): 8.021635863 2025-04-21T23:48:48.9784591Z pass 2025-04-21T23:48:48.9786170Z TIMING: entire_frame_compile:7.31414 gc:0.01107 _recursive_pre_grad_passes:0.0103 _recursive_joint_graph_passes:0.20826 inductor_compile:4.06301 backend_compile:5.40963 pad_mm_benchmark:0.01103 _recursive_post_grad_passes:0.05786 async_compile.wait:2.4351 code_gen:3.06213 cudagraphify.get_container:0.1749 entire_backward_compile:0.7075 CUDAGraphNode.record:1.06898 total_wall_time:8.02164 2025-04-21T23:48:48.9788290Z STATS: call_* op count: 128 | FakeTensorMode.__torch_dispatch__:5030 | ProxyTorchDispatchMode.__torch_dispatch__:1882 | FakeTensor.__torch_dispatch__:1927 2025-04-21T23:48:48.9789243Z Dynamo produced 8 graphs covering 128 ops with 20 graph breaks (6 unique) 2025-04-21T23:48:54.4913396Z 2025-04-21T23:48:56.6569255Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:48:56.6569712Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:48:56.6570054Z cuda train basic_gnn_gcn 2025-04-21T23:48:59.5642234Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/gcn_conv.py", line 100, in torch_dynamo_resume_in_gcn_norm_at_91 2025-04-21T23:48:59.5643507Z deg = scatter(edge_weight, idx, dim=0, dim_size=num_nodes, reduce='sum') 2025-04-21T23:48:59.5644203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-04-21T23:48:59.5644853Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-04-21T23:48:59.5645121Z 2025-04-21T23:48:59.5645126Z 2025-04-21T23:49:00.0504755Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmp6y77nkle.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-04-21T23:49:00.0506759Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-04-21T23:49:00.0507503Z File "/tmp/jenkins_pyg/tmp6y77nkle.py", line 188, in propagate 2025-04-21T23:49:00.0508356Z out = self.aggregate(out, dim_size=kwargs.dim_size, index=kwargs.index, ptr=kwargs.ptr) 2025-04-21T23:49:00.0509581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-04-21T23:49:00.0510758Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-04-21T23:49:00.0511804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-04-21T23:49:00.0512800Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-04-21T23:49:00.0513792Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-04-21T23:49:00.0514777Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-04-21T23:49:00.0515775Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-04-21T23:49:00.0516696Z return scatter(x, index, dim, dim_size, reduce) 2025-04-21T23:49:00.0517603Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-04-21T23:49:00.0518572Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-04-21T23:49:00.0518994Z 2025-04-21T23:49:00.0519000Z 2025-04-21T23:49:00.6005739Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmpl9dtmlqk.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-04-21T23:49:00.6006890Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-04-21T23:49:00.6007386Z File "/tmp/jenkins_pyg/tmpl9dtmlqk.py", line 188, in propagate 2025-04-21T23:49:00.6007932Z out = self.aggregate(out, dim_size=kwargs.dim_size, index=kwargs.index, ptr=kwargs.ptr) 2025-04-21T23:49:00.6009038Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-04-21T23:49:00.6009774Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-04-21T23:49:00.6010448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-04-21T23:49:00.6011112Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-04-21T23:49:00.6011756Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-04-21T23:49:00.6012409Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-04-21T23:49:00.6013058Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-04-21T23:49:00.6013662Z return scatter(x, index, dim, dim_size, reduce) 2025-04-21T23:49:00.6014265Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-04-21T23:49:00.6014902Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-04-21T23:49:00.6015164Z 2025-04-21T23:49:00.6015175Z 2025-04-21T23:49:00.8689636Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/tmp/jenkins_pyg/tmpl9dtmlqk.py", line 243, in torch_dynamo_resume_in_forward_at_221 2025-04-21T23:49:00.8691157Z out = self.propagate(edge_index, x=x, edge_weight=edge_weight, 2025-04-21T23:49:00.8691623Z File "/tmp/jenkins_pyg/tmpl9dtmlqk.py", line 188, in propagate 2025-04-21T23:49:00.8692172Z out = self.aggregate(out, dim_size=kwargs.dim_size, index=kwargs.index, ptr=kwargs.ptr) 2025-04-21T23:49:00.8693364Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-04-21T23:49:00.8694108Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-04-21T23:49:00.8695225Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-04-21T23:49:00.8695991Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-04-21T23:49:00.8696740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-04-21T23:49:00.8697498Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-04-21T23:49:00.8698280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-04-21T23:49:00.8698883Z return scatter(x, index, dim, dim_size, reduce) 2025-04-21T23:49:00.8699490Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-04-21T23:49:00.8700147Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-04-21T23:49:00.8700412Z 2025-04-21T23:49:00.8700417Z 2025-04-21T23:49:01.4896755Z W0421 23:49:01.488000 61736 site-packages/torch/_logging/_internal.py:1130] [18/0] Profiler function will be ignored 2025-04-21T23:49:04.8908777Z Compilation time (from dynamo_timed): 5.964835584 2025-04-21T23:49:04.8909126Z pass 2025-04-21T23:49:04.8917495Z TIMING: entire_frame_compile:5.48881 gc:0.00968 _recursive_pre_grad_passes:0.01079 _recursive_joint_graph_passes:0.18092 inductor_compile:3.30372 backend_compile:4.50316 _recursive_post_grad_passes:0.03842 async_compile.wait:2.14888 code_gen:2.69883 cudagraphify.get_container:0.16725 pad_mm_benchmark:0.0054 entire_backward_compile:0.47602 CUDAGraphNode.record:1.37477 total_wall_time:5.96484 2025-04-21T23:49:04.8919398Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:3029 | FakeTensor.__torch_dispatch__:1093 | ProxyTorchDispatchMode.__torch_dispatch__:1055 2025-04-21T23:49:04.8920241Z Dynamo produced 10 graphs covering 112 ops with 13 graph breaks (6 unique) 2025-04-21T23:49:10.3308197Z 2025-04-21T23:49:12.4336026Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:49:12.4336375Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:49:12.4336682Z cuda train basic_gnn_gin 2025-04-21T23:49:15.3435346Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:49:15.3436552Z pred = mod(*cloned_inputs) 2025-04-21T23:49:15.3437151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward 2025-04-21T23:49:15.3437771Z x = self.convs[i](x, edge_index) 2025-04-21T23:49:15.3438179Z File "/tmp/jenkins_pyg/tmpnwefz8fe.py", line 225, in forward 2025-04-21T23:49:15.3438657Z out = self.propagate(edge_index, x=x, size=size) 2025-04-21T23:49:15.3439082Z File "/tmp/jenkins_pyg/tmpnwefz8fe.py", line 182, in propagate 2025-04-21T23:49:15.3439649Z out = self.aggregate(out, dim_size=kwargs.dim_size, ptr=kwargs.ptr, index=kwargs.index) 2025-04-21T23:49:15.3440458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-04-21T23:49:15.3441185Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-04-21T23:49:15.3441866Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-04-21T23:49:15.3442532Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-04-21T23:49:15.3443187Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 21, in forward 2025-04-21T23:49:15.3444193Z return self.reduce(x, index, ptr, dim_size, dim, reduce='sum') 2025-04-21T23:49:15.3444848Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-04-21T23:49:15.3445462Z return scatter(x, index, dim, dim_size, reduce) 2025-04-21T23:49:15.3446067Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 74, in scatter 2025-04-21T23:49:15.3446714Z return src.new_zeros(size).scatter_add_(dim, index, src) 2025-04-21T23:49:15.3446984Z 2025-04-21T23:49:15.3446989Z 2025-04-21T23:49:16.0811289Z W0421 23:49:16.080000 62072 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-21T23:49:19.6882192Z Compilation time (from dynamo_timed): 6.167512220000001 2025-04-21T23:49:19.6884157Z pass 2025-04-21T23:49:19.6885786Z TIMING: entire_frame_compile:5.62626 gc:0.00373 _recursive_pre_grad_passes:0.00863 pad_mm_benchmark:0.01116 _recursive_joint_graph_passes:0.1977 _recursive_post_grad_passes:0.08148 async_compile.wait:1.24013 code_gen:2.10319 inductor_compile:2.91586 backend_compile:4.27583 entire_backward_compile:0.54126 cudagraphify.get_container:0.21148 CUDAGraphNode.record:0.20849 total_wall_time:6.16751 2025-04-21T23:49:19.6887984Z STATS: call_* op count: 135 | FakeTensorMode.__torch_dispatch__:4435 | ProxyTorchDispatchMode.__torch_dispatch__:1846 | FakeTensor.__torch_dispatch__:2028 2025-04-21T23:49:19.6888890Z Dynamo produced 3 graphs covering 135 ops with 7 graph breaks (5 unique) 2025-04-21T23:49:25.1529889Z 2025-04-21T23:49:27.2662455Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:49:27.2662800Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:49:27.2663107Z cuda train basic_gnn_sage 2025-04-21T23:49:30.1713312Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter_add.default Found from File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:49:30.1715053Z pred = mod(*cloned_inputs) 2025-04-21T23:49:30.1716336Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/models/basic_gnn.py", line 226, in forward 2025-04-21T23:49:30.1717301Z x = self.convs[i](x, edge_index) 2025-04-21T23:49:30.1717869Z File "/tmp/jenkins_pyg/tmpfo1lpj3s.py", line 228, in forward 2025-04-21T23:49:30.1718484Z out = self.propagate(edge_index, x=x, size=size) 2025-04-21T23:49:30.1719122Z File "/tmp/jenkins_pyg/tmpfo1lpj3s.py", line 182, in propagate 2025-04-21T23:49:30.1719948Z out = self.aggregate(out, ptr=kwargs.ptr, index=kwargs.index, dim_size=kwargs.dim_size) 2025-04-21T23:49:30.1721183Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/conv/message_passing.py", line 604, in aggregate 2025-04-21T23:49:30.1722290Z return self.aggr_module(inputs, index, ptr=ptr, dim_size=dim_size, 2025-04-21T23:49:30.1723321Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 109, in __call__ 2025-04-21T23:49:30.1724331Z return super().__call__(x, index, ptr, dim_size, dim, **kwargs) 2025-04-21T23:49:30.1725324Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/basic.py", line 34, in forward 2025-04-21T23:49:30.1726326Z return self.reduce(x, index, ptr, dim_size, dim, reduce='mean') 2025-04-21T23:49:30.1727327Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/nn/aggr/base.py", line 155, in reduce 2025-04-21T23:49:30.1728250Z return scatter(x, index, dim, dim_size, reduce) 2025-04-21T23:49:30.1729155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch_geometric/utils/scatter.py", line 78, in scatter 2025-04-21T23:49:30.1730156Z count.scatter_add_(0, index, src.new_ones(src.size(dim))) 2025-04-21T23:49:30.1730571Z 2025-04-21T23:49:30.1730589Z 2025-04-21T23:49:30.7843611Z W0421 23:49:30.783000 62321 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-21T23:49:34.0865637Z Compilation time (from dynamo_timed): 5.767490422 2025-04-21T23:49:34.0866163Z pass 2025-04-21T23:49:34.0867838Z TIMING: entire_frame_compile:5.32881 gc:0.00288 _recursive_pre_grad_passes:0.00808 pad_mm_benchmark:0.0106 _recursive_joint_graph_passes:0.19308 _recursive_post_grad_passes:0.06401 async_compile.wait:1.45765 code_gen:2.17392 inductor_compile:2.8542 backend_compile:4.34457 entire_backward_compile:0.43868 cudagraphify.get_container:0.20351 CUDAGraphNode.record:0.19865 total_wall_time:5.76749 2025-04-21T23:49:34.0869770Z STATS: call_* op count: 126 | FakeTensorMode.__torch_dispatch__:3746 | ProxyTorchDispatchMode.__torch_dispatch__:1471 | FakeTensor.__torch_dispatch__:1537 2025-04-21T23:49:34.0870609Z Dynamo produced 3 graphs covering 126 ops with 7 graph breaks (5 unique) 2025-04-21T23:49:39.5144712Z 2025-04-21T23:49:40.0960347Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:49:40.0960680Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:49:40.0960981Z cuda train dcgan 2025-04-21T23:49:45.6061893Z Compilation time (from dynamo_timed): 3.699421112 2025-04-21T23:49:45.6064106Z pass 2025-04-21T23:49:45.6084060Z TIMING: entire_frame_compile:2.76324 gc:0.00213 _recursive_pre_grad_passes:0.00387 _recursive_joint_graph_passes:0.14741 _recursive_post_grad_passes:0.04775 async_compile.wait:0.51039 code_gen:1.11016 inductor_compile:1.76873 backend_compile:2.20782 cudagraphify.get_container:0.16863 entire_backward_compile:0.93618 CUDAGraphNode.record:0.3423 total_wall_time:3.69942 2025-04-21T23:49:45.6085874Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:2160 | FakeTensor.__torch_dispatch__:1060 | ProxyTorchDispatchMode.__torch_dispatch__:834 2025-04-21T23:49:45.6086672Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2025-04-21T23:49:50.9111044Z 2025-04-21T23:49:54.6423656Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:49:54.6424085Z loading model: 0it [00:03, ?it/s] 2025-04-21T23:49:54.6424395Z cuda train demucs 2025-04-21T23:50:59.3026822Z E0421 23:50:59.301000 62958 site-packages/torch/_dynamo/utils.py:2939] RMSE (res-fp64): 0.13428, (ref-fp64): 0.01653 and shape=torch.Size([]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0 2025-04-21T23:50:59.3276329Z Compilation time (from dynamo_timed): 11.951810549000001 2025-04-21T23:50:59.3282594Z fail_accuracy 2025-04-21T23:50:59.3463643Z TIMING: entire_frame_compile:7.42332 gc:0.00563 _recursive_pre_grad_passes:0.0082 _recursive_joint_graph_passes:0.18489 inductor_compile:7.69021 backend_compile:5.36912 _recursive_post_grad_passes:0.12854 async_compile.precompile:0.28897 async_compile.wait:0.57087 code_gen:5.64989 cudagraphify.get_container:0.16961 entire_backward_compile:4.52849 CUDAGraphNode.record:1.19161 total_wall_time:11.95181 2025-04-21T23:50:59.3465612Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:6578 | ProxyTorchDispatchMode.__torch_dispatch__:2367 | FakeTensor.__torch_dispatch__:2892 2025-04-21T23:50:59.3466422Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2025-04-21T23:51:05.0364314Z 2025-04-21T23:51:07.2379541Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:51:07.2379883Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:51:07.2380192Z cuda train densenet121 2025-04-21T23:52:54.5186785Z Compilation time (from dynamo_timed): 98.11686742200001 2025-04-21T23:52:54.5234712Z pass 2025-04-21T23:52:54.5316994Z TIMING: entire_frame_compile:51.37318 gc:0.00355 _recursive_pre_grad_passes:0.01548 pad_mm_benchmark:0.21123 _recursive_joint_graph_passes:1.15714 _recursive_post_grad_passes:1.01345 async_compile.wait:14.98192 code_gen:49.69817 inductor_compile:64.57228 backend_compile:37.74812 cudagraphify.get_container:0.63647 entire_backward_compile:46.74368 CUDAGraphNode.record:2.0299 total_wall_time:98.11687 2025-04-21T23:52:54.5319380Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:61840 | ProxyTorchDispatchMode.__torch_dispatch__:30355 | FakeTensor.__torch_dispatch__:41523 2025-04-21T23:52:54.5320218Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2025-04-21T23:53:02.7461899Z 2025-04-21T23:53:05.3404915Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:53:05.3405279Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:53:05.3405599Z cuda train detectron2_maskrcnn_r_50_c4 2025-04-21T23:53:05.3419020Z Traceback (most recent call last): 2025-04-21T23:53:05.3419692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model 2025-04-21T23:53:05.3420248Z self.model_iter_fn(model, example_inputs) 2025-04-21T23:53:05.3420860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in forward_and_backward_pass 2025-04-21T23:53:05.3421444Z pred = mod(*cloned_inputs) 2025-04-21T23:53:05.3422075Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:53:05.3422709Z return self._call_impl(*args, **kwargs) 2025-04-21T23:53:05.3423296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:53:05.3423886Z return forward_call(*args, **kwargs) 2025-04-21T23:53:05.3424502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2025-04-21T23:53:05.3425127Z return self.inference(batched_inputs) 2025-04-21T23:53:05.3425744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2025-04-21T23:53:05.3426448Z results, _ = self.roi_heads(images, features, proposals, None) 2025-04-21T23:53:05.3427145Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:53:05.3427779Z return self._call_impl(*args, **kwargs) 2025-04-21T23:53:05.3428360Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:53:05.3429272Z return forward_call(*args, **kwargs) 2025-04-21T23:53:05.3429916Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2025-04-21T23:53:05.3430563Z box_features = self._shared_roi_transform( 2025-04-21T23:53:05.3431269Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2025-04-21T23:53:05.3431961Z x = self.pooler(features, boxes) 2025-04-21T23:53:05.3432565Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:53:05.3433191Z return self._call_impl(*args, **kwargs) 2025-04-21T23:53:05.3433778Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:53:05.3434359Z return forward_call(*args, **kwargs) 2025-04-21T23:53:05.3434944Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2025-04-21T23:53:05.3435582Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2025-04-21T23:53:05.3436242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl 2025-04-21T23:53:05.3436872Z return self._call_impl(*args, **kwargs) 2025-04-21T23:53:05.3437448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl 2025-04-21T23:53:05.3438032Z return forward_call(*args, **kwargs) 2025-04-21T23:53:05.3438614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2025-04-21T23:53:05.3439332Z return roi_align( 2025-04-21T23:53:05.3439861Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2025-04-21T23:53:05.3440664Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2025-04-21T23:53:05.3441461Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2025-04-21T23:53:05.3442219Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2025-04-21T23:53:05.3443024Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2025-04-21T23:53:05.3443657Z v1 = masked_index(y_low, x_low) 2025-04-21T23:53:05.3444231Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2025-04-21T23:53:05.3444805Z return input[ 2025-04-21T23:53:05.3446818Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21908.58 GiB. GPU 0 has a total capacity of 22.07 GiB of which 20.31 GiB is free. Including non-PyTorch memory, this process has 1.75 GiB memory in use. Of the allocated memory 1.38 GiB is allocated by PyTorch, and 77.04 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2025-04-21T23:53:05.3448804Z 2025-04-21T23:53:05.3449014Z The above exception was the direct cause of the following exception: 2025-04-21T23:53:05.3449334Z 2025-04-21T23:53:05.3449446Z Traceback (most recent call last): 2025-04-21T23:53:05.3449906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run 2025-04-21T23:53:05.3450367Z ) = runner.load_model( 2025-04-21T23:53:05.3450845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model 2025-04-21T23:53:05.3451388Z self.validate_model(model, example_inputs) 2025-04-21T23:53:05.3451925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model 2025-04-21T23:53:05.3452585Z raise RuntimeError("Eager run failed") from e 2025-04-21T23:53:05.3452934Z RuntimeError: Eager run failed 2025-04-21T23:53:05.3453117Z 2025-04-21T23:53:05.3453211Z eager_fail_to_run 2025-04-21T23:53:09.7333442Z 2025-04-21T23:53:18.7774628Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:53:18.7774964Z loading model: 0it [00:09, ?it/s] 2025-04-21T23:53:18.7775271Z cuda train dlrm 2025-04-21T23:53:25.2602983Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2025-04-21T23:53:25.2604448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:53:25.2605167Z pred = mod(*cloned_inputs) 2025-04-21T23:53:25.2605753Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 354, in forward 2025-04-21T23:53:25.2606438Z return self.sequential_forward(dense_x, lS_o, lS_i) 2025-04-21T23:53:25.2607155Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 366, in sequential_forward 2025-04-21T23:53:25.2607840Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2025-04-21T23:53:25.2608476Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 307, in apply_emb 2025-04-21T23:53:25.2609163Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2025-04-21T23:53:25.2609428Z 2025-04-21T23:53:25.7948249Z Compilation time (from dynamo_timed): 5.029116449 2025-04-21T23:53:25.7950374Z pass 2025-04-21T23:53:25.8365732Z TIMING: entire_frame_compile:4.26766 gc:0.00194 _recursive_pre_grad_passes:0.0048 pad_mm_benchmark:0.89718 _recursive_joint_graph_passes:1.07941 _recursive_post_grad_passes:0.07999 async_compile.wait:0.3641 code_gen:1.01457 inductor_compile:1.69489 backend_compile:3.53356 cudagraphify.get_container:0.19299 entire_backward_compile:0.76145 CUDAGraphNode.record:0.18499 total_wall_time:5.02912 2025-04-21T23:53:25.8367975Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:3726 | ProxyTorchDispatchMode.__torch_dispatch__:1383 | FakeTensor.__torch_dispatch__:1001 2025-04-21T23:53:25.8368793Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2025-04-21T23:53:31.1904970Z 2025-04-21T23:53:32.9643817Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:53:32.9644178Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:53:32.9644478Z cuda train drq 2025-04-21T23:53:39.1065691Z W0421 23:53:39.105000 67549 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-21T23:53:44.8746591Z Compilation time (from dynamo_timed): 10.549472336000001 2025-04-21T23:53:44.8747150Z pass 2025-04-21T23:53:44.8792172Z TIMING: entire_frame_compile:9.51811 gc:0.00433 _recursive_pre_grad_passes:0.01026 pad_mm_benchmark:0.54866 _recursive_joint_graph_passes:0.74056 _recursive_post_grad_passes:0.0807 async_compile.wait:3.24453 code_gen:4.49356 inductor_compile:5.68793 backend_compile:7.67424 cudagraphify.get_container:0.19014 entire_backward_compile:1.03136 total_wall_time:10.54947 2025-04-21T23:53:44.8794141Z STATS: call_* op count: 174 | FakeTensorMode.__torch_dispatch__:5849 | FakeTensor.__torch_dispatch__:3068 | ProxyTorchDispatchMode.__torch_dispatch__:2245 2025-04-21T23:53:44.8795288Z Dynamo produced 5 graphs covering 174 ops with 7 graph breaks (5 unique) 2025-04-21T23:53:50.5089799Z 2025-04-21T23:53:52.4282680Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:53:52.4283031Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:53:52.4283338Z cuda train fastNLP_Bert 2025-04-21T23:53:59.1997051Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] Graph break from `Tensor.item()`, consider setting: 2025-04-21T23:53:59.1999061Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] torch._dynamo.config.capture_scalar_outputs = True 2025-04-21T23:53:59.2000366Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] or: 2025-04-21T23:53:59.2001616Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2025-04-21T23:53:59.2003160Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] to include these operations in the captured graph. 2025-04-21T23:53:59.2004417Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 2025-04-21T23:53:59.2005581Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] Graph break: from user code at: 2025-04-21T23:53:59.2007594Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:53:59.2009578Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] pred = mod(*cloned_inputs) 2025-04-21T23:53:59.2022390Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward 2025-04-21T23:53:59.2024208Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] sequence_output = self.bert(words) 2025-04-21T23:53:59.2026205Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward 2025-04-21T23:53:59.2028473Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] outputs = self.model(words) 2025-04-21T23:53:59.2030429Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward 2025-04-21T23:53:59.2036052Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item() # 表示word piece的长度(包括padding) 2025-04-21T23:53:59.2037700Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 2025-04-21T23:53:59.2038712Z W0421 23:53:59.198000 68025 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 2025-04-21T23:54:29.5040252Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:54:29.5041125Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2025-04-21T23:54:29.5041857Z embedding_output = self.embeddings(input_ids, token_type_ids) 2025-04-21T23:54:29.5042581Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2025-04-21T23:54:29.5043221Z words_embeddings = self.word_embeddings(input_ids) 2025-04-21T23:54:29.5043477Z 2025-04-21T23:54:32.4286841Z Compilation time (from dynamo_timed): 30.097727407 2025-04-21T23:54:32.4310183Z pass 2025-04-21T23:54:32.4963866Z TIMING: entire_frame_compile:21.09927 gc:0.0061 _recursive_pre_grad_passes:0.01091 _recursive_joint_graph_passes:2.12035 inductor_compile:16.47666 backend_compile:17.74653 _recursive_post_grad_passes:0.90379 async_compile.precompile:0.2185 async_compile.wait:1.59922 code_gen:9.8479 cudagraphify.get_container:0.17612 pad_mm_benchmark:0.92225 CachingAutotuner.benchmark_all_configs:0.15393 entire_backward_compile:8.99846 CUDAGraphNode.record:2.54199 total_wall_time:30.09773 2025-04-21T23:54:32.4966124Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:41366 | ProxyTorchDispatchMode.__torch_dispatch__:19075 | FakeTensor.__torch_dispatch__:15615 2025-04-21T23:54:32.4967339Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2025-04-21T23:54:39.0002922Z 2025-04-21T23:54:40.0764423Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:54:40.0764796Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:54:40.0765139Z cuda train functorch_dp_cifar10 2025-04-21T23:55:02.7163377Z W0421 23:55:02.715000 68576 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-21T23:55:17.3057055Z Compilation time (from dynamo_timed): 34.025378792 2025-04-21T23:55:17.3062535Z pass 2025-04-21T23:55:17.3128651Z TIMING: entire_frame_compile:22.87449 gc:0.00292 _recursive_pre_grad_passes:0.01452 pad_mm_benchmark:0.2209 _recursive_joint_graph_passes:0.71807 _recursive_post_grad_passes:0.28853 async_compile.wait:5.24828 code_gen:15.66246 inductor_compile:21.74357 backend_compile:18.48813 cudagraphify.get_container:0.24273 entire_backward_compile:11.15089 CUDAGraphNode.record:1.20045 total_wall_time:34.02538 2025-04-21T23:55:17.3130638Z STATS: call_* op count: 402 | FakeTensorMode.__torch_dispatch__:20122 | ProxyTorchDispatchMode.__torch_dispatch__:8437 | FakeTensor.__torch_dispatch__:16157 2025-04-21T23:55:17.3131467Z Dynamo produced 3 graphs covering 402 ops with 7 graph breaks (5 unique) 2025-04-21T23:55:23.6507442Z 2025-04-21T23:55:24.3969493Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:55:24.3969828Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:55:24.3970143Z cuda train functorch_maml_omniglot 2025-04-21T23:55:32.2956030Z W0421 23:55:32.294000 69648 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-21T23:55:36.8791253Z Compilation time (from dynamo_timed): 10.389764847999999 2025-04-21T23:55:36.8791909Z pass 2025-04-21T23:55:36.8793546Z TIMING: entire_frame_compile:7.23115 gc:0.00322 _recursive_pre_grad_passes:0.0085 pad_mm_benchmark:0.21655 _recursive_joint_graph_passes:0.38664 _recursive_post_grad_passes:0.05869 async_compile.wait:2.41859 code_gen:5.8382 inductor_compile:7.04617 backend_compile:6.16184 cudagraphify.get_container:0.17277 entire_backward_compile:3.15861 CUDAGraphNode.record:0.60623 total_wall_time:10.38976 2025-04-21T23:55:36.8795838Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:4186 | ProxyTorchDispatchMode.__torch_dispatch__:1548 | FakeTensor.__torch_dispatch__:2581 2025-04-21T23:55:36.8796660Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique) 2025-04-21T23:55:42.3461440Z 2025-04-21T23:55:44.1122763Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:55:44.1123145Z loading model: 0it [00:01, ?it/s] 2025-04-21T23:55:44.1123464Z cuda train hf_Albert 2025-04-21T23:56:17.2268081Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:56:17.2269018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:56:17.2269740Z pred = mod(**cloned_inputs) 2025-04-21T23:56:17.2270391Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2025-04-21T23:56:17.2271044Z outputs = self.albert( 2025-04-21T23:56:17.2271686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2025-04-21T23:56:17.2272363Z embedding_output = self.embeddings( 2025-04-21T23:56:17.2273010Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2025-04-21T23:56:17.2273699Z inputs_embeds = self.word_embeddings(input_ids) 2025-04-21T23:56:17.2273964Z 2025-04-21T23:56:17.4468088Z W0421 23:56:17.445000 70123 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-21T23:56:24.5101359Z Compilation time (from dynamo_timed): 31.410874248 2025-04-21T23:56:24.5110474Z pass 2025-04-21T23:56:24.5170732Z TIMING: entire_frame_compile:22.99638 gc:0.00289 _recursive_pre_grad_passes:0.01505 pad_mm_benchmark:0.1765 _recursive_joint_graph_passes:2.27602 _recursive_post_grad_passes:0.61378 async_compile.wait:2.87037 code_gen:10.06014 inductor_compile:16.48682 backend_compile:18.56385 cudagraphify.get_container:0.30221 entire_backward_compile:8.4145 CachingAutotuner.benchmark_all_configs:0.06485 CUDAGraphNode.record:0.90164 total_wall_time:31.41087 2025-04-21T23:56:24.5172848Z STATS: call_* op count: 602 | FakeTensorMode.__torch_dispatch__:47590 | FakeTensor.__torch_dispatch__:19012 | ProxyTorchDispatchMode.__torch_dispatch__:21902 2025-04-21T23:56:24.5173739Z Dynamo produced 3 graphs covering 602 ops with 6 graph breaks (5 unique) 2025-04-21T23:56:31.0099771Z 2025-04-21T23:56:35.2739325Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:56:35.2739667Z loading model: 0it [00:04, ?it/s] 2025-04-21T23:56:35.2739996Z cuda train hf_Bart 2025-04-21T23:57:18.6488118Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:57:18.6489366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:57:18.6490361Z pred = mod(**cloned_inputs) 2025-04-21T23:57:18.6491250Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2025-04-21T23:57:18.6492169Z outputs = self.model( 2025-04-21T23:57:18.6493029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2025-04-21T23:57:18.6494768Z encoder_outputs = self.encoder( 2025-04-21T23:57:18.6495664Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2025-04-21T23:57:18.6496713Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2025-04-21T23:57:18.6497149Z 2025-04-21T23:57:18.8090024Z W0421 23:57:18.807000 70729 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-21T23:58:08.1317061Z Compilation time (from dynamo_timed): 79.141870654 2025-04-21T23:58:08.1345999Z pass 2025-04-21T23:58:08.1712444Z TIMING: entire_frame_compile:69.56495 gc:0.00555 _recursive_pre_grad_passes:0.04147 pad_mm_benchmark:0.52953 _recursive_joint_graph_passes:1.9632 _recursive_post_grad_passes:1.22975 async_compile.wait:3.18423 code_gen:23.54477 inductor_compile:40.4438 backend_compile:53.86298 cudagraphify.get_container:0.38779 entire_backward_compile:9.57692 CUDAGraphNode.record:2.09086 total_wall_time:79.14187 2025-04-21T23:58:08.1714409Z STATS: call_* op count: 1780 | FakeTensorMode.__torch_dispatch__:79171 | FakeTensor.__torch_dispatch__:44768 | ProxyTorchDispatchMode.__torch_dispatch__:34392 2025-04-21T23:58:08.1715256Z Dynamo produced 3 graphs covering 1780 ops with 6 graph breaks (5 unique) 2025-04-21T23:58:16.8376708Z 2025-04-21T23:58:19.7666544Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:58:19.7666873Z loading model: 0it [00:02, ?it/s] 2025-04-21T23:58:19.7667180Z cuda train hf_Bert 2025-04-21T23:58:56.6354317Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-21T23:58:56.6355251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-21T23:58:56.6356019Z pred = mod(**cloned_inputs) 2025-04-21T23:58:56.6356736Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2025-04-21T23:58:56.6357479Z outputs = self.bert( 2025-04-21T23:58:56.6358086Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2025-04-21T23:58:56.6358814Z embedding_output = self.embeddings( 2025-04-21T23:58:56.6360062Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2025-04-21T23:58:56.6360803Z inputs_embeds = self.word_embeddings(input_ids) 2025-04-21T23:58:56.6361091Z 2025-04-21T23:58:56.8060986Z W0421 23:58:56.804000 71275 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-21T23:59:34.6376668Z Compilation time (from dynamo_timed): 63.798570633000004 2025-04-21T23:59:34.6400601Z pass 2025-04-21T23:59:34.6704530Z TIMING: entire_frame_compile:56.51465 gc:0.0054 _recursive_pre_grad_passes:0.03435 pad_mm_benchmark:0.45353 _recursive_joint_graph_passes:2.46 _recursive_post_grad_passes:0.69277 async_compile.wait:2.5983 code_gen:18.15822 inductor_compile:30.22576 backend_compile:43.30639 cudagraphify.get_container:0.34501 entire_backward_compile:7.28392 CUDAGraphNode.record:1.70009 total_wall_time:63.79857 2025-04-21T23:59:34.6706484Z STATS: call_* op count: 1403 | FakeTensorMode.__torch_dispatch__:67966 | FakeTensor.__torch_dispatch__:34960 | ProxyTorchDispatchMode.__torch_dispatch__:29552 2025-04-21T23:59:34.6707313Z Dynamo produced 3 graphs covering 1403 ops with 6 graph breaks (5 unique) 2025-04-21T23:59:42.6079457Z 2025-04-21T23:59:47.7089859Z loading model: 0it [00:00, ?it/s] 2025-04-21T23:59:47.7090238Z loading model: 0it [00:05, ?it/s] 2025-04-21T23:59:47.7090550Z cuda train hf_Bert_large 2025-04-22T00:01:01.0618133Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-22T00:01:01.0619079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-22T00:01:01.0620302Z pred = mod(**cloned_inputs) 2025-04-22T00:01:01.0620988Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2025-04-22T00:01:01.0621638Z outputs = self.bert( 2025-04-22T00:01:01.0622238Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2025-04-22T00:01:01.0622889Z embedding_output = self.embeddings( 2025-04-22T00:01:01.0623518Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2025-04-22T00:01:01.0624179Z inputs_embeds = self.word_embeddings(input_ids) 2025-04-22T00:01:01.0624427Z 2025-04-22T00:01:01.2991578Z W0422 00:01:01.298000 71749 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:02:19.4305668Z Compilation time (from dynamo_timed): 126.039072846 2025-04-22T00:02:19.4351225Z pass 2025-04-22T00:02:19.5400927Z TIMING: entire_frame_compile:112.43322 gc:0.00402 _recursive_pre_grad_passes:0.06392 pad_mm_benchmark:0.64271 _recursive_joint_graph_passes:4.84449 _recursive_post_grad_passes:1.35121 async_compile.wait:4.61806 code_gen:37.07202 inductor_compile:60.79838 backend_compile:85.97636 cudagraphify.get_container:0.57084 entire_backward_compile:13.60585 CUDAGraphNode.record:3.38543 total_wall_time:126.03907 2025-04-22T00:02:19.5403133Z STATS: call_* op count: 2711 | FakeTensorMode.__torch_dispatch__:132262 | FakeTensor.__torch_dispatch__:67792 | ProxyTorchDispatchMode.__torch_dispatch__:57728 2025-04-22T00:02:19.5403962Z Dynamo produced 3 graphs covering 2711 ops with 6 graph breaks (5 unique) 2025-04-22T00:02:30.2699415Z 2025-04-22T00:02:34.2152384Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:02:34.2152697Z loading model: 0it [00:03, ?it/s] 2025-04-22T00:02:34.2153089Z cuda train hf_BigBird 2025-04-22T00:02:36.9851293Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2025-04-22T00:03:34.8688061Z W0422 00:03:34.867000 72250 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program 2025-04-22T00:03:59.9420083Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_1). Found from : 2025-04-22T00:03:59.9421002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-22T00:03:59.9421679Z pred = mod(**cloned_inputs) 2025-04-22T00:03:59.9422325Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2025-04-22T00:03:59.9422982Z outputs = self.bert( 2025-04-22T00:03:59.9423594Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2025-04-22T00:03:59.9424274Z encoder_outputs = self.encoder( 2025-04-22T00:03:59.9424914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2025-04-22T00:03:59.9425581Z layer_outputs = layer_module( 2025-04-22T00:03:59.9426216Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2025-04-22T00:03:59.9426890Z self_attention_outputs = self.attention( 2025-04-22T00:03:59.9427548Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2025-04-22T00:03:59.9428208Z self_outputs = self.self( 2025-04-22T00:03:59.9428821Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2025-04-22T00:03:59.9429628Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2025-04-22T00:03:59.9430686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention 2025-04-22T00:03:59.9431454Z rand_attn = np.stack(rand_attn, axis=0) 2025-04-22T00:03:59.9431666Z 2025-04-22T00:04:29.2725731Z W0422 00:04:29.271000 72250 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:05:08.6526121Z Compilation time (from dynamo_timed): 143.509446501 2025-04-22T00:05:08.6557085Z pass 2025-04-22T00:05:08.6730119Z TIMING: entire_frame_compile:115.429 gc:0.00579 _recursive_pre_grad_passes:0.06449 pad_mm_benchmark:1.50836 _recursive_joint_graph_passes:6.07539 _recursive_post_grad_passes:4.00105 async_compile.wait:4.18292 code_gen:47.63313 inductor_compile:75.81931 backend_compile:93.74572 entire_backward_compile:28.08045 cudagraphify.get_container:1.52018 CUDAGraphNode.record:1.43701 total_wall_time:143.50945 2025-04-22T00:05:08.6739990Z STATS: call_* op count: 4383 | FakeTensorMode.__torch_dispatch__:202218 | FakeTensor.__torch_dispatch__:68391 | ProxyTorchDispatchMode.__torch_dispatch__:86243 2025-04-22T00:05:08.6740892Z Dynamo produced 3 graphs covering 4383 ops with 6 graph breaks (5 unique) 2025-04-22T00:05:19.3351412Z 2025-04-22T00:05:21.7901236Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:05:21.7901654Z loading model: 0it [00:02, ?it/s] 2025-04-22T00:05:21.7901962Z cuda train hf_DistilBert 2025-04-22T00:05:43.7504451Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-22T00:05:43.7505302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-22T00:05:43.7506000Z pred = mod(**cloned_inputs) 2025-04-22T00:05:43.7506685Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2025-04-22T00:05:43.7507405Z dlbrt_output = self.distilbert( 2025-04-22T00:05:43.7508080Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2025-04-22T00:05:43.7509288Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2025-04-22T00:05:43.7510113Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2025-04-22T00:05:43.7510912Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2025-04-22T00:05:43.7511237Z 2025-04-22T00:05:43.8941949Z W0422 00:05:43.893000 73093 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:06:04.6185524Z Compilation time (from dynamo_timed): 35.811746049 2025-04-22T00:06:04.6197300Z pass 2025-04-22T00:06:04.6367880Z TIMING: entire_frame_compile:31.24807 gc:0.0032 _recursive_pre_grad_passes:0.02122 pad_mm_benchmark:0.13176 _recursive_joint_graph_passes:0.91229 _recursive_post_grad_passes:0.63884 async_compile.wait:2.58355 code_gen:10.89841 inductor_compile:18.18268 backend_compile:24.23631 cudagraphify.get_container:0.29172 entire_backward_compile:4.56367 CUDAGraphNode.record:0.99218 total_wall_time:35.81175 2025-04-22T00:06:04.6371656Z STATS: call_* op count: 754 | FakeTensorMode.__torch_dispatch__:35458 | FakeTensor.__torch_dispatch__:19688 | ProxyTorchDispatchMode.__torch_dispatch__:15200 2025-04-22T00:06:04.6373280Z Dynamo produced 3 graphs covering 754 ops with 6 graph breaks (5 unique) 2025-04-22T00:06:11.3012444Z 2025-04-22T00:06:14.8650301Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:06:14.8650646Z loading model: 0it [00:03, ?it/s] 2025-04-22T00:06:14.8650952Z cuda train hf_GPT2 2025-04-22T00:06:48.8963482Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-22T00:06:48.8965730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-22T00:06:48.8967078Z pred = mod(**cloned_inputs) 2025-04-22T00:06:48.8967977Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2025-04-22T00:06:48.8968845Z transformer_outputs = self.transformer( 2025-04-22T00:06:48.8969588Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2025-04-22T00:06:48.8970315Z inputs_embeds = self.wte(input_ids) 2025-04-22T00:06:48.8970531Z 2025-04-22T00:06:49.0338694Z W0422 00:06:49.032000 73555 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:07:17.2282083Z Compilation time (from dynamo_timed): 54.568022766 2025-04-22T00:07:17.2301926Z pass 2025-04-22T00:07:17.2898609Z TIMING: entire_frame_compile:45.85508 gc:0.00469 _recursive_pre_grad_passes:0.0308 pad_mm_benchmark:0.42488 _recursive_joint_graph_passes:1.79049 _recursive_post_grad_passes:0.82819 async_compile.wait:2.99602 code_gen:17.40246 inductor_compile:28.00893 backend_compile:35.5862 cudagraphify.get_container:0.31784 entire_backward_compile:8.71295 CUDAGraphNode.record:1.43435 total_wall_time:54.56802 2025-04-22T00:07:17.2900549Z STATS: call_* op count: 1393 | FakeTensorMode.__torch_dispatch__:55364 | FakeTensor.__torch_dispatch__:31247 | ProxyTorchDispatchMode.__torch_dispatch__:23394 2025-04-22T00:07:17.2901377Z Dynamo produced 3 graphs covering 1393 ops with 6 graph breaks (5 unique) 2025-04-22T00:07:24.8982898Z 2025-04-22T00:07:38.2717808Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:07:38.2718130Z loading model: 0it [00:13, ?it/s] 2025-04-22T00:07:38.2718433Z cuda train hf_GPT2_large 2025-04-22T00:07:38.2951419Z Compilation time (from dynamo_timed): 0 2025-04-22T00:07:38.2951866Z pass_due_to_skip 2025-04-22T00:07:38.8042000Z TIMING: total_wall_time:0 2025-04-22T00:07:38.8042358Z STATS: call_* op count: 0 2025-04-22T00:07:38.8042754Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-04-22T00:07:43.1276335Z 2025-04-22T00:07:45.2666923Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:07:45.2668216Z loading model: 0it [00:02, ?it/s] 2025-04-22T00:07:45.2668843Z cuda train hf_Reformer 2025-04-22T00:08:03.8845340Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-04-22T00:08:03.8847076Z attn_outputs = self.attention( 2025-04-22T00:08:03.8847744Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-04-22T00:08:03.8848442Z self_attention_outputs = self.self_attention( 2025-04-22T00:08:03.8849169Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-04-22T00:08:03.8850036Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-04-22T00:08:03.8851069Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-04-22T00:08:03.8853121Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-04-22T00:08:03.8853423Z 2025-04-22T00:08:03.8853427Z 2025-04-22T00:08:06.7686519Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-04-22T00:08:06.7688336Z attn_outputs = self.attention( 2025-04-22T00:08:06.7689992Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-04-22T00:08:06.7691050Z self_attention_outputs = self.self_attention( 2025-04-22T00:08:06.7692153Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-04-22T00:08:06.7693470Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-04-22T00:08:06.7695319Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-04-22T00:08:06.7696813Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-04-22T00:08:06.7697334Z 2025-04-22T00:08:06.7697342Z 2025-04-22T00:08:08.2913523Z skipping cudagraphs due to disabling cudagraphs due to incompatible op aten.scatter.src Found from File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1480, in torch_dynamo_resume_in_forward_at_1478 2025-04-22T00:08:08.2914902Z attn_outputs = self.attention( 2025-04-22T00:08:08.2915623Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 1313, in forward 2025-04-22T00:08:08.2916402Z self_attention_outputs = self.self_attention( 2025-04-22T00:08:08.2917174Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 482, in forward 2025-04-22T00:08:08.2918123Z sorted_bucket_idx, undo_sorted_bucket_idx = self._get_sorted_bucket_idx_and_undo_sorted_bucket_idx( 2025-04-22T00:08:08.2919282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 702, in _get_sorted_bucket_idx_and_undo_sorted_bucket_idx 2025-04-22T00:08:08.2920275Z undo_sorted_bucket_idx.scatter_(-1, sorted_bucket_idx, indices) 2025-04-22T00:08:08.2920581Z 2025-04-22T00:08:08.2920596Z 2025-04-22T00:08:11.5160285Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-22T00:08:11.5161502Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2025-04-22T00:08:11.5162324Z embedding_output = self.embeddings( 2025-04-22T00:08:11.5162991Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2025-04-22T00:08:11.5163702Z inputs_embeds = self.word_embeddings(input_ids) 2025-04-22T00:08:11.5163949Z 2025-04-22T00:08:11.5307139Z W0422 00:08:11.529000 74217 site-packages/torch/_logging/_internal.py:1130] [32/0] Profiler function will be ignored 2025-04-22T00:08:32.9639661Z Compilation time (from dynamo_timed): 31.941430034 2025-04-22T00:08:32.9646608Z pass 2025-04-22T00:08:32.9742173Z TIMING: entire_frame_compile:30.62231 gc:0.01511 _recursive_pre_grad_passes:0.02785 _recursive_joint_graph_passes:0.51258 inductor_compile:17.37152 backend_compile:23.42991 _recursive_post_grad_passes:0.16336 async_compile.wait:5.8707 code_gen:12.77521 cudagraphify.get_container:0.19363 pad_mm_benchmark:0.02966 entire_backward_compile:1.31912 CUDAGraphNode.record:6.51335 total_wall_time:31.94143 2025-04-22T00:08:32.9744318Z STATS: call_* op count: 844 | FakeTensorMode.__torch_dispatch__:24596 | ProxyTorchDispatchMode.__torch_dispatch__:9117 | FakeTensor.__torch_dispatch__:10597 | attempt fast:21 | fast is_contiguous:21 2025-04-22T00:08:32.9745301Z Dynamo produced 17 graphs covering 844 ops with 23 graph breaks (8 unique) 2025-04-22T00:08:39.4022649Z 2025-04-22T00:08:46.1458736Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:08:46.1459175Z loading model: 0it [00:06, ?it/s] 2025-04-22T00:08:46.1459535Z cuda train hf_Roberta_base 2025-04-22T00:08:50.6551803Z WARNING:common:fp64 golden ref were not generated for hf_Roberta_base. Setting accuracy check to cosine 2025-04-22T00:09:22.1721283Z skipping cudagraphs due to deterministic index put. Found from : 2025-04-22T00:09:22.1722881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 467, in torch_dynamo_resume_in_forward_and_backward_pass_at_464 2025-04-22T00:09:22.1724146Z pred = mod(**cloned_inputs) 2025-04-22T00:09:22.1725383Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 1092, in forward 2025-04-22T00:09:22.1726651Z outputs = self.roberta( 2025-04-22T00:09:22.1727835Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 830, in forward 2025-04-22T00:09:22.1728611Z embedding_output = self.embeddings( 2025-04-22T00:09:22.1729300Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/xlm_roberta/modeling_xlm_roberta.py", line 126, in forward 2025-04-22T00:09:22.1730046Z inputs_embeds = self.word_embeddings(input_ids) 2025-04-22T00:09:22.1730302Z 2025-04-22T00:09:22.3714407Z W0422 00:09:22.370000 74901 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:09:59.5684868Z Compilation time (from dynamo_timed): 63.805856297000005 2025-04-22T00:09:59.5698610Z pass 2025-04-22T00:09:59.5844023Z TIMING: entire_frame_compile:56.36853 gc:0.00527 _recursive_pre_grad_passes:0.03489 pad_mm_benchmark:1.23675 _recursive_joint_graph_passes:3.22688 _recursive_post_grad_passes:0.695 async_compile.wait:2.3641 code_gen:17.86386 inductor_compile:29.88745 backend_compile:43.30949 cudagraphify.get_container:0.36579 entire_backward_compile:7.43733 CUDAGraphNode.record:1.69664 total_wall_time:63.80586 2025-04-22T00:09:59.5845953Z STATS: call_* op count: 1411 | FakeTensorMode.__torch_dispatch__:68077 | FakeTensor.__torch_dispatch__:35013 | ProxyTorchDispatchMode.__torch_dispatch__:29586 2025-04-22T00:09:59.5846830Z Dynamo produced 3 graphs covering 1411 ops with 6 graph breaks (5 unique) 2025-04-22T00:10:07.5479050Z 2025-04-22T00:10:12.2569202Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:10:12.2570599Z loading model: 0it [00:04, ?it/s] 2025-04-22T00:10:12.2571223Z cuda train hf_T5_base 2025-04-22T00:10:15.3973259Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2025-04-22T00:10:17.5262782Z ERROR:common: 2025-04-22T00:10:17.5263120Z Traceback (most recent call last): 2025-04-22T00:10:17.5263654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2164, in check_accuracy 2025-04-22T00:10:17.5264207Z correct_rerun_result = self.run_n_iterations( 2025-04-22T00:10:17.5264762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1940, in run_n_iterations 2025-04-22T00:10:17.5265329Z model_iter_fn(mod, inputs, collect_outputs=False) 2025-04-22T00:10:17.5265970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 471, in forward_and_backward_pass 2025-04-22T00:10:17.5266581Z self.grad_scaler.scale(loss).backward() 2025-04-22T00:10:17.5267143Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward 2025-04-22T00:10:17.5267673Z torch.autograd.backward( 2025-04-22T00:10:17.5268208Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 354, in backward 2025-04-22T00:10:17.5268769Z _engine_run_backward( 2025-04-22T00:10:17.5269332Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 824, in _engine_run_backward 2025-04-22T00:10:17.5270153Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2025-04-22T00:10:17.5272492Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 252.00 MiB. GPU 0 has a total capacity of 22.07 GiB of which 222.44 MiB is free. Including non-PyTorch memory, this process has 21.84 GiB memory in use. Of the allocated memory 21.20 GiB is allocated by PyTorch, and 300.20 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2025-04-22T00:10:17.5507018Z Compilation time (from dynamo_timed): 0 2025-04-22T00:10:17.6396435Z eager_2nd_run_OOM 2025-04-22T00:10:17.6396733Z TIMING: total_wall_time:0 2025-04-22T00:10:17.6397008Z STATS: call_* op count: 0 2025-04-22T00:10:17.6397389Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-04-22T00:10:22.0023665Z 2025-04-22T00:10:32.7794833Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:10:32.7795606Z loading model: 0it [00:10, ?it/s] 2025-04-22T00:10:32.7796264Z cuda train hf_T5_large 2025-04-22T00:10:32.8031692Z Compilation time (from dynamo_timed): 0 2025-04-22T00:10:32.8032064Z pass_due_to_skip 2025-04-22T00:10:33.2556660Z TIMING: total_wall_time:0 2025-04-22T00:10:33.2556944Z STATS: call_* op count: 0 2025-04-22T00:10:33.2557368Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-04-22T00:10:37.6081882Z 2025-04-22T00:10:39.1206631Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:10:39.1206978Z loading model: 0it [00:01, ?it/s] 2025-04-22T00:10:39.1207279Z cuda train hf_Whisper 2025-04-22T00:10:58.2616740Z W0422 00:10:58.260000 75309 site-packages/torch/_logging/_internal.py:1130] [5/0] Profiler function will be ignored 2025-04-22T00:11:13.3294833Z Compilation time (from dynamo_timed): 28.596852779 2025-04-22T00:11:13.3298193Z pass 2025-04-22T00:11:13.3367457Z TIMING: entire_frame_compile:23.67815 gc:0.00387 _recursive_pre_grad_passes:0.01599 pad_mm_benchmark:0.2637 _recursive_joint_graph_passes:0.92377 _recursive_post_grad_passes:0.2999 async_compile.wait:5.59361 code_gen:12.43775 inductor_compile:16.96823 backend_compile:19.12182 cudagraphify.get_container:0.25204 CachingAutotuner.benchmark_all_configs:0.37264 entire_backward_compile:4.91871 CUDAGraphNode.record:1.16394 total_wall_time:28.59685 2025-04-22T00:11:13.3369607Z STATS: call_* op count: 488 | FakeTensorMode.__torch_dispatch__:21806 | FakeTensor.__torch_dispatch__:12755 | ProxyTorchDispatchMode.__torch_dispatch__:9170 2025-04-22T00:11:13.3370430Z Dynamo produced 3 graphs covering 488 ops with 6 graph breaks (5 unique) 2025-04-22T00:11:19.6960349Z 2025-04-22T00:11:19.8716996Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:11:19.8717345Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:11:19.8717652Z cuda train hf_distil_whisper 2025-04-22T00:11:19.8723245Z Traceback (most recent call last): 2025-04-22T00:11:19.8723741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run 2025-04-22T00:11:19.8724254Z ) = runner.load_model( 2025-04-22T00:11:19.8724719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 321, in load_model 2025-04-22T00:11:19.8725235Z benchmark = benchmark_cls( 2025-04-22T00:11:19.8725768Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__ 2025-04-22T00:11:19.8726326Z obj = type.__call__(cls, *args, **kwargs) 2025-04-22T00:11:19.8726969Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 13, in __init__ 2025-04-22T00:11:19.8727692Z raise NotImplementedError("Training is not implemented.") 2025-04-22T00:11:19.8728129Z NotImplementedError: Training is not implemented. 2025-04-22T00:11:19.8728378Z 2025-04-22T00:11:19.8728468Z model_fail_to_load 2025-04-22T00:11:23.9989376Z 2025-04-22T00:11:24.5736246Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:11:24.5736572Z loading model: 0it [00:00, ?it/s] 2025-04-22T00:11:24.5737281Z cuda train lennard_jones 2025-04-22T00:11:28.9960204Z W0422 00:11:28.995000 76055 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function will be ignored 2025-04-22T00:11:32.0141207Z Compilation time (from dynamo_timed): 5.494352701 2025-04-22T00:11:32.0141691Z pass 2025-04-22T00:11:32.0143309Z TIMING: entire_frame_compile:5.11603 gc:0.00268 _recursive_pre_grad_passes:0.00831 pad_mm_benchmark:0.5288 _recursive_joint_graph_passes:0.70303 _recursive_post_grad_passes:0.06797 async_compile.precompile:0.15278 async_compile.wait:1.27046 code_gen:1.96848 inductor_compile:2.59722 backend_compile:4.38721 cudagraphify.get_container:0.1683 entire_backward_compile:0.37832 CUDAGraphNode.record:0.55005 total_wall_time:5.49435 2025-04-22T00:11:32.0145448Z STATS: call_* op count: 82 | FakeTensorMode.__torch_dispatch__:3232 | ProxyTorchDispatchMode.__torch_dispatch__:1304 | FakeTensor.__torch_dispatch__:1603 2025-04-22T00:11:32.0146280Z Dynamo produced 3 graphs covering 82 ops with 7 graph breaks (5 unique) 2025-04-22T00:11:34.0749043Z accuracy pass_rate=79.41% 2025-04-22T00:11:34.0753035Z calls_captured gmean=0.00x mean=579.147x 2025-04-22T00:11:34.0756265Z unique_graphs gmean=0.00x mean=3.294x 2025-04-22T00:11:34.0759754Z graph_breaks gmean=0.00x mean=6.471x 2025-04-22T00:11:34.0763303Z unique_graph_breaks gmean=0.00x mean=4.324x 2025-04-22T00:11:34.0766695Z autograd_captures gmean=0.00x mean=0.000x 2025-04-22T00:11:34.0770018Z autograd_compiles gmean=0.00x mean=0.000x 2025-04-22T00:11:34.0773642Z cudagraph_skips gmean=0.00x mean=0.735x 2025-04-22T00:11:34.0774420Z compilation_latency mean=30.246 seconds 2025-04-22T00:11:35.8052896Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2025-04-22T00:11:36.1364250Z torchrec_dlrm PASS 2025-04-22T00:11:36.1367864Z BERT_pytorch PASS 2025-04-22T00:11:36.1372558Z Background_Matting XFAIL 2025-04-22T00:11:36.1377755Z LearningToPaint PASS 2025-04-22T00:11:36.1382698Z Super_SloMo PASS 2025-04-22T00:11:36.1387588Z alexnet PASS 2025-04-22T00:11:36.1392082Z basic_gnn_edgecnn PASS 2025-04-22T00:11:36.1396794Z basic_gnn_gcn PASS 2025-04-22T00:11:36.1401817Z basic_gnn_gin PASS 2025-04-22T00:11:36.1406316Z basic_gnn_sage PASS 2025-04-22T00:11:36.1411065Z dcgan PASS 2025-04-22T00:11:36.1415679Z demucs FAIL: accuracy=fail_accuracy, expected=pass 2025-04-22T00:11:36.1420701Z densenet121 PASS 2025-04-22T00:11:36.1425259Z detectron2_maskrcnn_r_50_c4 XFAIL 2025-04-22T00:11:36.1429891Z dlrm PASS 2025-04-22T00:11:36.1434556Z drq PASS 2025-04-22T00:11:36.1439168Z fastNLP_Bert PASS 2025-04-22T00:11:36.1443577Z functorch_dp_cifar10 PASS 2025-04-22T00:11:36.1448180Z functorch_maml_omniglot PASS 2025-04-22T00:11:36.1452577Z hf_Albert PASS 2025-04-22T00:11:36.1457108Z hf_Bart PASS 2025-04-22T00:11:36.1462234Z hf_Bert PASS 2025-04-22T00:11:36.1466759Z hf_Bert_large PASS 2025-04-22T00:11:36.1471393Z hf_BigBird PASS 2025-04-22T00:11:36.1475931Z hf_DistilBert PASS 2025-04-22T00:11:36.1480601Z hf_GPT2 PASS 2025-04-22T00:11:36.1485200Z hf_GPT2_large XFAIL 2025-04-22T00:11:36.1489984Z hf_Reformer PASS 2025-04-22T00:11:36.1496572Z hf_Roberta_base PASS 2025-04-22T00:11:36.1501439Z hf_T5_base XFAIL 2025-04-22T00:11:36.1506026Z hf_T5_large XFAIL 2025-04-22T00:11:36.1510575Z hf_Whisper PASS 2025-04-22T00:11:36.1515249Z hf_distil_whisper XFAIL 2025-04-22T00:11:36.1519999Z lennard_jones PASS 2025-04-22T00:11:36.1520259Z 2025-04-22T00:11:36.1520404Z Error: 1 models have accuracy status regressed: 2025-04-22T00:11:36.1520739Z demucs 2025-04-22T00:11:36.1520863Z 2025-04-22T00:11:36.1520868Z 2025-04-22T00:11:36.1521372Z If this change is expected, you can update `benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv` to reflect the new baseline. 2025-04-22T00:11:36.1522091Z from pytorch/pytorch root, run 2025-04-22T00:11:36.1522661Z `python benchmarks/dynamo/ci_expected_accuracy/update_expected.py b68c0effe97351b0c24b2f7aef671559be6aa977` 2025-04-22T00:11:36.1523381Z and then `git add` the resulting local changes to expected CSVs to your commit. 2025-04-22T00:11:36.1523727Z 2025-04-22T00:11:36.2053881Z + cleanup_workspace 2025-04-22T00:11:36.2063560Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-04-22T00:11:36.2064370Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-04-22T00:11:36.2064975Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-04-22T00:11:36.2065429Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-04-22T00:11:36.2065950Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-04-22T00:11:36.2066514Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-04-22T00:11:36.2066975Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-04-22T00:11:39.3249694Z ##[error]Process completed with exit code 1. 2025-04-22T00:11:39.3316689Z Prepare all required actions 2025-04-22T00:11:39.3317058Z Getting action download info 2025-04-22T00:11:39.4778113Z ##[group]Run ./.github/actions/pytest-cache-upload 2025-04-22T00:11:39.4778464Z with: 2025-04-22T00:11:39.4778678Z cache_dir: .pytest_cache 2025-04-22T00:11:39.4778919Z shard: 1 2025-04-22T00:11:39.4779165Z sha: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-22T00:11:39.4779501Z test_config: inductor_torchbench 2025-04-22T00:11:39.4779877Z job_identifier: inductor_linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-22T00:11:39.4780257Z env: 2025-04-22T00:11:39.4780468Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:39.4780787Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:39.4781335Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:39.4781812Z ##[endgroup] 2025-04-22T00:11:39.4823460Z ##[group]Run nick-fields/retry@v3.0.0 2025-04-22T00:11:39.4823759Z with: 2025-04-22T00:11:39.4823983Z shell: bash 2025-04-22T00:11:39.4824198Z timeout_minutes: 5 2025-04-22T00:11:39.4824435Z max_attempts: 5 2025-04-22T00:11:39.4824670Z retry_wait_seconds: 30 2025-04-22T00:11:39.4825002Z command: set -eu python3 -m pip install boto3==1.35.42 2025-04-22T00:11:39.4825379Z polling_interval_seconds: 1 2025-04-22T00:11:39.4825656Z warning_on_retry: true 2025-04-22T00:11:39.4825915Z continue_on_error: false 2025-04-22T00:11:39.4826157Z env: 2025-04-22T00:11:39.4826365Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:39.4826686Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:39.4827222Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:39.4827708Z ##[endgroup] 2025-04-22T00:11:39.8496115Z Defaulting to user installation because normal site-packages is not writeable 2025-04-22T00:11:39.8692116Z Requirement already satisfied: boto3==1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.42) 2025-04-22T00:11:39.8741285Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-04-22T00:11:39.8745168Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-04-22T00:11:39.8749008Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-04-22T00:11:39.8803698Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-04-22T00:11:39.8808922Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (2.8.2) 2025-04-22T00:11:39.8849630Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.15.0) 2025-04-22T00:11:40.5637309Z Command completed after 1 attempt(s). 2025-04-22T00:11:40.5731082Z ##[group]Run python3 .github/scripts/pytest_cache.py \ 2025-04-22T00:11:40.5731528Z python3 .github/scripts/pytest_cache.py \ 2025-04-22T00:11:40.5731860Z  --upload \ 2025-04-22T00:11:40.5732152Z  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \ 2025-04-22T00:11:40.5732508Z  --pr_identifier $GITHUB_REF \ 2025-04-22T00:11:40.5732835Z  --job_identifier $JOB_IDENTIFIER \ 2025-04-22T00:11:40.5733148Z  --sha $SHA \ 2025-04-22T00:11:40.5733412Z  --test_config $TEST_CONFIG \ 2025-04-22T00:11:40.5733710Z  --shard $SHARD \ 2025-04-22T00:11:40.5733967Z  --repo $REPO \ 2025-04-22T00:11:40.5734231Z  --temp_dir $RUNNER_TEMP \ 2025-04-22T00:11:40.5747682Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:40.5748040Z env: 2025-04-22T00:11:40.5748251Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:40.5748786Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:40.5749322Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:40.5749816Z CACHE_DIR: .pytest_cache 2025-04-22T00:11:40.5750171Z JOB_IDENTIFIER: inductor_linux-focal-cuda12.6-py3.10-gcc9-sm86 2025-04-22T00:11:40.5750590Z SHA: b68c0effe97351b0c24b2f7aef671559be6aa977 2025-04-22T00:11:40.5750913Z TEST_CONFIG: inductor_torchbench 2025-04-22T00:11:40.5751179Z SHARD: 1 2025-04-22T00:11:40.5751392Z REPO: pytorch/pytorch 2025-04-22T00:11:40.5751621Z ##[endgroup] 2025-04-22T00:11:40.7409420Z PR identifier for `refs/tags/ciflow/inductor/148893` is `a691aef95d58746c90a1abe7d18dc621` 2025-04-22T00:11:40.7411745Z Uploading cache with args Namespace(upload=True, download=False, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', pr_identifier='refs/tags/ciflow/inductor/148893', job_identifier='inductor_linux-focal-cuda12.6-py3.10-gcc9-sm86', sha='b68c0effe97351b0c24b2f7aef671559be6aa977', test_config='inductor_torchbench', shard='1', repo='pytorch/pytorch', temp_dir='/home/ec2-user/actions-runner/_work/_temp', bucket=None) 2025-04-22T00:11:40.7413861Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload 2025-04-22T00:11:40.7638085Z ##[group]Run cat test/**/*_toprint.log || true 2025-04-22T00:11:40.7638469Z cat test/**/*_toprint.log || true 2025-04-22T00:11:40.7647227Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:40.7647589Z env: 2025-04-22T00:11:40.7647796Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:40.7648112Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:40.7648652Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:40.7649129Z ##[endgroup] 2025-04-22T00:11:40.7738733Z cat: 'test/**/*_toprint.log': No such file or directory 2025-04-22T00:11:40.7775324Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-04-22T00:11:40.7775688Z kill "$MONITOR_SCRIPT_PID" 2025-04-22T00:11:40.7783904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:40.7784436Z env: 2025-04-22T00:11:40.7784643Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:40.7784958Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:40.7785497Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:40.7785988Z MONITOR_SCRIPT_PID: 289927 2025-04-22T00:11:40.7786251Z ##[endgroup] 2025-04-22T00:11:40.7930239Z Prepare all required actions 2025-04-22T00:11:40.7930661Z Getting action download info 2025-04-22T00:11:40.9039619Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-04-22T00:11:41.3641937Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-04-22T00:11:41.3642290Z with: 2025-04-22T00:11:41.3642688Z file-suffix: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-22T00:11:41.3643176Z s3-bucket: gha-artifacts 2025-04-22T00:11:41.3643426Z env: 2025-04-22T00:11:41.3643650Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.3643978Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.3644515Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.3644995Z ##[endgroup] 2025-04-22T00:11:41.3684353Z ##[group]Run # Remove any previous test jsons if they exist 2025-04-22T00:11:41.3684819Z # Remove any previous test jsons if they exist 2025-04-22T00:11:41.3685175Z rm -f test-jsons-*.zip 2025-04-22T00:11:41.3685585Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-04-22T00:11:41.3695336Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:41.3695689Z env: 2025-04-22T00:11:41.3695885Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.3696196Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.3696751Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.3697422Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-22T00:11:41.3698011Z ##[endgroup] 2025-04-22T00:11:41.3809354Z adding: test/test-reports/inference_torchbench.json (deflated 99%) 2025-04-22T00:11:41.3839152Z adding: test/test-reports/inference_torchbench_graph_breaks.json (deflated 98%) 2025-04-22T00:11:41.3886593Z adding: test/test-reports/training_torchbench.json (deflated 99%) 2025-04-22T00:11:41.4015778Z adding: test/test-reports/training_torchbench_graph_breaks.json (deflated 98%) 2025-04-22T00:11:41.4053536Z ##[group]Run # Remove any previous test reports if they exist 2025-04-22T00:11:41.4053998Z # Remove any previous test reports if they exist 2025-04-22T00:11:41.4054366Z rm -f test-reports-*.zip 2025-04-22T00:11:41.4054834Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-04-22T00:11:41.4063878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:41.4064268Z env: 2025-04-22T00:11:41.4064481Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.4064814Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.4065357Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.4066006Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-22T00:11:41.4066477Z ##[endgroup] 2025-04-22T00:11:41.4121032Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2025-04-22T00:11:41.4125155Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2025-04-22T00:11:41.4126962Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2025-04-22T00:11:41.4127818Z adding: test/test-reports/training_torchbench.csv (deflated 59%) 2025-04-22T00:11:41.4138473Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%) 2025-04-22T00:11:41.4140391Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 78%) 2025-04-22T00:11:41.4170171Z ##[group]Run # Remove any previous usage logs if they exist 2025-04-22T00:11:41.4170602Z # Remove any previous usage logs if they exist 2025-04-22T00:11:41.4170945Z rm -f logs-*.zip 2025-04-22T00:11:41.4171279Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-04-22T00:11:41.4171966Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-04-22T00:11:41.4180267Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:41.4180619Z env: 2025-04-22T00:11:41.4180825Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.4181146Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.4181677Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.4182334Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-22T00:11:41.4182809Z ##[endgroup] 2025-04-22T00:11:41.4301043Z adding: usage_log.txt (deflated 96%) 2025-04-22T00:11:41.4313699Z 2025-04-22T00:11:41.4314378Z zip error: Nothing to do! (logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip) 2025-04-22T00:11:41.4342472Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-04-22T00:11:41.4342967Z # Remove any previous debugging artifacts if they exist 2025-04-22T00:11:41.4343367Z rm -f debug-*.zip 2025-04-22T00:11:41.4343639Z if [ -d 'test/debug' ]; then 2025-04-22T00:11:41.4343979Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-04-22T00:11:41.4344306Z fi 2025-04-22T00:11:41.4352428Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:41.4352788Z env: 2025-04-22T00:11:41.4353002Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.4353325Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.4353860Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.4354529Z FILE_SUFFIX: test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323 2025-04-22T00:11:41.4354995Z ##[endgroup] 2025-04-22T00:11:41.4450761Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-04-22T00:11:41.4462776Z with: 2025-04-22T00:11:41.4463013Z s3-bucket: gha-artifacts 2025-04-22T00:11:41.4463452Z s3-prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:41.4463975Z retention-days: 14 2025-04-22T00:11:41.4464370Z if-no-files-found: warn 2025-04-22T00:11:41.4464786Z path: test-jsons-*.zip 2025-04-22T00:11:41.4465178Z name: artifact 2025-04-22T00:11:41.4465520Z region: us-east-1 2025-04-22T00:11:41.4465852Z env: 2025-04-22T00:11:41.4466189Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.4466700Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.4467542Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.4468306Z ##[endgroup] 2025-04-22T00:11:41.8120042Z NOTE: s3-prefix specified, ignoring name parameter 2025-04-22T00:11:41.8120492Z With the provided path, there will be 1 file uploaded 2025-04-22T00:11:41.8120932Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:41.8175633Z Starting upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:41.9690645Z Finished upload of test-jsons-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:41.9982056Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-04-22T00:11:41.9982401Z with: 2025-04-22T00:11:41.9982610Z s3-bucket: gha-artifacts 2025-04-22T00:11:41.9982921Z s3-prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:41.9983261Z retention-days: 14 2025-04-22T00:11:41.9983508Z if-no-files-found: error 2025-04-22T00:11:41.9983769Z path: test-reports-*.zip 2025-04-22T00:11:41.9984021Z name: artifact 2025-04-22T00:11:41.9984436Z region: us-east-1 2025-04-22T00:11:41.9984646Z env: 2025-04-22T00:11:41.9984850Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:41.9985170Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:41.9985701Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:41.9986201Z ##[endgroup] 2025-04-22T00:11:42.3376447Z NOTE: s3-prefix specified, ignoring name parameter 2025-04-22T00:11:42.3378410Z With the provided path, there will be 1 file uploaded 2025-04-22T00:11:42.3378872Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:42.3431747Z Starting upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:42.4568024Z Finished upload of test-reports-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:42.4858736Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-04-22T00:11:42.4859071Z with: 2025-04-22T00:11:42.4859298Z s3-bucket: gha-artifacts 2025-04-22T00:11:42.4859617Z s3-prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:42.4859955Z retention-days: 14 2025-04-22T00:11:42.4860201Z if-no-files-found: ignore 2025-04-22T00:11:42.4860469Z path: logs-*.zip 2025-04-22T00:11:42.4860700Z name: artifact 2025-04-22T00:11:42.4860917Z region: us-east-1 2025-04-22T00:11:42.4861134Z env: 2025-04-22T00:11:42.4861341Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:42.4861669Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:42.4862212Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:42.4862692Z ##[endgroup] 2025-04-22T00:11:42.8315067Z NOTE: s3-prefix specified, ignoring name parameter 2025-04-22T00:11:42.8315652Z With the provided path, there will be 1 file uploaded 2025-04-22T00:11:42.8316186Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:42.8369487Z Starting upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:43.0432492Z Finished upload of logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:43.0725508Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-04-22T00:11:43.0725830Z with: 2025-04-22T00:11:43.0726038Z s3-bucket: gha-artifacts 2025-04-22T00:11:43.0726354Z s3-prefix: pytorch/pytorch/14580756816/1/artifact 2025-04-22T00:11:43.0726754Z retention-days: 14 2025-04-22T00:11:43.0726997Z if-no-files-found: ignore 2025-04-22T00:11:43.0727258Z path: debug-*.zip 2025-04-22T00:11:43.0727483Z name: artifact 2025-04-22T00:11:43.0727696Z region: us-east-1 2025-04-22T00:11:43.0727913Z env: 2025-04-22T00:11:43.0728122Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:43.0728433Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:43.0728974Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:43.0729453Z ##[endgroup] 2025-04-22T00:11:43.4064531Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-04-22T00:11:43.4377980Z ##[group]Run # shellcheck disable=SC2156 2025-04-22T00:11:43.4378349Z # shellcheck disable=SC2156 2025-04-22T00:11:43.4378901Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-04-22T00:11:43.4388402Z shell: /usr/bin/bash -e {0} 2025-04-22T00:11:43.4388654Z env: 2025-04-22T00:11:43.4388860Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:43.4389171Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:43.4389707Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:43.4390186Z ##[endgroup] 2025-04-22T00:11:43.7305178Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-04-22T00:11:43.7305510Z with: 2025-04-22T00:11:43.7305838Z name: coredumps-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu 2025-04-22T00:11:43.7306430Z retention-days: 14 2025-04-22T00:11:43.7306672Z if-no-files-found: ignore 2025-04-22T00:11:43.7306932Z path: ./**/core.[1-9]* 2025-04-22T00:11:43.7307183Z s3-bucket: gha-artifacts 2025-04-22T00:11:43.7307436Z region: us-east-1 2025-04-22T00:11:43.7307642Z env: 2025-04-22T00:11:43.7307842Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:43.7308157Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:43.7308684Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:43.7309159Z ##[endgroup] 2025-04-22T00:11:55.1790766Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded. 2025-04-22T00:11:55.2186176Z Prepare all required actions 2025-04-22T00:11:55.2186525Z Getting action download info 2025-04-22T00:11:55.3129263Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-04-22T00:11:55.3129609Z with: 2025-04-22T00:11:55.3129827Z job_id: 40900324323 2025-04-22T00:11:55.3130302Z job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-22T00:11:55.3130823Z workflow_name: inductor 2025-04-22T00:11:55.3131076Z workflow_run_id: 14580756816 2025-04-22T00:11:55.3131333Z workflow_attempt: 1 2025-04-22T00:11:55.3131558Z env: 2025-04-22T00:11:55.3131750Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:55.3132064Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:55.3132596Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:55.3133074Z ##[endgroup] 2025-04-22T00:11:55.3169818Z ##[group]Run echo "workflow_id: 14580756816" 2025-04-22T00:11:55.3170246Z echo "workflow_id: 14580756816" 2025-04-22T00:11:55.3170550Z echo "workflow_attempt: 1" 2025-04-22T00:11:55.3170853Z echo "workflow_Name: inductor" 2025-04-22T00:11:55.3171152Z echo "job_id: 40900324323" 2025-04-22T00:11:55.3171690Z echo "job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)" 2025-04-22T00:11:55.3182067Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:55.3182419Z env: 2025-04-22T00:11:55.3182628Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:55.3182941Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:55.3183472Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:55.3183945Z ##[endgroup] 2025-04-22T00:11:55.3213172Z workflow_id: 14580756816 2025-04-22T00:11:55.3213516Z workflow_attempt: 1 2025-04-22T00:11:55.3213810Z workflow_Name: inductor 2025-04-22T00:11:55.3214040Z job_id: 40900324323 2025-04-22T00:11:55.3214498Z job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2025-04-22T00:11:55.3260590Z ##[group]Run nick-fields/retry@v3.0.0 2025-04-22T00:11:55.3260891Z with: 2025-04-22T00:11:55.3261096Z shell: bash 2025-04-22T00:11:55.3261308Z timeout_minutes: 5 2025-04-22T00:11:55.3261567Z max_attempts: 5 2025-04-22T00:11:55.3261802Z retry_wait_seconds: 30 2025-04-22T00:11:55.3262241Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 2025-04-22T00:11:55.3262735Z polling_interval_seconds: 1 2025-04-22T00:11:55.3263008Z warning_on_retry: true 2025-04-22T00:11:55.3263252Z continue_on_error: false 2025-04-22T00:11:55.3263498Z env: 2025-04-22T00:11:55.3263707Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:55.3264027Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:55.3264560Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:55.3265047Z ##[endgroup] 2025-04-22T00:11:55.6565098Z Defaulting to user installation because normal site-packages is not writeable 2025-04-22T00:11:55.6747215Z Requirement already satisfied: python-dateutil==2.8.2 in /home/ec2-user/.local/lib/python3.9/site-packages (2.8.2) 2025-04-22T00:11:55.6750642Z Requirement already satisfied: boto3==1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.42) 2025-04-22T00:11:55.6755172Z Requirement already satisfied: pandas==2.1.3 in /home/ec2-user/.local/lib/python3.9/site-packages (2.1.3) 2025-04-22T00:11:55.6769521Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-04-22T00:11:55.6817988Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-04-22T00:11:55.6821843Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-04-22T00:11:55.6827193Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-04-22T00:11:55.7571435Z Requirement already satisfied: numpy<2,>=1.22.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (1.26.4) 2025-04-22T00:11:55.7573711Z Requirement already satisfied: tzdata>=2022.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (2025.2) 2025-04-22T00:11:55.7577201Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-04-22T00:11:55.7635984Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-04-22T00:11:56.4045798Z Command completed after 1 attempt(s). 2025-04-22T00:11:56.4126627Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-04-22T00:11:56.4127309Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-04-22T00:11:56.4127806Z  --workflow-run-id "14580756816" \ 2025-04-22T00:11:56.4128149Z  --workflow-name "inductor" \ 2025-04-22T00:11:56.4128482Z  --workflow-run-attempt "1" \ 2025-04-22T00:11:56.4128791Z  --job-id "40900324323" \ 2025-04-22T00:11:56.4129341Z  --job-name "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)" 2025-04-22T00:11:56.4138610Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:56.4138963Z env: 2025-04-22T00:11:56.4139176Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:56.4139505Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:56.4140046Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:56.4140520Z ##[endgroup] 2025-04-22T00:11:59.4783014Z repo: pytorch/pytorch 2025-04-22T00:11:59.4784434Z Downloading logs-test-inductor_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_40900324323.zip 2025-04-22T00:11:59.4785308Z Converted Log Model: UtilizationMetadata: 2025-04-22T00:11:59.4787405Z UtilizationMetadata(level='metadata', workflow_id='14580756816', job_id='40900324323', workflow_name='inductor', job_name='cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)', usage_collect_interval=1.0, data_model_version=1.0, start_at=1745272231, gpu_count=1, cpu_count=16, gpu_type='pynvml', error=None) 2025-04-22T00:11:59.4789624Z [Db Segments] detected pytest cmd: 20, generated segments: 20 2025-04-22T00:11:59.4790214Z [db model] Peek db timeseries 2025-04-22T00:11:59.4790602Z :{ 2025-04-22T00:11:59.4790919Z "created_at": 1745280718, 2025-04-22T00:11:59.4791314Z "type": "utilization", 2025-04-22T00:11:59.4791668Z "tags": [ 2025-04-22T00:11:59.4791969Z "record" 2025-04-22T00:11:59.4792280Z ], 2025-04-22T00:11:59.4792575Z "time_stamp": 1745272231, 2025-04-22T00:11:59.4792995Z "repo": "pytorch/pytorch", 2025-04-22T00:11:59.4793396Z "workflow_id": 14580756816, 2025-04-22T00:11:59.4793779Z "run_attempt": 1, 2025-04-22T00:11:59.4794144Z "job_id": 40900324323, 2025-04-22T00:11:59.4794856Z "workflow_name": "inductor", 2025-04-22T00:11:59.4795600Z "job_name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)", 2025-04-22T00:11:59.4796849Z "json_data": "{}" 2025-04-22T00:11:59.4797189Z } 2025-04-22T00:11:59.4797866Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/14580756816/1/40900324323/metadata 2025-04-22T00:11:59.4799165Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/14580756816/1/40900324323/metadata 2025-04-22T00:11:59.4800522Z Writing 1691 documents to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/14580756816/1/40900324323/time_series 2025-04-22T00:11:59.4802232Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/14580756816/1/40900324323/time_series 2025-04-22T00:11:59.6016214Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-04-22T00:11:59.6016632Z with: 2025-04-22T00:11:59.6016824Z env: 2025-04-22T00:11:59.6017021Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:59.6017354Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:59.6018032Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:59.6018498Z ##[endgroup] 2025-04-22T00:11:59.6042669Z ##[group]Run set -eou pipefail 2025-04-22T00:11:59.6042973Z set -eou pipefail 2025-04-22T00:11:59.6043226Z  2025-04-22T00:11:59.6043582Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-04-22T00:11:59.6057209Z for _ in $(seq 1440); do 2025-04-22T00:11:59.6057747Z  # Break if no ssh session exists anymore 2025-04-22T00:11:59.6058132Z  if [ "$(who)" = "" ]; then 2025-04-22T00:11:59.6058414Z  break 2025-04-22T00:11:59.6058668Z  fi 2025-04-22T00:11:59.6058882Z  echo "." 2025-04-22T00:11:59.6059114Z  sleep 5 2025-04-22T00:11:59.6059338Z done 2025-04-22T00:11:59.6068026Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:59.6068387Z env: 2025-04-22T00:11:59.6068594Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:59.6068905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:59.6069439Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:59.6069912Z ##[endgroup] 2025-04-22T00:11:59.6099566Z Holding runner for 2 hours until all ssh sessions have logged out 2025-04-22T00:11:59.6175701Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-04-22T00:11:59.6176233Z # ignore expansion of "docker ps -q" since it could be empty 2025-04-22T00:11:59.6176640Z # shellcheck disable=SC2046 2025-04-22T00:11:59.6176975Z docker stop $(docker ps -q) || true 2025-04-22T00:11:59.6177304Z # Prune all of the docker images 2025-04-22T00:11:59.6177778Z docker system prune -af 2025-04-22T00:11:59.6185655Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:11:59.6186020Z env: 2025-04-22T00:11:59.6186227Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:11:59.6186551Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:11:59.6187090Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:11:59.6187562Z ##[endgroup] 2025-04-22T00:12:00.5447271Z a89da563ed2a 2025-04-22T00:12:04.4695209Z Deleted Containers: 2025-04-22T00:12:04.4695657Z a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:12:04.4695987Z 2025-04-22T00:12:17.0699200Z Deleted Images: 2025-04-22T00:12:17.0700362Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 2025-04-22T00:12:17.0702503Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks@sha256:745e08885687216ce5adf3196dea7f49a9467042491210addb990693d7fef90a 2025-04-22T00:12:17.0704070Z deleted: sha256:7b29eacbf93f5bdd9e6b666f27edb34c0d2f289aa58ceaf3bdfcf4e06f505e55 2025-04-22T00:12:17.0704702Z deleted: sha256:0d03769fa66092351763fc95d7ff70e856e1c6dad39dffd06cd3cc6f705c2b90 2025-04-22T00:12:17.0705458Z deleted: sha256:03ad561430a1da2dad623afcca922b9bf999b9beec551534fb227d2905d98683 2025-04-22T00:12:17.0706168Z deleted: sha256:8999b0bbe1ce0aa447deea13eab8ab1f8c7e23d827f1cfb9ea3eda2d5862ff30 2025-04-22T00:12:17.0706837Z deleted: sha256:8c9a34c905c38106c707a3de05f76313f1dc1b5480e33030262e0c036fd86e2e 2025-04-22T00:12:17.0707478Z deleted: sha256:7aa296e90575147a3bdf977176396c671082fb5dc5c240a22ac72b9e1bc0e268 2025-04-22T00:12:17.0708154Z deleted: sha256:35429085a82ca073e718dc4ecc1c57868bb6c3ca77201e12ea81aa653bdde323 2025-04-22T00:12:17.0709068Z deleted: sha256:9a73de98215b59030f9b38fd04ed9f006b41baa1a1df96e1b130fd6771aafa70 2025-04-22T00:12:17.0709778Z deleted: sha256:aa989e7f91a166b7b6f2138d15d3a9eab2852c094c0f76f2ef4b85a015d5247a 2025-04-22T00:12:17.0710478Z deleted: sha256:aab94433009bfa1b6b672cbf53be6cfcc67e12bae3fccb39392f2ba356fed8e8 2025-04-22T00:12:17.0711128Z deleted: sha256:cf4b1db59c30d2e1191b22f938168f28c237ca50a4479f4bd96300300167a10d 2025-04-22T00:12:17.0711778Z deleted: sha256:d80a61f49181681ea66aabe7e2bd1a43e3652d30fbe47699b3c75bc320b3fd86 2025-04-22T00:12:17.0712459Z deleted: sha256:7ce0d2a5b6420f3c29e66c3a4c4e37437e961ea57a801549d154c17377b9cbe7 2025-04-22T00:12:17.0713143Z deleted: sha256:b035e1e0eefb5d757d8642ca66b5a56d8acd60b27793d8bb0fee024b61424f18 2025-04-22T00:12:17.0713830Z deleted: sha256:e9f028a07e4a18a1eb31d1377ffdc18189e530d827ec085900dab9634404cbfd 2025-04-22T00:12:17.0714461Z deleted: sha256:75a2b3d5fb5eabe18f21153ffe34412d0f6adb78a89cbb579271dc29a107109f 2025-04-22T00:12:17.0715139Z deleted: sha256:980637ca90042fdf0d35c1aa5459851186b2d884c583ac0c6018565e4123275d 2025-04-22T00:12:17.0715807Z deleted: sha256:353541bdf9a9dc69f68d46df3e6af379ce0d8fb710e2ae3ed989b4fd73a7f69e 2025-04-22T00:12:17.0716499Z deleted: sha256:9739cf32fe6a869643327bf4f2ddbdfd714451c07e0463e887cfc7827acf81b8 2025-04-22T00:12:17.0717170Z deleted: sha256:ea8da7f58fe158b166472f208d4787bd84a5f24715f3a3693b2ce09db3b56870 2025-04-22T00:12:17.0717783Z deleted: sha256:64d128ae465a7433018bb4f041b913f2c3d4fe4f7a4191fb15a976492613d25a 2025-04-22T00:12:17.0718453Z deleted: sha256:407bfba786761588aea6fe18b2486cd23dddd885d72c60a41949d7dfc6a942d5 2025-04-22T00:12:17.0719123Z deleted: sha256:9889d65ed67766c496c923d00c5e6fcf5d77647d1384b278c39e389fc2a99f71 2025-04-22T00:12:17.0719794Z deleted: sha256:885a49a89eaa7cd605832bad81aaef59d663c8b6a9941c4fd8a56414a5c15361 2025-04-22T00:12:17.0720409Z deleted: sha256:a754629243d15f78ad5b02c613ca5b138bae988a653e77ab7d7a1c2b1ff8430b 2025-04-22T00:12:17.0721096Z deleted: sha256:69986b2cde5dc2eedee2c34cc9a0f2862bd0bb76de325a159b3eb286e31ab267 2025-04-22T00:12:17.0721786Z deleted: sha256:1eb1b82d13079306ebed0331351fb0299aab14b537bd38ac3dfe9f86daba4a7c 2025-04-22T00:12:17.0722477Z deleted: sha256:3a53d5f678b95144bfb2f43df7c8f62c6eacdc36c4c9278d2672658ba981a6f1 2025-04-22T00:12:17.0723143Z deleted: sha256:816f3880d16e691a376b22135e30c5c92e77a2390692a2aec75c92a61a1028d6 2025-04-22T00:12:17.0723745Z deleted: sha256:6e9897a601a0378a8a2499be37549ea2df6a9dfbec84fad00d2e0796b0d8c897 2025-04-22T00:12:17.0724423Z deleted: sha256:4253008065add4d2e0a51dd0dc024420211cfa90f52ccf2fb3cd59a97941e820 2025-04-22T00:12:17.0725099Z deleted: sha256:84fc991bd2259ddde33c56bd6d83cd6856ce393c638a21f6c61183206847f599 2025-04-22T00:12:17.0725761Z deleted: sha256:52c94651685b1b2c788a091c20c8325005b4cb03693709ff02b9e659e790f122 2025-04-22T00:12:17.0726365Z deleted: sha256:afa943b0954b848b45f79913b746502e986aba75e5a7393fb4569f3450da4315 2025-04-22T00:12:17.0727039Z deleted: sha256:b3d00015deda08e79f7f4c3b4bf277c14ddf7cc07986f983b4e160ea069ffc47 2025-04-22T00:12:17.0727728Z deleted: sha256:e3b8a4f8a44b3187a6f78df4ee606a2a8ec8060bfca4163e47c3db94ff685c9e 2025-04-22T00:12:17.0728419Z deleted: sha256:5ad2fc574f42e35faf6c336d17334edc2e67b6601a49fbe3c12000a8091da6e8 2025-04-22T00:12:17.0729233Z deleted: sha256:397028d746ad951508252fafab3946aa1b8825cd1d8edb6c947cc54c9db7576a 2025-04-22T00:12:17.0729866Z deleted: sha256:454a57acb38698129b7675d5e6d4358906230673548a777f8a3dec9c4fab8500 2025-04-22T00:12:17.0730488Z deleted: sha256:95f1c8b10ced79e9ce5a542b118703418c66d52cc43cc8e7c9405916383bb5ac 2025-04-22T00:12:17.0731179Z deleted: sha256:06af89fbc9fb5a1ff9cc0ddaa04378aa25ac8a7d9e16b77b6aa6a48558d10cd6 2025-04-22T00:12:17.0731888Z deleted: sha256:d4d7f04a8b38bc30eb120874f5d53b51aefc2754f34b43cd5bcad3b58f14791d 2025-04-22T00:12:17.0732558Z deleted: sha256:a39fece22c881dde7689a0873a48724a18471faf44ad4de34c5987032a2f8aed 2025-04-22T00:12:17.0733162Z deleted: sha256:0b59474a1f61590a4946eb94287e7b9531ed471d49b34516ada330bf6e62ef3d 2025-04-22T00:12:17.0733982Z deleted: sha256:76af19197c611f8dd2aec8623ffa2ae9dfc96031e57c885acbf763d47cd4e10a 2025-04-22T00:12:17.0734605Z deleted: sha256:cec724490b7de23aa9af56de906b95d0722b33e990443300bc5fbaafd2d6ce7f 2025-04-22T00:12:17.0735211Z deleted: sha256:993012f9b10558b092cbaa6afb2f8de46c856adf9f39ccb3affb1ab88b4034f5 2025-04-22T00:12:17.0735820Z deleted: sha256:c03f3c99e80a992966c8a13067ca8d551914f2016063aabd58e10fc66a0b4aa7 2025-04-22T00:12:17.0736450Z deleted: sha256:a3826acb0870ba3e5dcb72651ebbae184d8964a70dfc99d88a6ada64df52070a 2025-04-22T00:12:17.0737092Z deleted: sha256:59eff532c4a87d4492cfb7a2dca5f091aba58f41f09dd44d78ed251566552fc0 2025-04-22T00:12:17.0737883Z deleted: sha256:7af9312575f7d280f4e404bfa4229d29b2f863308cd2847737dab0a80f912abd 2025-04-22T00:12:17.0738467Z deleted: sha256:255b3dc15181408481cc137ae98284eb8bbe550185f4c3600538c7f27cb97630 2025-04-22T00:12:17.0739049Z deleted: sha256:a46c03834bc35bcd52270359c7503317e2517a1bef96d6174b37510d49240a96 2025-04-22T00:12:17.0739834Z deleted: sha256:697ce89585f3321899a308e7d5fa44cfd01661762b83d4886fe6c7df23067deb 2025-04-22T00:12:17.0740435Z deleted: sha256:fe1a83ec01181b320ed6d1d9d96c7b05e0ec90ecae542be803fc5a55fd49563c 2025-04-22T00:12:17.0741039Z deleted: sha256:47a48329e8ee10fb1592b7f54749c23245dfdbdecb46c08c4b79dd9b141a13b9 2025-04-22T00:12:17.0741642Z deleted: sha256:b7b7528c3adac4746b228449b90274fd02e75b69ff3f626366683dea239108b4 2025-04-22T00:12:17.0742228Z deleted: sha256:b1563735209b77977fec8743d8915d3bb4acecdce565bf90f42e43b7a9d41eba 2025-04-22T00:12:17.0742830Z deleted: sha256:8f3dad83b005c9344155ed29ce5613f918718d2ceab31a2fc20adf404021a9ee 2025-04-22T00:12:17.0743435Z deleted: sha256:d19dff217a30a035796fe482ed413f6eda45d02ca46daa5cd9ce30ac4a5b86a9 2025-04-22T00:12:17.0744042Z deleted: sha256:ec94e4b914db9f6e5913eb4a203bf4c14cf9dffa794aaf0d01426e1d2d540662 2025-04-22T00:12:17.0744636Z deleted: sha256:94e49b5119d7680f10fa65841158de93cc070c7b19bb83c566b874158bec2fc3 2025-04-22T00:12:17.0745226Z deleted: sha256:6866ef40cf6f50d4d4c3823444758606f80314d683f650be1b6d69eff2d35329 2025-04-22T00:12:17.0745822Z deleted: sha256:f06c4dcc941cd4fa9c6b0f507c8be752e7be6b2191a3332128ab32fdb8c613eb 2025-04-22T00:12:17.0746431Z deleted: sha256:e7a9c0fbc4da83cebcbeab6a1ffc42508561041f97259ff2cf2e2e14a73d47bd 2025-04-22T00:12:17.0747044Z deleted: sha256:67f07ac2be3331b5421f04c08746bdb4ce5952c7094018813bdc134ddefd5ec4 2025-04-22T00:12:17.0747637Z deleted: sha256:9a1906e08abd2cbaad6411b296796a97964e0f89011633541b1fd2ac24e8d53a 2025-04-22T00:12:17.0748232Z deleted: sha256:594048c05b4663fff16eed21ea2fe1cba949ca1b0426b134bb275181ab727272 2025-04-22T00:12:17.0748831Z deleted: sha256:9ab83b5a3614325d1733a24e22eeae2e7ecd37cd0d0024660d9b43265e7c496b 2025-04-22T00:12:17.0749425Z deleted: sha256:de82c3b65910240582bad1cbc96209ab68e59c252450a7d42aad3a0673b154f4 2025-04-22T00:12:17.0750014Z deleted: sha256:46bc902662130b8d13f25b0294fac16ca6248034d5ccfaf3ee51cb56572946c5 2025-04-22T00:12:17.0750596Z deleted: sha256:43659ee56fac7f221f9388ec263ad2042847103213025f0a7a0ec7ca6a81b20f 2025-04-22T00:12:17.0751185Z deleted: sha256:5839c16cf80c38c4f69268b2c73c3d7ec37a33e18c9c5f49e6624239a182dbf8 2025-04-22T00:12:17.0751769Z deleted: sha256:e16d8e25c4393c96891af6d6a42889638fc30a7490ef9c5c5517798ea9860bda 2025-04-22T00:12:17.0752356Z deleted: sha256:868539926d1d83fe2b4fdb0231d8b57fb5b8b34144d733a4cf862d241e16c667 2025-04-22T00:12:17.0753036Z deleted: sha256:cc4224d01139aef3aeb29c568d184283602d52f47bb8e5e29de97bf6aa54d951 2025-04-22T00:12:17.0753632Z deleted: sha256:bd18b4fe898c9279c6e9ad952165522e24de55ab48d83289ca0ff6c8df97d85d 2025-04-22T00:12:17.0754227Z deleted: sha256:c7917cebb757d994f33b0f4bbad70fbb563939c2630d8f96113d437fa09e68cd 2025-04-22T00:12:17.0754822Z deleted: sha256:34fb1ecb50c48f883ea450e0364c90bdf16ce5e35e8a132012851ac3b040addf 2025-04-22T00:12:17.0755422Z deleted: sha256:92300fc3f1a943c0b6a295d87da98cb3d593b18b8ddc4f0b229e3ca909d03206 2025-04-22T00:12:17.0756027Z deleted: sha256:fffe76c64ef2dee2d80a8bb3ad13d65d596d04a45510b1956a976a69215dae92 2025-04-22T00:12:17.0756387Z 2025-04-22T00:12:17.0756598Z Total reclaimed space: 51.31GB 2025-04-22T00:12:17.0798532Z ##[group]Run set +e 2025-04-22T00:12:17.0798856Z set +e 2025-04-22T00:12:17.0799084Z set -x 2025-04-22T00:12:17.0799300Z  2025-04-22T00:12:17.0799503Z nvidia-smi 2025-04-22T00:12:17.0799948Z # NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in 2025-04-22T00:12:17.0800608Z # the case where the driver has already crashed as it still can get the driver version 2025-04-22T00:12:17.0801257Z # and some basic information like the bus ID. However, the rest of the information 2025-04-22T00:12:17.0801752Z # would be missing (ERR!), for example: 2025-04-22T00:12:17.0802068Z # 2025-04-22T00:12:17.0802366Z # +-----------------------------------------------------------------------------+ 2025-04-22T00:12:17.0802875Z # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | 2025-04-22T00:12:17.0803410Z # |-------------------------------+----------------------+----------------------+ 2025-04-22T00:12:17.0803923Z # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-04-22T00:12:17.0804486Z # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | 2025-04-22T00:12:17.0804956Z # | | | MIG M. | 2025-04-22T00:12:17.0805318Z # |===============================+======================+======================| 2025-04-22T00:12:17.0805723Z # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | 2025-04-22T00:12:17.0806180Z # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | 2025-04-22T00:12:17.0806604Z # | | | ERR! | 2025-04-22T00:12:17.0807012Z # +-------------------------------+----------------------+----------------------+ 2025-04-22T00:12:17.0807368Z # 2025-04-22T00:12:17.0807662Z # +-----------------------------------------------------------------------------+ 2025-04-22T00:12:17.0808108Z # | Processes: | 2025-04-22T00:12:17.0808570Z # | GPU GI CI PID Type Process name GPU Memory | 2025-04-22T00:12:17.0808999Z # | ID ID Usage | 2025-04-22T00:12:17.0809373Z # |=============================================================================| 2025-04-22T00:12:17.0809779Z # +-----------------------------------------------------------------------------+ 2025-04-22T00:12:17.0810142Z # 2025-04-22T00:12:17.0810516Z # This should be reported as a failure instead as it will guarantee to fail when 2025-04-22T00:12:17.0811015Z # Docker tries to run with --gpus all 2025-04-22T00:12:17.0811333Z # 2025-04-22T00:12:17.0811687Z # So, the correct check here is to query one of the missing piece of info like 2025-04-22T00:12:17.0812183Z # GPU name, so that the command can fail accordingly 2025-04-22T00:12:17.0812827Z nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-04-22T00:12:17.0813241Z NVIDIA_SMI_STATUS=$? 2025-04-22T00:12:17.0813500Z  2025-04-22T00:12:17.0813923Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-04-22T00:12:17.0814554Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-04-22T00:12:17.0815117Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-04-22T00:12:17.0815602Z  .github/scripts/stop_runner_service.sh 2025-04-22T00:12:17.0815949Z fi 2025-04-22T00:12:17.0816192Z  2025-04-22T00:12:17.0816727Z # For runner with multiple GPUs, we also want to confirm that the number of GPUs are the 2025-04-22T00:12:17.0817337Z # power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails 2025-04-22T00:12:17.0818038Z # https://github.com/pytorch/test-infra/issues/4000 2025-04-22T00:12:17.0818464Z GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) 2025-04-22T00:12:17.0818803Z NVIDIA_SMI_STATUS=$? 2025-04-22T00:12:17.0819066Z  2025-04-22T00:12:17.0819484Z # These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action 2025-04-22T00:12:17.0820098Z if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then 2025-04-22T00:12:17.0820644Z  echo "NVIDIA driver installation has failed, shutting down the runner..." 2025-04-22T00:12:17.0821127Z  .github/scripts/stop_runner_service.sh 2025-04-22T00:12:17.0821441Z fi 2025-04-22T00:12:17.0821649Z  2025-04-22T00:12:17.0821891Z # Check the GPU count to be a power of 2 2025-04-22T00:12:17.0822424Z if [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then 2025-04-22T00:12:17.0823152Z  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..." 2025-04-22T00:12:17.0823704Z  .github/scripts/stop_runner_service.sh 2025-04-22T00:12:17.0824023Z fi 2025-04-22T00:12:17.0835739Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:12:17.0836089Z env: 2025-04-22T00:12:17.0836297Z GIT_DEFAULT_BRANCH: main 2025-04-22T00:12:17.0836614Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2025-04-22T00:12:17.0837145Z DOCKER_CONTAINER_ID: a89da563ed2a2e41bacfa4f415aa073edd8078b0c86f6e8d30b874ea72338cee 2025-04-22T00:12:17.0837612Z ##[endgroup] 2025-04-22T00:12:17.0871665Z + nvidia-smi 2025-04-22T00:12:17.1060772Z Tue Apr 22 00:12:17 2025 2025-04-22T00:12:17.1061511Z +-----------------------------------------------------------------------------------------+ 2025-04-22T00:12:17.1062512Z | NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 | 2025-04-22T00:12:17.1063494Z |-----------------------------------------+------------------------+----------------------+ 2025-04-22T00:12:17.1064498Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2025-04-22T00:12:17.1065571Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2025-04-22T00:12:17.1066297Z | | | MIG M. | 2025-04-22T00:12:17.1066636Z |=========================================+========================+======================| 2025-04-22T00:12:17.1219793Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2025-04-22T00:12:17.1220266Z | 0% 29C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2025-04-22T00:12:17.1220698Z | | | N/A | 2025-04-22T00:12:17.1221276Z +-----------------------------------------+------------------------+----------------------+ 2025-04-22T00:12:17.1224630Z 2025-04-22T00:12:17.1225036Z +-----------------------------------------------------------------------------------------+ 2025-04-22T00:12:17.1225485Z | Processes: | 2025-04-22T00:12:17.1225933Z | GPU GI CI PID Type Process name GPU Memory | 2025-04-22T00:12:17.1226360Z | ID ID Usage | 2025-04-22T00:12:17.1226878Z |=========================================================================================| 2025-04-22T00:12:17.1229605Z | No running processes found | 2025-04-22T00:12:17.1230083Z +-----------------------------------------------------------------------------------------+ 2025-04-22T00:12:17.3622191Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2025-04-22T00:12:17.3791679Z NVIDIA A10G 2025-04-22T00:12:17.3836702Z + NVIDIA_SMI_STATUS=0 2025-04-22T00:12:17.3837430Z + '[' 0 -ne 0 ']' 2025-04-22T00:12:17.3843986Z ++ nvidia-smi --list-gpus 2025-04-22T00:12:17.3844470Z ++ wc -l 2025-04-22T00:12:17.4065801Z + GPU_COUNT=1 2025-04-22T00:12:17.4066081Z + NVIDIA_SMI_STATUS=0 2025-04-22T00:12:17.4066333Z + '[' 0 -ne 0 ']' 2025-04-22T00:12:17.4066554Z + '[' 1 -le 8 ']' 2025-04-22T00:12:17.4066781Z + '[' 1 -ne 1 ']' 2025-04-22T00:12:17.4156087Z Post job cleanup. 2025-04-22T00:12:17.4212908Z Post job cleanup. 2025-04-22T00:12:17.5254850Z [command]/usr/bin/git version 2025-04-22T00:12:17.5319210Z git version 2.47.1 2025-04-22T00:12:17.5358282Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/f1a28856-720f-4baf-bbfa-3136e0b90931/.gitconfig' 2025-04-22T00:12:17.5369557Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f1a28856-720f-4baf-bbfa-3136e0b90931' before making global git config changes 2025-04-22T00:12:17.5370480Z Adding repository directory to the temporary git global config as a safe directory 2025-04-22T00:12:17.5376055Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-04-22T00:12:17.5424909Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-04-22T00:12:17.5461811Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-04-22T00:12:17.5852303Z Entering 'android/libs/fbjni' 2025-04-22T00:12:17.5923679Z Entering 'third_party/FP16' 2025-04-22T00:12:17.5992035Z Entering 'third_party/FXdiv' 2025-04-22T00:12:17.6060907Z Entering 'third_party/NNPACK' 2025-04-22T00:12:17.6130586Z Entering 'third_party/NVTX' 2025-04-22T00:12:17.6200410Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-22T00:12:17.6271064Z Entering 'third_party/XNNPACK' 2025-04-22T00:12:17.6353309Z Entering 'third_party/benchmark' 2025-04-22T00:12:17.6424175Z Entering 'third_party/composable_kernel' 2025-04-22T00:12:17.6499967Z Entering 'third_party/cpp-httplib' 2025-04-22T00:12:17.6570450Z Entering 'third_party/cpuinfo' 2025-04-22T00:12:17.6641257Z Entering 'third_party/cudnn_frontend' 2025-04-22T00:12:17.6709112Z Entering 'third_party/cutlass' 2025-04-22T00:12:17.6787831Z Entering 'third_party/eigen' 2025-04-22T00:12:17.6860083Z Entering 'third_party/fbgemm' 2025-04-22T00:12:17.6929662Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-22T00:12:17.6996183Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-22T00:12:17.7062952Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-22T00:12:17.7136241Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-22T00:12:17.7201729Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-22T00:12:17.7272048Z Entering 'third_party/flash-attention' 2025-04-22T00:12:17.7342392Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-22T00:12:17.7416873Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-22T00:12:17.7495088Z Entering 'third_party/flatbuffers' 2025-04-22T00:12:17.7565795Z Entering 'third_party/fmt' 2025-04-22T00:12:17.7634274Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-22T00:12:17.7704634Z Entering 'third_party/gloo' 2025-04-22T00:12:17.7773929Z Entering 'third_party/googletest' 2025-04-22T00:12:17.7847931Z Entering 'third_party/ideep' 2025-04-22T00:12:17.7918271Z Entering 'third_party/ideep/mkl-dnn' 2025-04-22T00:12:17.7997649Z Entering 'third_party/ittapi' 2025-04-22T00:12:17.8067267Z Entering 'third_party/kineto' 2025-04-22T00:12:17.8134966Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-22T00:12:17.8201221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-22T00:12:17.8270787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-22T00:12:17.8338070Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-22T00:12:17.8410744Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-22T00:12:17.8477571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-22T00:12:17.8549372Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-22T00:12:17.8618230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-22T00:12:17.8685380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-22T00:12:17.8756698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-22T00:12:17.8827733Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-22T00:12:17.8893401Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-22T00:12:17.8963201Z Entering 'third_party/kleidiai' 2025-04-22T00:12:17.9034417Z Entering 'third_party/mimalloc' 2025-04-22T00:12:17.9103925Z Entering 'third_party/nlohmann' 2025-04-22T00:12:17.9174893Z Entering 'third_party/onnx' 2025-04-22T00:12:17.9258946Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-22T00:12:17.9332813Z Entering 'third_party/opentelemetry-cpp' 2025-04-22T00:12:17.9403763Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-22T00:12:17.9472330Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-22T00:12:17.9540452Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-22T00:12:17.9608387Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-22T00:12:17.9675839Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-22T00:12:17.9744415Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-22T00:12:17.9810940Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-22T00:12:17.9875802Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-22T00:12:17.9945522Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-22T00:12:18.0015820Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-22T00:12:18.0108345Z Entering 'third_party/pocketfft' 2025-04-22T00:12:18.0177411Z Entering 'third_party/protobuf' 2025-04-22T00:12:18.0250790Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-22T00:12:18.0319754Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-22T00:12:18.0390779Z Entering 'third_party/psimd' 2025-04-22T00:12:18.0460156Z Entering 'third_party/pthreadpool' 2025-04-22T00:12:18.0529618Z Entering 'third_party/pybind11' 2025-04-22T00:12:18.0600495Z Entering 'third_party/python-peachpy' 2025-04-22T00:12:18.0670599Z Entering 'third_party/sleef' 2025-04-22T00:12:18.0741318Z Entering 'third_party/tensorpipe' 2025-04-22T00:12:18.0805094Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-22T00:12:18.0869626Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-22T00:12:18.0935461Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-22T00:12:18.0999729Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-22T00:12:18.1061660Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-22T00:12:18.1155493Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-04-22T00:12:18.1179400Z http.https://github.com/.extraheader 2025-04-22T00:12:18.1191307Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-04-22T00:12:18.1226059Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-04-22T00:12:18.1569571Z Entering 'android/libs/fbjni' 2025-04-22T00:12:18.1617886Z http.https://github.com/.extraheader 2025-04-22T00:12:18.1663010Z Entering 'third_party/FP16' 2025-04-22T00:12:18.1705272Z http.https://github.com/.extraheader 2025-04-22T00:12:18.1748877Z Entering 'third_party/FXdiv' 2025-04-22T00:12:18.1791582Z http.https://github.com/.extraheader 2025-04-22T00:12:18.1840208Z Entering 'third_party/NNPACK' 2025-04-22T00:12:18.1886110Z http.https://github.com/.extraheader 2025-04-22T00:12:18.1931522Z Entering 'third_party/NVTX' 2025-04-22T00:12:18.1975730Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2022730Z Entering 'third_party/VulkanMemoryAllocator' 2025-04-22T00:12:18.2068069Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2114193Z Entering 'third_party/XNNPACK' 2025-04-22T00:12:18.2158920Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2217920Z Entering 'third_party/benchmark' 2025-04-22T00:12:18.2263143Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2308782Z Entering 'third_party/composable_kernel' 2025-04-22T00:12:18.2354490Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2404468Z Entering 'third_party/cpp-httplib' 2025-04-22T00:12:18.2449425Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2493865Z Entering 'third_party/cpuinfo' 2025-04-22T00:12:18.2539026Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2582748Z Entering 'third_party/cudnn_frontend' 2025-04-22T00:12:18.2627165Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2670707Z Entering 'third_party/cutlass' 2025-04-22T00:12:18.2715166Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2768296Z Entering 'third_party/eigen' 2025-04-22T00:12:18.2811772Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2857855Z Entering 'third_party/fbgemm' 2025-04-22T00:12:18.2902266Z http.https://github.com/.extraheader 2025-04-22T00:12:18.2947293Z Entering 'third_party/fbgemm/third_party/asmjit' 2025-04-22T00:12:18.2989077Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3034707Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2025-04-22T00:12:18.3076968Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3119954Z Entering 'third_party/fbgemm/third_party/cutlass' 2025-04-22T00:12:18.3161400Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3211333Z Entering 'third_party/fbgemm/third_party/googletest' 2025-04-22T00:12:18.3253308Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3299928Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2025-04-22T00:12:18.3345688Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3393878Z Entering 'third_party/flash-attention' 2025-04-22T00:12:18.3437556Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3481013Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-04-22T00:12:18.3524021Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3575485Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-04-22T00:12:18.3617345Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3671064Z Entering 'third_party/flatbuffers' 2025-04-22T00:12:18.3714760Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3761084Z Entering 'third_party/fmt' 2025-04-22T00:12:18.3805288Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3849324Z Entering 'third_party/gemmlowp/gemmlowp' 2025-04-22T00:12:18.3892189Z http.https://github.com/.extraheader 2025-04-22T00:12:18.3937044Z Entering 'third_party/gloo' 2025-04-22T00:12:18.3986044Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4031272Z Entering 'third_party/googletest' 2025-04-22T00:12:18.4074794Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4118175Z Entering 'third_party/ideep' 2025-04-22T00:12:18.4162050Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4203677Z Entering 'third_party/ideep/mkl-dnn' 2025-04-22T00:12:18.4246254Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4299020Z Entering 'third_party/ittapi' 2025-04-22T00:12:18.4342435Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4385271Z Entering 'third_party/kineto' 2025-04-22T00:12:18.4430044Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4472206Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-04-22T00:12:18.4515306Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4557974Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-04-22T00:12:18.4605087Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4650563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-04-22T00:12:18.4693633Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4740514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-04-22T00:12:18.4782945Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4828590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-04-22T00:12:18.4872351Z http.https://github.com/.extraheader 2025-04-22T00:12:18.4915146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-04-22T00:12:18.4958619Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5006834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-04-22T00:12:18.5052169Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5097090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-04-22T00:12:18.5144409Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5188069Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-04-22T00:12:18.5232252Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5277837Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-04-22T00:12:18.5320921Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5367407Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-04-22T00:12:18.5415141Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5462296Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-04-22T00:12:18.5507381Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5553256Z Entering 'third_party/kleidiai' 2025-04-22T00:12:18.5598833Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5643045Z Entering 'third_party/mimalloc' 2025-04-22T00:12:18.5687275Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5733608Z Entering 'third_party/nlohmann' 2025-04-22T00:12:18.5776497Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5824963Z Entering 'third_party/onnx' 2025-04-22T00:12:18.5868718Z http.https://github.com/.extraheader 2025-04-22T00:12:18.5927938Z Entering 'third_party/onnx/third_party/pybind11' 2025-04-22T00:12:18.5971496Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6022972Z Entering 'third_party/opentelemetry-cpp' 2025-04-22T00:12:18.6066053Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6111778Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-04-22T00:12:18.6154295Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6198244Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-04-22T00:12:18.6240282Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6282798Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-04-22T00:12:18.6326963Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6370107Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-04-22T00:12:18.6412531Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6457945Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-04-22T00:12:18.6500736Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6542993Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-04-22T00:12:18.6584994Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6628181Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-04-22T00:12:18.6671094Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6714222Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-04-22T00:12:18.6757944Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6803301Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-04-22T00:12:18.6846601Z http.https://github.com/.extraheader 2025-04-22T00:12:18.6892849Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-04-22T00:12:18.6935122Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7001633Z Entering 'third_party/pocketfft' 2025-04-22T00:12:18.7046237Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7088892Z Entering 'third_party/protobuf' 2025-04-22T00:12:18.7134413Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7178542Z Entering 'third_party/protobuf/third_party/benchmark' 2025-04-22T00:12:18.7223922Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7267146Z Entering 'third_party/protobuf/third_party/googletest' 2025-04-22T00:12:18.7309857Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7356423Z Entering 'third_party/psimd' 2025-04-22T00:12:18.7405408Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7449127Z Entering 'third_party/pthreadpool' 2025-04-22T00:12:18.7494718Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7537907Z Entering 'third_party/pybind11' 2025-04-22T00:12:18.7581771Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7628277Z Entering 'third_party/python-peachpy' 2025-04-22T00:12:18.7673418Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7717463Z Entering 'third_party/sleef' 2025-04-22T00:12:18.7761149Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7807166Z Entering 'third_party/tensorpipe' 2025-04-22T00:12:18.7851210Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7892631Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-04-22T00:12:18.7935778Z http.https://github.com/.extraheader 2025-04-22T00:12:18.7979241Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-04-22T00:12:18.8020764Z http.https://github.com/.extraheader 2025-04-22T00:12:18.8062998Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-04-22T00:12:18.8106199Z http.https://github.com/.extraheader 2025-04-22T00:12:18.8149491Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-04-22T00:12:18.8191531Z http.https://github.com/.extraheader 2025-04-22T00:12:18.8233705Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-04-22T00:12:18.8277780Z http.https://github.com/.extraheader 2025-04-22T00:12:18.8440624Z A job completed hook has been configured by the self-hosted runner administrator 2025-04-22T00:12:18.8467889Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-04-22T00:12:18.8475706Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-04-22T00:12:18.8476074Z ##[endgroup] 2025-04-22T00:12:18.8582373Z [!ALERT!] Swap in detected! [!ALERT!] 2025-04-22T00:12:36.5669107Z Cleaning up orphan processes