2024-06-26T07:00:42.7480935Z Current runner version: '2.317.0' 2024-06-26T07:00:42.7486882Z Runner name: 'i-0189c918a23a6d1e2' 2024-06-26T07:00:42.7487673Z Runner group name: 'Default' 2024-06-26T07:00:42.7488512Z Machine name: 'ip-10-0-2-209' 2024-06-26T07:00:42.7493225Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:00:42.7495198Z Actions: read 2024-06-26T07:00:42.7495680Z Attestations: read 2024-06-26T07:00:42.7496192Z Checks: read 2024-06-26T07:00:42.7496642Z Contents: read 2024-06-26T07:00:42.7497117Z Deployments: read 2024-06-26T07:00:42.7497596Z Discussions: read 2024-06-26T07:00:42.7498114Z Issues: read 2024-06-26T07:00:42.7498564Z Metadata: read 2024-06-26T07:00:42.7499030Z Packages: read 2024-06-26T07:00:42.7499496Z Pages: read 2024-06-26T07:00:42.7499996Z PullRequests: read 2024-06-26T07:00:42.7500578Z RepositoryProjects: read 2024-06-26T07:00:42.7501197Z SecurityEvents: read 2024-06-26T07:00:42.7501703Z Statuses: read 2024-06-26T07:00:42.7502201Z ##[endgroup] 2024-06-26T07:00:42.7505388Z Secret source: Actions 2024-06-26T07:00:42.7506183Z Prepare workflow directory 2024-06-26T07:00:43.1024782Z Prepare all required actions 2024-06-26T07:00:43.1194976Z Getting action download info 2024-06-26T07:00:43.2994582Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:00:43.6772396Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T07:00:46.5055798Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T07:00:46.6543306Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T07:00:46.9319045Z Getting action download info 2024-06-26T07:00:47.0276540Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:00:47.1990038Z Getting action download info 2024-06-26T07:00:47.2973777Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:00:47.5006354Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:00:47.5008622Z ##[group] Inputs 2024-06-26T07:00:47.5009180Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:00:47.5017648Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:00:47.5027062Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:00:47.5028283Z sync-tag: 2024-06-26T07:00:47.5029209Z timeout-minutes: 240 2024-06-26T07:00:47.5029620Z use-gha: 2024-06-26T07:00:47.5029969Z dashboard-tag: 2024-06-26T07:00:47.5030357Z s3-bucket: gha-artifacts 2024-06-26T07:00:47.5030779Z aws-role-to-assume: 2024-06-26T07:00:47.5031187Z ##[endgroup] 2024-06-26T07:00:47.5032419Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:00:47.5625307Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T07:00:47.5767343Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T07:00:47.5777924Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:00:47.5778536Z ##[endgroup] 2024-06-26T07:00:48.1382390Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T07:00:48.1383007Z with: 2024-06-26T07:00:48.1383615Z github-secret: *** 2024-06-26T07:00:48.1384536Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T07:00:48.1385521Z activate-with-label: false 2024-06-26T07:00:48.1385884Z label: with-ssh 2024-06-26T07:00:48.1386218Z remove-existing-keys: true 2024-06-26T07:00:48.1386609Z fail-silently: true 2024-06-26T07:00:48.1386930Z env: 2024-06-26T07:00:48.1387221Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:48.1387570Z ##[endgroup] 2024-06-26T07:00:48.2231017Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T07:00:48.2232369Z ciflow reference detected, attempting to extract PR number 2024-06-26T07:00:48.5366896Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T07:00:48.5957875Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T07:00:48.5958707Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T07:00:48.6758651Z No SSH keys found for user leslie-fang-intel 2024-06-26T07:00:48.6865168Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:00:48.6865927Z with: 2024-06-26T07:00:48.6866313Z submodules: recursive 2024-06-26T07:00:48.6866765Z fetch-depth: 0 2024-06-26T07:00:48.6867168Z env: 2024-06-26T07:00:48.6867555Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:48.6868024Z ##[endgroup] 2024-06-26T07:00:48.7062292Z ##[group]Run retry () { 2024-06-26T07:00:48.7062798Z retry () { 2024-06-26T07:00:48.7063448Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:00:48.7064122Z } 2024-06-26T07:00:48.7064571Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:00:48.7065122Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:00:48.7065726Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:00:48.7066299Z else 2024-06-26T07:00:48.7094408Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:00:48.7094963Z fi 2024-06-26T07:00:48.7095402Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:00:48.7103767Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:00:48.7104351Z env: 2024-06-26T07:00:48.7104697Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:48.7105112Z NO_SUDO: 2024-06-26T07:00:48.7105449Z ##[endgroup] 2024-06-26T07:00:48.7128429Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:50.8437455Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:00:50.8437891Z with: 2024-06-26T07:00:50.8438207Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:50.8438638Z fetch-depth: 0 2024-06-26T07:00:50.8438945Z submodules: recursive 2024-06-26T07:00:50.8439283Z quiet-checkout: true 2024-06-26T07:00:50.8439620Z repository: pytorch/pytorch 2024-06-26T07:00:50.8440097Z token: *** 2024-06-26T07:00:50.8440384Z ssh-strict: true 2024-06-26T07:00:50.8440708Z persist-credentials: true 2024-06-26T07:00:50.8441055Z clean: true 2024-06-26T07:00:50.8441387Z sparse-checkout-cone-mode: true 2024-06-26T07:00:50.8441771Z lfs: false 2024-06-26T07:00:50.8442067Z set-safe-directory: true 2024-06-26T07:00:50.8442403Z env: 2024-06-26T07:00:50.8442673Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:00:50.8443010Z ##[endgroup] 2024-06-26T07:00:50.9342284Z Syncing repository: pytorch/pytorch 2024-06-26T07:00:50.9343816Z ##[group]Getting Git version info 2024-06-26T07:00:50.9344621Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:00:50.9345452Z [command]/usr/bin/git version 2024-06-26T07:00:50.9345820Z git version 2.40.1 2024-06-26T07:00:50.9347171Z ##[endgroup] 2024-06-26T07:00:50.9358293Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/4f46f900-a8bb-480d-bb19-fce807b780f8' before making global git config changes 2024-06-26T07:00:50.9359516Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:00:50.9360628Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:50.9392396Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:00:50.9396391Z ##[group]Initializing the repository 2024-06-26T07:00:50.9399222Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:00:50.9419096Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:00:50.9420364Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:00:50.9421864Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:00:50.9422423Z hint: 2024-06-26T07:00:50.9423055Z hint: git config --global init.defaultBranch 2024-06-26T07:00:50.9423525Z hint: 2024-06-26T07:00:50.9424026Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:00:50.9424844Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:00:50.9425606Z hint: 2024-06-26T07:00:50.9425925Z hint: git branch -m 2024-06-26T07:00:50.9426658Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:00:50.9428422Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:00:50.9448931Z ##[endgroup] 2024-06-26T07:00:50.9449664Z ##[group]Disabling automatic garbage collection 2024-06-26T07:00:50.9451076Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:00:50.9474963Z ##[endgroup] 2024-06-26T07:00:50.9475613Z ##[group]Setting up auth 2024-06-26T07:00:50.9480558Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:00:50.9500234Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:00:50.9695066Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:00:50.9716939Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:00:50.9906071Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:00:50.9942888Z ##[endgroup] 2024-06-26T07:00:50.9943425Z ##[group]Fetching the repository 2024-06-26T07:00:50.9949376Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:00:53.3949048Z remote: Enumerating objects: 987619 2024-06-26T07:00:53.3949853Z remote: Enumerating objects: 990209, done. 2024-06-26T07:00:53.3950787Z remote: Counting objects: 0% (1/2590) 2024-06-26T07:00:53.3952728Z remote: Counting objects: 1% (26/2590) 2024-06-26T07:00:53.3954268Z remote: Counting objects: 2% (52/2590) 2024-06-26T07:00:53.3955652Z remote: Counting objects: 3% (78/2590) 2024-06-26T07:00:53.3956431Z remote: Counting objects: 4% (104/2590) 2024-06-26T07:00:53.3957346Z remote: Counting objects: 5% (130/2590) 2024-06-26T07:00:53.3958170Z remote: Counting objects: 6% (156/2590) 2024-06-26T07:00:53.3959225Z remote: Counting objects: 7% (182/2590) 2024-06-26T07:00:53.3959964Z remote: Counting objects: 8% (208/2590) 2024-06-26T07:00:53.3960551Z remote: Counting objects: 9% (234/2590) 2024-06-26T07:00:53.3961124Z remote: Counting objects: 10% (259/2590) 2024-06-26T07:00:53.3963453Z remote: Counting objects: 11% (285/2590) 2024-06-26T07:00:53.3964129Z remote: Counting objects: 12% (311/2590) 2024-06-26T07:00:53.3964702Z remote: Counting objects: 13% (337/2590) 2024-06-26T07:00:53.3965270Z remote: Counting objects: 14% (363/2590) 2024-06-26T07:00:53.3967310Z remote: Counting objects: 15% (389/2590) 2024-06-26T07:00:53.3968075Z remote: Counting objects: 16% (415/2590) 2024-06-26T07:00:53.3968704Z remote: Counting objects: 17% (441/2590) 2024-06-26T07:00:53.3969290Z remote: Counting objects: 18% (467/2590) 2024-06-26T07:00:53.3969869Z remote: Counting objects: 19% (493/2590) 2024-06-26T07:00:53.3970585Z remote: Counting objects: 20% (518/2590) 2024-06-26T07:00:53.3971318Z remote: Counting objects: 21% (544/2590) 2024-06-26T07:00:53.3971903Z remote: Counting objects: 22% (570/2590) 2024-06-26T07:00:53.3972467Z remote: Counting objects: 23% (596/2590) 2024-06-26T07:00:53.3973115Z remote: Counting objects: 24% (622/2590) 2024-06-26T07:00:53.3973918Z remote: Counting objects: 25% (648/2590) 2024-06-26T07:00:53.3974699Z remote: Counting objects: 26% (674/2590) 2024-06-26T07:00:53.3975409Z remote: Counting objects: 27% (700/2590) 2024-06-26T07:00:53.3976506Z remote: Counting objects: 28% (726/2590) 2024-06-26T07:00:53.3977159Z remote: Counting objects: 29% (752/2590) 2024-06-26T07:00:53.3977753Z remote: Counting objects: 30% (777/2590) 2024-06-26T07:00:53.3978338Z remote: Counting objects: 31% (803/2590) 2024-06-26T07:00:53.3978929Z remote: Counting objects: 32% (829/2590) 2024-06-26T07:00:53.3979573Z remote: Counting objects: 33% (855/2590) 2024-06-26T07:00:53.3980299Z remote: Counting objects: 34% (881/2590) 2024-06-26T07:00:53.3981065Z remote: Counting objects: 35% (907/2590) 2024-06-26T07:00:53.3981662Z remote: Counting objects: 36% (933/2590) 2024-06-26T07:00:53.3982396Z remote: Counting objects: 37% (959/2590) 2024-06-26T07:00:53.3983058Z remote: Counting objects: 38% (985/2590) 2024-06-26T07:00:53.3983653Z remote: Counting objects: 39% (1011/2590) 2024-06-26T07:00:53.3984379Z remote: Counting objects: 40% (1036/2590) 2024-06-26T07:00:53.3985042Z remote: Counting objects: 41% (1062/2590) 2024-06-26T07:00:53.3985695Z remote: Counting objects: 42% (1088/2590) 2024-06-26T07:00:53.3986438Z remote: Counting objects: 43% (1114/2590) 2024-06-26T07:00:53.3987045Z remote: Counting objects: 44% (1140/2590) 2024-06-26T07:00:53.3987640Z remote: Counting objects: 45% (1166/2590) 2024-06-26T07:00:53.3988390Z remote: Counting objects: 46% (1192/2590) 2024-06-26T07:00:53.3988977Z remote: Counting objects: 47% (1218/2590) 2024-06-26T07:00:53.3989569Z remote: Counting objects: 48% (1244/2590) 2024-06-26T07:00:53.3990161Z remote: Counting objects: 49% (1270/2590) 2024-06-26T07:00:53.3990759Z remote: Counting objects: 50% (1295/2590) 2024-06-26T07:00:53.3991350Z remote: Counting objects: 51% (1321/2590) 2024-06-26T07:00:53.3992193Z remote: Counting objects: 52% (1347/2590) 2024-06-26T07:00:53.3992788Z remote: Counting objects: 53% (1373/2590) 2024-06-26T07:00:53.3993370Z remote: Counting objects: 54% (1399/2590) 2024-06-26T07:00:53.3993944Z remote: Counting objects: 55% (1425/2590) 2024-06-26T07:00:53.3994529Z remote: Counting objects: 56% (1451/2590) 2024-06-26T07:00:53.3995115Z remote: Counting objects: 57% (1477/2590) 2024-06-26T07:00:53.3995696Z remote: Counting objects: 58% (1503/2590) 2024-06-26T07:00:53.3996277Z remote: Counting objects: 59% (1529/2590) 2024-06-26T07:00:53.3996859Z remote: Counting objects: 60% (1554/2590) 2024-06-26T07:00:53.3997439Z remote: Counting objects: 61% (1580/2590) 2024-06-26T07:00:53.3998025Z remote: Counting objects: 62% (1606/2590) 2024-06-26T07:00:53.3998622Z remote: Counting objects: 63% (1632/2590) 2024-06-26T07:00:53.3999199Z remote: Counting objects: 64% (1658/2590) 2024-06-26T07:00:53.3999781Z remote: Counting objects: 65% (1684/2590) 2024-06-26T07:00:53.4000366Z remote: Counting objects: 66% (1710/2590) 2024-06-26T07:00:53.4000950Z remote: Counting objects: 67% (1736/2590) 2024-06-26T07:00:53.4001526Z remote: Counting objects: 68% (1762/2590) 2024-06-26T07:00:53.4002107Z remote: Counting objects: 69% (1788/2590) 2024-06-26T07:00:53.4002697Z remote: Counting objects: 70% (1813/2590) 2024-06-26T07:00:53.4003287Z remote: Counting objects: 71% (1839/2590) 2024-06-26T07:00:53.4003862Z remote: Counting objects: 72% (1865/2590) 2024-06-26T07:00:53.4004445Z remote: Counting objects: 73% (1891/2590) 2024-06-26T07:00:53.4005030Z remote: Counting objects: 74% (1917/2590) 2024-06-26T07:00:53.4005617Z remote: Counting objects: 75% (1943/2590) 2024-06-26T07:00:53.4006247Z remote: Counting objects: 76% (1969/2590) 2024-06-26T07:00:53.4013090Z remote: Counting objects: 77% (1995/2590) 2024-06-26T07:00:53.4013689Z remote: Counting objects: 78% (2021/2590) 2024-06-26T07:00:53.4014380Z remote: Counting objects: 79% (2047/2590) 2024-06-26T07:00:53.4014977Z remote: Counting objects: 80% (2072/2590) 2024-06-26T07:00:53.4015563Z remote: Counting objects: 81% (2098/2590) 2024-06-26T07:00:53.4016153Z remote: Counting objects: 82% (2124/2590) 2024-06-26T07:00:53.4016742Z remote: Counting objects: 83% (2150/2590) 2024-06-26T07:00:53.4017354Z remote: Counting objects: 84% (2176/2590) 2024-06-26T07:00:53.4017939Z remote: Counting objects: 85% (2202/2590) 2024-06-26T07:00:53.4018529Z remote: Counting objects: 86% (2228/2590) 2024-06-26T07:00:53.4019117Z remote: Counting objects: 87% (2254/2590) 2024-06-26T07:00:53.4022163Z remote: Counting objects: 88% (2280/2590) 2024-06-26T07:00:53.4022746Z remote: Counting objects: 89% (2306/2590) 2024-06-26T07:00:53.4023339Z remote: Counting objects: 90% (2331/2590) 2024-06-26T07:00:53.4023926Z remote: Counting objects: 91% (2357/2590) 2024-06-26T07:00:53.4024516Z remote: Counting objects: 92% (2383/2590) 2024-06-26T07:00:53.4025092Z remote: Counting objects: 93% (2409/2590) 2024-06-26T07:00:53.4025674Z remote: Counting objects: 94% (2435/2590) 2024-06-26T07:00:53.4026301Z remote: Counting objects: 95% (2461/2590) 2024-06-26T07:00:53.4026889Z remote: Counting objects: 96% (2487/2590) 2024-06-26T07:00:53.4027569Z remote: Counting objects: 97% (2513/2590) 2024-06-26T07:00:53.4028156Z remote: Counting objects: 98% (2539/2590) 2024-06-26T07:00:53.4028739Z remote: Counting objects: 99% (2565/2590) 2024-06-26T07:00:53.4029324Z remote: Counting objects: 100% (2590/2590) 2024-06-26T07:00:53.4029937Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T07:00:53.4173376Z remote: Compressing objects: 0% (1/1068) 2024-06-26T07:00:53.4614113Z remote: Compressing objects: 1% (11/1068) 2024-06-26T07:00:53.5080479Z remote: Compressing objects: 2% (22/1068) 2024-06-26T07:00:53.5477204Z remote: Compressing objects: 3% (33/1068) 2024-06-26T07:00:53.5702422Z remote: Compressing objects: 4% (43/1068) 2024-06-26T07:00:53.5981960Z remote: Compressing objects: 5% (54/1068) 2024-06-26T07:00:53.6183362Z remote: Compressing objects: 6% (65/1068) 2024-06-26T07:00:53.6573625Z remote: Compressing objects: 7% (75/1068) 2024-06-26T07:00:53.7451611Z remote: Compressing objects: 8% (86/1068) 2024-06-26T07:00:53.8254123Z remote: Compressing objects: 9% (97/1068) 2024-06-26T07:00:53.8507533Z remote: Compressing objects: 10% (107/1068) 2024-06-26T07:00:53.8696755Z remote: Compressing objects: 11% (118/1068) 2024-06-26T07:00:53.8812669Z remote: Compressing objects: 12% (129/1068) 2024-06-26T07:00:53.8813289Z remote: Compressing objects: 13% (139/1068) 2024-06-26T07:00:53.8947134Z remote: Compressing objects: 14% (150/1068) 2024-06-26T07:00:53.8947757Z remote: Compressing objects: 15% (161/1068) 2024-06-26T07:00:53.9017427Z remote: Compressing objects: 16% (171/1068) 2024-06-26T07:00:53.9018046Z remote: Compressing objects: 17% (182/1068) 2024-06-26T07:00:53.9018658Z remote: Compressing objects: 18% (193/1068) 2024-06-26T07:00:53.9036579Z remote: Compressing objects: 19% (203/1068) 2024-06-26T07:00:53.9060135Z remote: Compressing objects: 20% (214/1068) 2024-06-26T07:00:53.9075794Z remote: Compressing objects: 21% (225/1068) 2024-06-26T07:00:53.9091642Z remote: Compressing objects: 22% (235/1068) 2024-06-26T07:00:53.9119639Z remote: Compressing objects: 23% (246/1068) 2024-06-26T07:00:53.9132360Z remote: Compressing objects: 24% (257/1068) 2024-06-26T07:00:53.9140439Z remote: Compressing objects: 25% (267/1068) 2024-06-26T07:00:53.9147754Z remote: Compressing objects: 26% (278/1068) 2024-06-26T07:00:53.9152410Z remote: Compressing objects: 27% (289/1068) 2024-06-26T07:00:53.9156125Z remote: Compressing objects: 28% (300/1068) 2024-06-26T07:00:53.9164188Z remote: Compressing objects: 29% (310/1068) 2024-06-26T07:00:53.9170743Z remote: Compressing objects: 30% (321/1068) 2024-06-26T07:00:53.9176095Z remote: Compressing objects: 31% (332/1068) 2024-06-26T07:00:53.9183717Z remote: Compressing objects: 32% (342/1068) 2024-06-26T07:00:53.9195466Z remote: Compressing objects: 33% (353/1068) 2024-06-26T07:00:53.9205127Z remote: Compressing objects: 34% (364/1068) 2024-06-26T07:00:53.9225575Z remote: Compressing objects: 35% (374/1068) 2024-06-26T07:00:53.9238298Z remote: Compressing objects: 36% (385/1068) 2024-06-26T07:00:53.9253456Z remote: Compressing objects: 37% (396/1068) 2024-06-26T07:00:53.9269006Z remote: Compressing objects: 38% (406/1068) 2024-06-26T07:00:53.9284832Z remote: Compressing objects: 39% (417/1068) 2024-06-26T07:00:53.9304572Z remote: Compressing objects: 40% (428/1068) 2024-06-26T07:00:53.9325917Z remote: Compressing objects: 41% (438/1068) 2024-06-26T07:00:53.9342811Z remote: Compressing objects: 42% (449/1068) 2024-06-26T07:00:53.9359693Z remote: Compressing objects: 43% (460/1068) 2024-06-26T07:00:53.9365026Z remote: Compressing objects: 44% (470/1068) 2024-06-26T07:00:53.9370828Z remote: Compressing objects: 45% (481/1068) 2024-06-26T07:00:53.9374224Z remote: Compressing objects: 46% (492/1068) 2024-06-26T07:00:53.9389940Z remote: Compressing objects: 47% (502/1068) 2024-06-26T07:00:53.9403830Z remote: Compressing objects: 48% (513/1068) 2024-06-26T07:00:53.9406185Z remote: Compressing objects: 49% (524/1068) 2024-06-26T07:00:53.9413482Z remote: Compressing objects: 50% (534/1068) 2024-06-26T07:00:53.9424818Z remote: Compressing objects: 51% (545/1068) 2024-06-26T07:00:53.9436988Z remote: Compressing objects: 52% (556/1068) 2024-06-26T07:00:53.9454179Z remote: Compressing objects: 53% (567/1068) 2024-06-26T07:00:53.9470959Z remote: Compressing objects: 54% (577/1068) 2024-06-26T07:00:53.9481004Z remote: Compressing objects: 55% (588/1068) 2024-06-26T07:00:53.9491888Z remote: Compressing objects: 56% (599/1068) 2024-06-26T07:00:53.9509506Z remote: Compressing objects: 57% (609/1068) 2024-06-26T07:00:53.9519236Z remote: Compressing objects: 58% (620/1068) 2024-06-26T07:00:53.9527142Z remote: Compressing objects: 59% (631/1068) 2024-06-26T07:00:53.9538319Z remote: Compressing objects: 60% (641/1068) 2024-06-26T07:00:53.9554357Z remote: Compressing objects: 61% (652/1068) 2024-06-26T07:00:53.9564135Z remote: Compressing objects: 62% (663/1068) 2024-06-26T07:00:53.9576195Z remote: Compressing objects: 63% (673/1068) 2024-06-26T07:00:53.9578367Z remote: Compressing objects: 64% (684/1068) 2024-06-26T07:00:53.9585673Z remote: Compressing objects: 65% (695/1068) 2024-06-26T07:00:53.9586423Z remote: Compressing objects: 66% (705/1068) 2024-06-26T07:00:53.9599198Z remote: Compressing objects: 67% (716/1068) 2024-06-26T07:00:53.9602134Z remote: Compressing objects: 68% (727/1068) 2024-06-26T07:00:53.9605336Z remote: Compressing objects: 69% (737/1068) 2024-06-26T07:00:53.9608883Z remote: Compressing objects: 70% (748/1068) 2024-06-26T07:00:53.9610302Z remote: Compressing objects: 71% (759/1068) 2024-06-26T07:00:53.9612795Z remote: Compressing objects: 72% (769/1068) 2024-06-26T07:00:53.9613686Z remote: Compressing objects: 73% (780/1068) 2024-06-26T07:00:53.9615966Z remote: Compressing objects: 74% (791/1068) 2024-06-26T07:00:53.9616580Z remote: Compressing objects: 75% (801/1068) 2024-06-26T07:00:53.9617232Z remote: Compressing objects: 76% (812/1068) 2024-06-26T07:00:53.9617966Z remote: Compressing objects: 77% (823/1068) 2024-06-26T07:00:53.9618579Z remote: Compressing objects: 78% (834/1068) 2024-06-26T07:00:53.9621812Z remote: Compressing objects: 79% (844/1068) 2024-06-26T07:00:53.9626528Z remote: Compressing objects: 80% (855/1068) 2024-06-26T07:00:53.9630854Z remote: Compressing objects: 81% (866/1068) 2024-06-26T07:00:53.9631469Z remote: Compressing objects: 82% (876/1068) 2024-06-26T07:00:53.9639504Z remote: Compressing objects: 83% (887/1068) 2024-06-26T07:00:53.9640125Z remote: Compressing objects: 84% (898/1068) 2024-06-26T07:00:53.9640724Z remote: Compressing objects: 85% (908/1068) 2024-06-26T07:00:53.9643859Z remote: Compressing objects: 86% (919/1068) 2024-06-26T07:00:53.9644478Z remote: Compressing objects: 87% (930/1068) 2024-06-26T07:00:53.9645085Z remote: Compressing objects: 88% (940/1068) 2024-06-26T07:00:53.9645682Z remote: Compressing objects: 89% (951/1068) 2024-06-26T07:00:53.9646299Z remote: Compressing objects: 90% (962/1068) 2024-06-26T07:00:53.9646909Z remote: Compressing objects: 91% (972/1068) 2024-06-26T07:00:53.9647518Z remote: Compressing objects: 92% (983/1068) 2024-06-26T07:00:53.9648125Z remote: Compressing objects: 93% (994/1068) 2024-06-26T07:00:53.9648738Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T07:00:53.9649459Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T07:00:53.9650075Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T07:00:53.9650682Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T07:00:53.9651307Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T07:00:53.9651924Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T07:00:53.9652537Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T07:00:53.9653180Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:01:13.3965285Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:01:39.4637387Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:01:39.4656632Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:39.4661069Z ##[endgroup] 2024-06-26T07:01:39.4662093Z ##[group]Determining the checkout info 2024-06-26T07:01:39.4663444Z ##[endgroup] 2024-06-26T07:01:39.4664469Z ##[group]Checking out the ref 2024-06-26T07:01:39.4666960Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:01:40.6831107Z ##[endgroup] 2024-06-26T07:01:40.6832144Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:01:40.6836386Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:01:40.6878543Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:01:40.6898038Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:01:40.6923216Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:01:40.6943948Z ##[endgroup] 2024-06-26T07:01:40.6944498Z ##[group]Fetching submodules 2024-06-26T07:01:40.6947689Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:01:40.7151540Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:01:40.7351083Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:01:40.7352769Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:01:40.7354142Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:01:40.7356293Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:01:40.7358824Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:01:40.7360561Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:01:40.7363032Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:01:40.7365454Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:01:40.7367821Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:01:40.7370432Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:01:40.7372861Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:01:40.7375461Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:01:40.7378116Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:01:40.7381193Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:01:40.7383894Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:01:40.7386779Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:01:40.7389842Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:01:40.7393206Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:01:40.7396239Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:01:40.7399192Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:01:40.7402451Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:01:40.7405725Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:01:40.7409308Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:01:40.7412631Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:01:40.7416214Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:01:40.7419693Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:01:40.7423797Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:01:40.7427226Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:01:40.7431026Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:01:40.7435617Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:01:40.7439576Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:01:40.7443498Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:01:40.7447496Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:01:40.7451397Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:01:40.7455592Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:01:40.7475253Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:01:40.9931319Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:01:41.1962974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:01:41.3580074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:01:41.5611871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:01:43.5207000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:01:53.3208969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:01:53.6832848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:01:54.1634577Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:01:54.7325872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:01:55.8741835Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:01:57.6331881Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:02:03.4476831Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:02:04.7257841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:02:06.3649286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:02:07.6110126Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:02:07.7679242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:02:08.1946153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:02:08.5276832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:02:09.4816912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:02:09.8081489Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:02:10.0658265Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:02:11.5417339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:02:12.2442532Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:02:13.0090976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:02:18.6158870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:02:21.0892033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:02:26.5491830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:02:26.7798166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:02:35.0968820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:02:35.3071312Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:02:35.5015618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:02:36.5160583Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:02:36.8393962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:02:37.4467205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:02:37.8822905Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:02:37.8903467Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:02:37.8963742Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:02:37.9145045Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:02:37.9448438Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:02:38.7184362Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:02:38.7357600Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:02:38.7695477Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:02:38.8539878Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:02:38.8809752Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:02:39.3062436Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:02:39.5203847Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:02:39.5833831Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:02:39.5845659Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:39.5847504Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:39.5849392Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:39.5851752Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:39.5854262Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:39.5871443Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:02:40.5973450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:02:41.1571328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:02:42.8491434Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:02:43.7676804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:02:44.0791035Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:02:44.1624709Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:02:44.5064305Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:02:44.5595646Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:02:44.5682194Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:02:44.6573595Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:02:44.6847046Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:02:44.6908097Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:02:44.7210937Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:02:44.7402979Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:02:44.7777657Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:44.7867932Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:02:44.7877838Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:44.7891972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:02:56.4193735Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:02:56.4331863Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:02:56.5104537Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:02:56.5115586Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:56.5117520Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:56.5119448Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:56.5136693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:02:57.0430694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:02:58.3593926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:02:59.3794083Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:02:59.3804134Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:59.3806307Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:59.3808485Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:59.3810502Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:59.3812568Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:59.3814944Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:59.3817091Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:59.3818905Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:59.3836452Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:03:00.4594727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:03:00.8131893Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:03:02.0243338Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:03:02.3071284Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:03:02.7636057Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:03:03.7491380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:03:09.2442176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:03:09.6033399Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:03:09.6169135Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:03:09.6457897Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:03:09.6554828Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:03:09.6564967Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:09.6581479Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:03:09.9522517Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:03:09.9656974Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:03:09.9992469Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:03:10.0779900Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:03:10.0900145Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:03:10.1195095Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:03:10.1683493Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:03:10.1989240Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:03:10.2186893Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:03:10.3037756Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:03:10.5732158Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:03:10.5761231Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:10.5763167Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:10.5778690Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:03:10.9870427Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:03:12.1150929Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:03:12.1424351Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:03:12.1937516Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:03:12.1950105Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:12.1957511Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:12.1959071Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:12.1960654Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:12.1962489Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:12.1964437Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:12.1966265Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:12.1967811Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:12.1979820Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:03:12.6205615Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:03:13.7270708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:03:14.0869632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:03:19.6984685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:03:19.9780409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:03:20.1795485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:03:20.4848636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:03:27.8098417Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:03:27.8435540Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:03:27.8557357Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:03:27.9392460Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:03:27.9482869Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:03:27.9589140Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:03:27.9702221Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:03:27.9712629Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:27.9714658Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:27.9731702Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:03:29.7177174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:03:30.8803882Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:03:30.9167913Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:03:31.2795370Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:03:31.2872756Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:03:31.5047631Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:03:31.5060941Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:31.5063104Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:31.5079303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:03:31.8873896Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:03:32.8171336Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:03:32.8791525Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:03:32.8850380Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:03:32.8936680Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:03:32.9212719Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:03:32.9431352Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:03:32.9763568Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:03:32.9959381Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:03:32.9969796Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:32.9971595Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:32.9973196Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:32.9974939Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:32.9992327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:03:33.9763280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:03:34.2268736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:03:35.2346269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:03:36.2667609Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:03:36.2778990Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:03:36.3341578Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:03:36.3569272Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:03:36.3579574Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:36.3598242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:03:36.5367091Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:03:36.5391256Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:03:36.5590173Z Entering 'android/libs/fbjni' 2024-06-26T07:03:36.5617629Z Entering 'third_party/FP16' 2024-06-26T07:03:36.5645568Z Entering 'third_party/FXdiv' 2024-06-26T07:03:36.5671971Z Entering 'third_party/NNPACK' 2024-06-26T07:03:36.5697697Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:36.5725992Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:36.5763499Z Entering 'third_party/benchmark' 2024-06-26T07:03:36.5791577Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:36.5822516Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:36.5848927Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:36.5876135Z Entering 'third_party/cutlass' 2024-06-26T07:03:36.5908887Z Entering 'third_party/eigen' 2024-06-26T07:03:36.5938536Z Entering 'third_party/fbgemm' 2024-06-26T07:03:36.5966502Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:36.5992700Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:36.6019675Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:36.6051186Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:36.6078361Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:36.6105966Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:36.6136590Z Entering 'third_party/fmt' 2024-06-26T07:03:36.6162640Z Entering 'third_party/foxi' 2024-06-26T07:03:36.6190346Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:36.6218959Z Entering 'third_party/gloo' 2024-06-26T07:03:36.6245989Z Entering 'third_party/googletest' 2024-06-26T07:03:36.6272281Z Entering 'third_party/ideep' 2024-06-26T07:03:36.6298392Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:36.6331513Z Entering 'third_party/ittapi' 2024-06-26T07:03:36.6359314Z Entering 'third_party/kineto' 2024-06-26T07:03:36.6386026Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:36.6413934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:36.6441654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:36.6474951Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:36.6500607Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:36.6527159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:36.6554606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:36.6581350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:36.6610153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:36.6638784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:36.6666534Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:36.6693678Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:36.6721119Z Entering 'third_party/mimalloc' 2024-06-26T07:03:36.6747779Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:36.6775935Z Entering 'third_party/nlohmann' 2024-06-26T07:03:36.6804426Z Entering 'third_party/onnx' 2024-06-26T07:03:36.6843889Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:36.6872402Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:36.6900943Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:36.6931345Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:36.6956043Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:36.6982018Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:36.7007613Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:36.7036149Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:36.7061906Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:36.7087066Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:36.7113642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:36.7142318Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:36.7169642Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:36.7212426Z Entering 'third_party/pocketfft' 2024-06-26T07:03:36.7240000Z Entering 'third_party/protobuf' 2024-06-26T07:03:36.7267925Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:36.7296066Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:36.7322594Z Entering 'third_party/psimd' 2024-06-26T07:03:36.7350592Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:36.7377914Z Entering 'third_party/pybind11' 2024-06-26T07:03:36.7406128Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:36.7433785Z Entering 'third_party/sleef' 2024-06-26T07:03:36.7462338Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:36.7490861Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:36.7516931Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:36.7543321Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:36.7569258Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:36.7594272Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:36.7634330Z ##[endgroup] 2024-06-26T07:03:36.7636532Z ##[group]Persisting credentials for submodules 2024-06-26T07:03:36.7638573Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:03:36.7843810Z Entering 'android/libs/fbjni' 2024-06-26T07:03:36.7878156Z Entering 'third_party/FP16' 2024-06-26T07:03:36.7912151Z Entering 'third_party/FXdiv' 2024-06-26T07:03:36.7946388Z Entering 'third_party/NNPACK' 2024-06-26T07:03:36.7981192Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:36.8016501Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:36.8062469Z Entering 'third_party/benchmark' 2024-06-26T07:03:36.8097735Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:36.8132734Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:36.8167987Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:36.8202195Z Entering 'third_party/cutlass' 2024-06-26T07:03:36.8241765Z Entering 'third_party/eigen' 2024-06-26T07:03:36.8277033Z Entering 'third_party/fbgemm' 2024-06-26T07:03:36.8311656Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:36.8346066Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:36.8380335Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:36.8417979Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:36.8452279Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:36.8488326Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:36.8524469Z Entering 'third_party/fmt' 2024-06-26T07:03:36.8560067Z Entering 'third_party/foxi' 2024-06-26T07:03:36.8593371Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:36.8628108Z Entering 'third_party/gloo' 2024-06-26T07:03:36.8663291Z Entering 'third_party/googletest' 2024-06-26T07:03:36.8699172Z Entering 'third_party/ideep' 2024-06-26T07:03:36.8731018Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:36.8769842Z Entering 'third_party/ittapi' 2024-06-26T07:03:36.8804038Z Entering 'third_party/kineto' 2024-06-26T07:03:36.8836663Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:36.8870129Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:36.8905029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:36.8938741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:36.8971740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:36.9003763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:36.9039271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:36.9072431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:36.9106649Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:36.9141843Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:36.9176233Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:36.9208635Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:36.9243167Z Entering 'third_party/mimalloc' 2024-06-26T07:03:36.9276471Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:36.9311076Z Entering 'third_party/nlohmann' 2024-06-26T07:03:36.9344647Z Entering 'third_party/onnx' 2024-06-26T07:03:36.9391304Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:36.9426387Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:36.9462203Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:36.9497581Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:36.9530677Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:36.9561847Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:36.9593833Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:36.9627312Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:36.9660729Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:36.9693647Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:36.9725606Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:36.9759866Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:36.9793604Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:36.9841538Z Entering 'third_party/pocketfft' 2024-06-26T07:03:36.9875501Z Entering 'third_party/protobuf' 2024-06-26T07:03:36.9912734Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:36.9944311Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:36.9980672Z Entering 'third_party/psimd' 2024-06-26T07:03:37.0015306Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:37.0049573Z Entering 'third_party/pybind11' 2024-06-26T07:03:37.0084401Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:37.0118176Z Entering 'third_party/sleef' 2024-06-26T07:03:37.0154824Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:37.0188260Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:37.0220574Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:37.0253756Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:37.0288449Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:37.0321480Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:37.0370045Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:03:37.0579172Z Entering 'android/libs/fbjni' 2024-06-26T07:03:37.0610925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:03:37.0622967Z Entering 'third_party/FP16' 2024-06-26T07:03:37.0654428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:03:37.0666244Z Entering 'third_party/FXdiv' 2024-06-26T07:03:37.0698433Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:03:37.0711608Z Entering 'third_party/NNPACK' 2024-06-26T07:03:37.0742952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:03:37.0755960Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:37.0786477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:03:37.0799158Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:37.0829664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:03:37.0852778Z Entering 'third_party/benchmark' 2024-06-26T07:03:37.0883677Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:37.0897088Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:37.0929539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:03:37.0942492Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:37.0973425Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:03:37.0985326Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:37.1015901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:03:37.1029359Z Entering 'third_party/cutlass' 2024-06-26T07:03:37.1061821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:03:37.1080611Z Entering 'third_party/eigen' 2024-06-26T07:03:37.1111548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:03:37.1126951Z Entering 'third_party/fbgemm' 2024-06-26T07:03:37.1157976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:03:37.1171177Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:37.1203086Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:03:37.1216324Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:37.1248196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:03:37.1260922Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:37.1292523Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:03:37.1309736Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:37.1341756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.1355170Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:37.1387127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:03:37.1399783Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:37.1432810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:03:37.1448557Z Entering 'third_party/fmt' 2024-06-26T07:03:37.1479639Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:03:37.1492115Z Entering 'third_party/foxi' 2024-06-26T07:03:37.1523840Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:03:37.1538542Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:37.1568175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:03:37.1581961Z Entering 'third_party/gloo' 2024-06-26T07:03:37.1614133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:03:37.1626603Z Entering 'third_party/googletest' 2024-06-26T07:03:37.1657297Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.1670944Z Entering 'third_party/ideep' 2024-06-26T07:03:37.1702698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:03:37.1713118Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:37.1742719Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:03:37.1762855Z Entering 'third_party/ittapi' 2024-06-26T07:03:37.1795117Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:03:37.1808061Z Entering 'third_party/kineto' 2024-06-26T07:03:37.1839572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:03:37.1850966Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:37.1882569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:03:37.1894059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:37.1925851Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:03:37.1939731Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:37.1970751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:03:37.1983334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:37.2018018Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:03:37.2029807Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:37.2060548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:03:37.2072678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:37.2104541Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:03:37.2118395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:37.2149262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:03:37.2161567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:37.2192405Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.2205002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:37.2235431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:03:37.2248229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:37.2278160Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:03:37.2291246Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:37.2322453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:03:37.2334839Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:37.2367087Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.2381368Z Entering 'third_party/mimalloc' 2024-06-26T07:03:37.2414191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:03:37.2426245Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:37.2457393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:03:37.2469644Z Entering 'third_party/nlohmann' 2024-06-26T07:03:37.2501760Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:03:37.2514791Z Entering 'third_party/onnx' 2024-06-26T07:03:37.2545204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:03:37.2569676Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:37.2600608Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:37.2613585Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:37.2644516Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:37.2659532Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:37.2691685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:03:37.2705963Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:37.2737224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:37.2750076Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:37.2780843Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.2791559Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:37.2823204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:03:37.2835393Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:37.2865429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:03:37.2878324Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:37.2908043Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:03:37.2919894Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:37.2951090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:03:37.2963215Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:37.2994412Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:03:37.3005860Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:37.3037173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:03:37.3049784Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:37.3080192Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:03:37.3093646Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:37.3125560Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:03:37.3153940Z Entering 'third_party/pocketfft' 2024-06-26T07:03:37.3185659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:03:37.3199141Z Entering 'third_party/protobuf' 2024-06-26T07:03:37.3229357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:03:37.3243043Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:37.3275003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:03:37.3287410Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:37.3317958Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.3331442Z Entering 'third_party/psimd' 2024-06-26T07:03:37.3362122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:03:37.3375551Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:37.3405343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:03:37.3415476Z Entering 'third_party/pybind11' 2024-06-26T07:03:37.3444604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:37.3457942Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:37.3489944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:03:37.3501604Z Entering 'third_party/sleef' 2024-06-26T07:03:37.3532909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:03:37.3544325Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:37.3576078Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:03:37.3587292Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:37.3618554Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:03:37.3631471Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:37.3663403Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:03:37.3676041Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:37.3706618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:03:37.3719643Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:37.3748922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:03:37.3759729Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:37.3790055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:03:37.4600018Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:03:37.4807634Z Entering 'android/libs/fbjni' 2024-06-26T07:03:37.4836245Z Entering 'third_party/FP16' 2024-06-26T07:03:37.4865056Z Entering 'third_party/FXdiv' 2024-06-26T07:03:37.4895443Z Entering 'third_party/NNPACK' 2024-06-26T07:03:37.4922324Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:37.4949583Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:37.4989619Z Entering 'third_party/benchmark' 2024-06-26T07:03:37.5019386Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:37.5048560Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:37.5076463Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:37.5105176Z Entering 'third_party/cutlass' 2024-06-26T07:03:37.5136263Z Entering 'third_party/eigen' 2024-06-26T07:03:37.5165398Z Entering 'third_party/fbgemm' 2024-06-26T07:03:37.5193249Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:37.5221393Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:37.5249168Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:37.5282431Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:37.5309936Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:37.5339635Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:37.5371182Z Entering 'third_party/fmt' 2024-06-26T07:03:37.5399640Z Entering 'third_party/foxi' 2024-06-26T07:03:37.5427004Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:37.5456786Z Entering 'third_party/gloo' 2024-06-26T07:03:37.5483038Z Entering 'third_party/googletest' 2024-06-26T07:03:37.5509163Z Entering 'third_party/ideep' 2024-06-26T07:03:37.5535323Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:37.5567813Z Entering 'third_party/ittapi' 2024-06-26T07:03:37.5595711Z Entering 'third_party/kineto' 2024-06-26T07:03:37.5622090Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:37.5648518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:37.5676319Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:37.5704447Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:37.5733048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:37.5760939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:37.5790950Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:37.5817473Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:37.5845880Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:37.5872163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:37.5902302Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:37.5928557Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:37.5956942Z Entering 'third_party/mimalloc' 2024-06-26T07:03:37.5986058Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:37.6013527Z Entering 'third_party/nlohmann' 2024-06-26T07:03:37.6042742Z Entering 'third_party/onnx' 2024-06-26T07:03:37.6082781Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:37.6110749Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:37.6138725Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:37.6167206Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:37.6193908Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:37.6223418Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:37.6249640Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:37.6278456Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:37.6303858Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:37.6331006Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:37.6356139Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:37.6385955Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:37.6413400Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:37.6454577Z Entering 'third_party/pocketfft' 2024-06-26T07:03:37.6480298Z Entering 'third_party/protobuf' 2024-06-26T07:03:37.6509560Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:37.6537314Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:37.6565575Z Entering 'third_party/psimd' 2024-06-26T07:03:37.6594535Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:37.6622451Z Entering 'third_party/pybind11' 2024-06-26T07:03:37.6651124Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:37.6679468Z Entering 'third_party/sleef' 2024-06-26T07:03:37.6705528Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:37.6733450Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:37.6758600Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:37.6783658Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:37.6811218Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:37.6837463Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:37.6878137Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:03:37.7080958Z Entering 'android/libs/fbjni' 2024-06-26T07:03:37.7108659Z Entering 'third_party/FP16' 2024-06-26T07:03:37.7135652Z Entering 'third_party/FXdiv' 2024-06-26T07:03:37.7164763Z Entering 'third_party/NNPACK' 2024-06-26T07:03:37.7189919Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:03:37.7217695Z Entering 'third_party/XNNPACK' 2024-06-26T07:03:37.7254387Z Entering 'third_party/benchmark' 2024-06-26T07:03:37.7284295Z Entering 'third_party/cpp-httplib' 2024-06-26T07:03:37.7310332Z Entering 'third_party/cpuinfo' 2024-06-26T07:03:37.7338781Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:03:37.7366070Z Entering 'third_party/cutlass' 2024-06-26T07:03:37.7399281Z Entering 'third_party/eigen' 2024-06-26T07:03:37.7427803Z Entering 'third_party/fbgemm' 2024-06-26T07:03:37.7456031Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:03:37.7483205Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:03:37.7510726Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:03:37.7542683Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:03:37.7571268Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:03:37.7600173Z Entering 'third_party/flatbuffers' 2024-06-26T07:03:37.7628817Z Entering 'third_party/fmt' 2024-06-26T07:03:37.7657964Z Entering 'third_party/foxi' 2024-06-26T07:03:37.7687877Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:03:37.7714682Z Entering 'third_party/gloo' 2024-06-26T07:03:37.7744492Z Entering 'third_party/googletest' 2024-06-26T07:03:37.7771602Z Entering 'third_party/ideep' 2024-06-26T07:03:37.7799621Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:03:37.7832512Z Entering 'third_party/ittapi' 2024-06-26T07:03:37.7859322Z Entering 'third_party/kineto' 2024-06-26T07:03:37.7887000Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:03:37.7911272Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:03:37.7940925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:03:37.7969703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:03:37.7996913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:03:37.8022025Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:03:37.8050733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:03:37.8079376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:03:37.8106252Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:03:37.8132590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:03:37.8159883Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:03:37.8187007Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:03:37.8216275Z Entering 'third_party/mimalloc' 2024-06-26T07:03:37.8242542Z Entering 'third_party/nccl/nccl' 2024-06-26T07:03:37.8270384Z Entering 'third_party/nlohmann' 2024-06-26T07:03:37.8299929Z Entering 'third_party/onnx' 2024-06-26T07:03:37.8338203Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:03:37.8366763Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:03:37.8395347Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:03:37.8425912Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:03:37.8453028Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:03:37.8478295Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:03:37.8504015Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:03:37.8533457Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:03:37.8561774Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:03:37.8587037Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:03:37.8614485Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:03:37.8642406Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:03:37.8668801Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:03:37.8710104Z Entering 'third_party/pocketfft' 2024-06-26T07:03:37.8739241Z Entering 'third_party/protobuf' 2024-06-26T07:03:37.8768690Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:03:37.8793243Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:03:37.8821297Z Entering 'third_party/psimd' 2024-06-26T07:03:37.8850081Z Entering 'third_party/pthreadpool' 2024-06-26T07:03:37.8875542Z Entering 'third_party/pybind11' 2024-06-26T07:03:37.8902858Z Entering 'third_party/python-peachpy' 2024-06-26T07:03:37.8928611Z Entering 'third_party/sleef' 2024-06-26T07:03:37.8958578Z Entering 'third_party/tensorpipe' 2024-06-26T07:03:37.8984323Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:03:37.9010778Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:03:37.9036769Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:03:37.9064467Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:03:37.9090558Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:03:37.9129055Z ##[endgroup] 2024-06-26T07:03:37.9157920Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:03:37.9178655Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:03:37.9350456Z Prepare all required actions 2024-06-26T07:03:37.9351028Z Getting action download info 2024-06-26T07:03:38.0655368Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:03:38.0655802Z env: 2024-06-26T07:03:38.0656089Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:38.0656440Z ##[endgroup] 2024-06-26T07:03:38.0715122Z ##[group]Run set -euo pipefail 2024-06-26T07:03:38.0715564Z set -euo pipefail 2024-06-26T07:03:38.0715959Z function get_ec2_metadata() { 2024-06-26T07:03:38.0716502Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:03:38.0717381Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:03:38.0718375Z  category=$1 2024-06-26T07:03:38.0718911Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:03:38.0719545Z  runner_name_str=i-0189c918a23a6d1e2 2024-06-26T07:03:38.0720043Z  if [[ -f /.inarc ]]; then 2024-06-26T07:03:38.0720530Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:03:38.0721095Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:03:38.0721785Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:03:38.0722388Z  else 2024-06-26T07:03:38.0722874Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:03:38.0723449Z  fi 2024-06-26T07:03:38.0723740Z } 2024-06-26T07:03:38.0724106Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:03:38.0724687Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:03:38.0725336Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:03:38.0725907Z echo "system info $(uname -a)" 2024-06-26T07:03:38.0733652Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:38.0734139Z env: 2024-06-26T07:03:38.0734431Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:38.0734786Z ##[endgroup] 2024-06-26T07:03:38.0805658Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:03:38.0845038Z instance-id: i-0189c918a23a6d1e2 2024-06-26T07:03:38.0884653Z instance-type: g5.4xlarge 2024-06-26T07:03:38.0890569Z system info Linux ip-10-0-2-209.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:03:38.0912952Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:03:38.0913943Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:03:38.0921844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:38.0922533Z env: 2024-06-26T07:03:38.0922885Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:38.0923308Z ##[endgroup] 2024-06-26T07:03:38.0987332Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:03:38.0988214Z if systemctl is-active --quiet docker; then 2024-06-26T07:03:38.0988998Z  echo "Docker daemon is running..."; 2024-06-26T07:03:38.0989653Z else 2024-06-26T07:03:38.0990358Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:03:38.0991186Z fi 2024-06-26T07:03:38.1000202Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:38.1000707Z env: 2024-06-26T07:03:38.1000981Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:38.1001324Z ##[endgroup] 2024-06-26T07:03:38.1036764Z Docker daemon is running... 2024-06-26T07:03:38.1079037Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:03:38.1079589Z with: 2024-06-26T07:03:38.1079867Z shell: bash 2024-06-26T07:03:38.1080201Z timeout_minutes: 5 2024-06-26T07:03:38.1080518Z max_attempts: 3 2024-06-26T07:03:38.1080831Z retry_wait_seconds: 30 2024-06-26T07:03:38.1082306Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:03:38.1083708Z polling_interval_seconds: 1 2024-06-26T07:03:38.1084085Z warning_on_retry: true 2024-06-26T07:03:38.1084433Z continue_on_error: false 2024-06-26T07:03:38.1084765Z env: 2024-06-26T07:03:38.1085030Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:38.1085383Z AWS_RETRY_MODE: standard 2024-06-26T07:03:38.1085731Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:03:38.1086061Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:03:38.1086432Z ##[endgroup] 2024-06-26T07:03:38.9551647Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:38.9552661Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:38.9553490Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:38.9554000Z 2024-06-26T07:03:38.9554148Z Login Succeeded 2024-06-26T07:03:39.1575062Z Command completed after 1 attempt(s). 2024-06-26T07:03:39.1615836Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:39.1616628Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:39.1617359Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:03:39.1625495Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.1626057Z env: 2024-06-26T07:03:39.1626402Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.1626818Z ##[endgroup] 2024-06-26T07:03:39.1684665Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:03:39.1685422Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:03:39.1686024Z # shellcheck disable=SC2046 2024-06-26T07:03:39.1686481Z docker stop $(docker ps -q) || true 2024-06-26T07:03:39.1686965Z # Prune all of the docker images 2024-06-26T07:03:39.1687416Z docker system prune -af 2024-06-26T07:03:39.1694265Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.1694758Z env: 2024-06-26T07:03:39.1695028Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.1695371Z ##[endgroup] 2024-06-26T07:03:39.1931666Z "docker stop" requires at least 1 argument. 2024-06-26T07:03:39.1932348Z See 'docker stop --help'. 2024-06-26T07:03:39.1932595Z 2024-06-26T07:03:39.1932818Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:03:39.1933187Z 2024-06-26T07:03:39.1936451Z Stop one or more running containers 2024-06-26T07:03:39.2087729Z Total reclaimed space: 0B 2024-06-26T07:03:39.2126042Z ##[group]Run set +e 2024-06-26T07:03:39.2126533Z set +e 2024-06-26T07:03:39.2127158Z set -x 2024-06-26T07:03:39.2127594Z  2024-06-26T07:03:39.2128050Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:03:39.2129240Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:03:39.2130877Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:03:39.2132006Z # one is returned at random 2024-06-26T07:03:39.2132828Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:03:39.2133584Z  2024-06-26T07:03:39.2134041Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:03:39.2134953Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:03:39.2136090Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:03:39.2136903Z  2024-06-26T07:03:39.2137381Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:03:39.2138199Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:03:39.2138938Z  exit 1 2024-06-26T07:03:39.2139382Z  fi 2024-06-26T07:03:39.2139785Z fi 2024-06-26T07:03:39.2140352Z  2024-06-26T07:03:39.2140891Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:03:39.2141769Z  # Clean up any old records first 2024-06-26T07:03:39.2142545Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:03:39.2143226Z fi 2024-06-26T07:03:39.2143619Z  2024-06-26T07:03:39.2144233Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:03:39.2145052Z cat /etc/hosts 2024-06-26T07:03:39.2155334Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.2156074Z env: 2024-06-26T07:03:39.2156495Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.2157008Z ##[endgroup] 2024-06-26T07:03:39.2181978Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:03:39.2185786Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:03:39.2186224Z ++ tail -n1 2024-06-26T07:03:39.2667410Z + RESOLVED_IP=18.160.10.76 2024-06-26T07:03:39.2667840Z + '[' -z 18.160.10.76 ']' 2024-06-26T07:03:39.2668282Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:03:39.2673658Z 18.160.10.28 download.pytorch.org 2024-06-26T07:03:39.2674170Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:03:39.2756240Z + echo '18.160.10.76 download.pytorch.org' 2024-06-26T07:03:39.2756712Z + sudo tee -a /etc/hosts 2024-06-26T07:03:39.2816871Z 18.160.10.76 download.pytorch.org 2024-06-26T07:03:39.2829614Z + cat /etc/hosts 2024-06-26T07:03:39.2834593Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:03:39.2844906Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:03:39.2845421Z 18.160.10.76 download.pytorch.org 2024-06-26T07:03:39.3026938Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:03:39.3027540Z with: 2024-06-26T07:03:39.3028608Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3029788Z docker-build-dir: .ci/docker 2024-06-26T07:03:39.3030152Z working-directory: . 2024-06-26T07:03:39.3030607Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.3031121Z force-push: false 2024-06-26T07:03:39.3031411Z env: 2024-06-26T07:03:39.3031820Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.3032156Z ##[endgroup] 2024-06-26T07:03:39.3048716Z ##[group]Run set -ex 2024-06-26T07:03:39.3049066Z set -ex 2024-06-26T07:03:39.3049346Z  2024-06-26T07:03:39.3049882Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:03:39.3050828Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:03:39.3051746Z # job could then download the pre-built image as usual 2024-06-26T07:03:39.3052481Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:03:39.3053149Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3053771Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3054313Z  2024-06-26T07:03:39.3054812Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:03:39.3055434Z  exit 0 2024-06-26T07:03:39.3055733Z else 2024-06-26T07:03:39.3056104Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3056547Z fi 2024-06-26T07:03:39.3056824Z  2024-06-26T07:03:39.3057285Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:03:39.3058119Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:03:39.3058879Z  # use it as it is, but first let's extract the tag 2024-06-26T07:03:39.3059566Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:03:39.3060276Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3060964Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3061579Z else 2024-06-26T07:03:39.3062010Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:03:39.3062661Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3063535Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3064265Z fi 2024-06-26T07:03:39.3071670Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.3072282Z env: 2024-06-26T07:03:39.3072566Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.3072915Z REPO_NAME: pytorch 2024-06-26T07:03:39.3074019Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3075191Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:03:39.3075680Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.3076192Z ##[endgroup] 2024-06-26T07:03:39.3096061Z + [[ ! -d .ci/docker ]] 2024-06-26T07:03:39.3096456Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:03:39.3096834Z + echo skip=false 2024-06-26T07:03:39.3098706Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:03:39.3101283Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3102455Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:03:39.3107030Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3107669Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3109352Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3133506Z ##[group]Run set +e 2024-06-26T07:03:39.3133867Z set +e 2024-06-26T07:03:39.3134167Z set -x 2024-06-26T07:03:39.3134453Z  2024-06-26T07:03:39.3134731Z login() { 2024-06-26T07:03:39.3135386Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:03:39.3136096Z } 2024-06-26T07:03:39.3136364Z  2024-06-26T07:03:39.3136773Z retry () { 2024-06-26T07:03:39.3137158Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:03:39.3137604Z } 2024-06-26T07:03:39.3137870Z  2024-06-26T07:03:39.3138185Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:03:39.3138595Z  2024-06-26T07:03:39.3139054Z # Check if image already exists, if it does then skip building it 2024-06-26T07:03:39.3139743Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:03:39.3140245Z  exit 0 2024-06-26T07:03:39.3140541Z fi 2024-06-26T07:03:39.3140811Z  2024-06-26T07:03:39.3141406Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:03:39.3142224Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:03:39.3142956Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:03:39.3143611Z  # if we're on the base branch then use the parent commit 2024-06-26T07:03:39.3144198Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:03:39.3144621Z else 2024-06-26T07:03:39.3145084Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:03:39.3145760Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:03:39.3146260Z fi 2024-06-26T07:03:39.3146537Z  2024-06-26T07:03:39.3146853Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:03:39.3147339Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3147797Z  2024-06-26T07:03:39.3148433Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:03:39.3149176Z  exit 0 2024-06-26T07:03:39.3149467Z fi 2024-06-26T07:03:39.3149742Z  2024-06-26T07:03:39.3150176Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:03:39.3151126Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:03:39.3152031Z  exit 1 2024-06-26T07:03:39.3152330Z fi 2024-06-26T07:03:39.3152607Z  2024-06-26T07:03:39.3153090Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:03:39.3154022Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:03:39.3154887Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:03:39.3155834Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:03:39.3156915Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:03:39.3157637Z fi 2024-06-26T07:03:39.3157911Z  2024-06-26T07:03:39.3158270Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:03:39.3165045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.3165545Z env: 2024-06-26T07:03:39.3165814Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.3166174Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:03:39.3166632Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:03:39.3167877Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3169089Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.3169674Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.3170184Z ##[endgroup] 2024-06-26T07:03:39.3188911Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.3189607Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.3190433Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:03:39.3191405Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.7220454Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:39.7221398Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:39.7222342Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:39.7222889Z 2024-06-26T07:03:39.7223020Z Login Succeeded 2024-06-26T07:03:39.7233047Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.9272627Z { 2024-06-26T07:03:39.9273034Z "schemaVersion": 2, 2024-06-26T07:03:39.9273788Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:03:39.9274573Z "config": { 2024-06-26T07:03:39.9275178Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:03:39.9275917Z "size": 48681, 2024-06-26T07:03:39.9276677Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:03:39.9277640Z }, 2024-06-26T07:03:39.9278070Z "layers": [ 2024-06-26T07:03:39.9278474Z { 2024-06-26T07:03:39.9279086Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9279895Z "size": 28580681, 2024-06-26T07:03:39.9280708Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:03:39.9281587Z }, 2024-06-26T07:03:39.9281943Z { 2024-06-26T07:03:39.9282523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9283310Z "size": 7943451, 2024-06-26T07:03:39.9284079Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:03:39.9284962Z }, 2024-06-26T07:03:39.9285330Z { 2024-06-26T07:03:39.9285922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9286735Z "size": 55728572, 2024-06-26T07:03:39.9287437Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:03:39.9288072Z }, 2024-06-26T07:03:39.9288519Z + exit 0 2024-06-26T07:03:39.9288809Z { 2024-06-26T07:03:39.9289302Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9290060Z "size": 186, 2024-06-26T07:03:39.9290779Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:03:39.9291538Z }, 2024-06-26T07:03:39.9291846Z { 2024-06-26T07:03:39.9292272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9292885Z "size": 6886, 2024-06-26T07:03:39.9293487Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:03:39.9294390Z }, 2024-06-26T07:03:39.9294733Z { 2024-06-26T07:03:39.9295284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9295975Z "size": 1329001756, 2024-06-26T07:03:39.9296541Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:03:39.9297166Z }, 2024-06-26T07:03:39.9297403Z { 2024-06-26T07:03:39.9297987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9298696Z "size": 62501, 2024-06-26T07:03:39.9299375Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:03:39.9299974Z }, 2024-06-26T07:03:39.9300221Z { 2024-06-26T07:03:39.9300646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9301359Z "size": 1684, 2024-06-26T07:03:39.9301870Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:03:39.9302480Z }, 2024-06-26T07:03:39.9302732Z { 2024-06-26T07:03:39.9303157Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9303845Z "size": 1523, 2024-06-26T07:03:39.9304402Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:03:39.9305018Z }, 2024-06-26T07:03:39.9305257Z { 2024-06-26T07:03:39.9305677Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9306223Z "size": 2528295403, 2024-06-26T07:03:39.9306777Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:03:39.9307365Z }, 2024-06-26T07:03:39.9307610Z { 2024-06-26T07:03:39.9308030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9308567Z "size": 86016, 2024-06-26T07:03:39.9309100Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:03:39.9309705Z }, 2024-06-26T07:03:39.9309948Z { 2024-06-26T07:03:39.9310363Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9310909Z "size": 1823, 2024-06-26T07:03:39.9311427Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:03:39.9312334Z }, 2024-06-26T07:03:39.9312575Z { 2024-06-26T07:03:39.9312997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9313542Z "size": 246737542, 2024-06-26T07:03:39.9314079Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:03:39.9314664Z }, 2024-06-26T07:03:39.9314911Z { 2024-06-26T07:03:39.9315341Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9315884Z "size": 545, 2024-06-26T07:03:39.9316413Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:03:39.9317024Z }, 2024-06-26T07:03:39.9317269Z { 2024-06-26T07:03:39.9317696Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9318236Z "size": 1283, 2024-06-26T07:03:39.9318758Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:03:39.9319354Z }, 2024-06-26T07:03:39.9319592Z { 2024-06-26T07:03:39.9320012Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9320556Z "size": 485, 2024-06-26T07:03:39.9321089Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:03:39.9321684Z }, 2024-06-26T07:03:39.9321928Z { 2024-06-26T07:03:39.9322347Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9322892Z "size": 91710937, 2024-06-26T07:03:39.9323445Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:03:39.9324058Z }, 2024-06-26T07:03:39.9324302Z { 2024-06-26T07:03:39.9324713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9325371Z "size": 3233, 2024-06-26T07:03:39.9325918Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:03:39.9326525Z }, 2024-06-26T07:03:39.9326760Z { 2024-06-26T07:03:39.9327180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9327723Z "size": 1905, 2024-06-26T07:03:39.9328264Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:03:39.9328865Z }, 2024-06-26T07:03:39.9329111Z { 2024-06-26T07:03:39.9329532Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9330069Z "size": 702, 2024-06-26T07:03:39.9330597Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:39.9331215Z }, 2024-06-26T07:03:39.9331458Z { 2024-06-26T07:03:39.9331870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9332412Z "size": 2784789002, 2024-06-26T07:03:39.9332969Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:03:39.9333648Z }, 2024-06-26T07:03:39.9333886Z { 2024-06-26T07:03:39.9334312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9334850Z "size": 381, 2024-06-26T07:03:39.9335378Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:03:39.9335971Z }, 2024-06-26T07:03:39.9336215Z { 2024-06-26T07:03:39.9336639Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9337175Z "size": 11879, 2024-06-26T07:03:39.9337699Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:03:39.9338288Z }, 2024-06-26T07:03:39.9338533Z { 2024-06-26T07:03:39.9338944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9339489Z "size": 803, 2024-06-26T07:03:39.9340022Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:03:39.9340627Z }, 2024-06-26T07:03:39.9340865Z { 2024-06-26T07:03:39.9341400Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9341945Z "size": 106, 2024-06-26T07:03:39.9342457Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:03:39.9343047Z }, 2024-06-26T07:03:39.9343293Z { 2024-06-26T07:03:39.9343715Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9344250Z "size": 503, 2024-06-26T07:03:39.9344775Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:03:39.9345376Z }, 2024-06-26T07:03:39.9345618Z { 2024-06-26T07:03:39.9346030Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9346576Z "size": 121477177, 2024-06-26T07:03:39.9347126Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:03:39.9347720Z }, 2024-06-26T07:03:39.9347964Z { 2024-06-26T07:03:39.9348381Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9348931Z "size": 109, 2024-06-26T07:03:39.9349448Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:03:39.9350049Z }, 2024-06-26T07:03:39.9350294Z { 2024-06-26T07:03:39.9350718Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9351252Z "size": 491, 2024-06-26T07:03:39.9352060Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:03:39.9352662Z }, 2024-06-26T07:03:39.9353007Z { 2024-06-26T07:03:39.9353421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9353958Z "size": 291, 2024-06-26T07:03:39.9354485Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:03:39.9355184Z }, 2024-06-26T07:03:39.9355430Z { 2024-06-26T07:03:39.9355840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9356384Z "size": 101, 2024-06-26T07:03:39.9356901Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:03:39.9357501Z }, 2024-06-26T07:03:39.9357734Z { 2024-06-26T07:03:39.9358146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9358687Z "size": 1473, 2024-06-26T07:03:39.9359204Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:03:39.9359778Z }, 2024-06-26T07:03:39.9360022Z { 2024-06-26T07:03:39.9360439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9360974Z "size": 446710781, 2024-06-26T07:03:39.9361509Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:03:39.9362099Z }, 2024-06-26T07:03:39.9362346Z { 2024-06-26T07:03:39.9362758Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9363372Z "size": 161, 2024-06-26T07:03:39.9363878Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:03:39.9364457Z }, 2024-06-26T07:03:39.9364692Z { 2024-06-26T07:03:39.9365109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9365644Z "size": 564, 2024-06-26T07:03:39.9366165Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:03:39.9366755Z }, 2024-06-26T07:03:39.9366995Z { 2024-06-26T07:03:39.9367408Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9367933Z "size": 35874600, 2024-06-26T07:03:39.9368477Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:03:39.9369073Z }, 2024-06-26T07:03:39.9369314Z { 2024-06-26T07:03:39.9369726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9370268Z "size": 104, 2024-06-26T07:03:39.9370776Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:03:39.9371354Z }, 2024-06-26T07:03:39.9371587Z { 2024-06-26T07:03:39.9372001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9372535Z "size": 425, 2024-06-26T07:03:39.9373047Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:03:39.9373626Z }, 2024-06-26T07:03:39.9373867Z { 2024-06-26T07:03:39.9374281Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9374806Z "size": 20262084, 2024-06-26T07:03:39.9375353Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:03:39.9375957Z }, 2024-06-26T07:03:39.9376198Z { 2024-06-26T07:03:39.9376612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9377144Z "size": 439, 2024-06-26T07:03:39.9377661Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:03:39.9378243Z }, 2024-06-26T07:03:39.9378481Z { 2024-06-26T07:03:39.9378897Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9379430Z "size": 702, 2024-06-26T07:03:39.9379951Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:39.9380546Z }, 2024-06-26T07:03:39.9380786Z { 2024-06-26T07:03:39.9381313Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9381866Z "size": 143, 2024-06-26T07:03:39.9382381Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:03:39.9382984Z }, 2024-06-26T07:03:39.9383231Z { 2024-06-26T07:03:39.9383637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9384251Z "size": 135, 2024-06-26T07:03:39.9384781Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:03:39.9385392Z }, 2024-06-26T07:03:39.9385625Z { 2024-06-26T07:03:39.9386042Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9386581Z "size": 4911495247, 2024-06-26T07:03:39.9387108Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:03:39.9387695Z }, 2024-06-26T07:03:39.9387939Z { 2024-06-26T07:03:39.9388364Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9388895Z "size": 190, 2024-06-26T07:03:39.9389428Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:03:39.9390040Z }, 2024-06-26T07:03:39.9390280Z { 2024-06-26T07:03:39.9390689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9391224Z "size": 565, 2024-06-26T07:03:39.9391951Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:03:39.9392642Z }, 2024-06-26T07:03:39.9392878Z { 2024-06-26T07:03:39.9393294Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9393828Z "size": 43163095, 2024-06-26T07:03:39.9394361Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:03:39.9394960Z }, 2024-06-26T07:03:39.9395199Z { 2024-06-26T07:03:39.9395615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9396146Z "size": 106, 2024-06-26T07:03:39.9396670Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:03:39.9397264Z }, 2024-06-26T07:03:39.9397505Z { 2024-06-26T07:03:39.9397916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9398475Z "size": 1213, 2024-06-26T07:03:39.9398998Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:03:39.9399590Z }, 2024-06-26T07:03:39.9399823Z { 2024-06-26T07:03:39.9400240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9400777Z "size": 702, 2024-06-26T07:03:39.9401301Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:39.9401897Z }, 2024-06-26T07:03:39.9402138Z { 2024-06-26T07:03:39.9402558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9403096Z "size": 138, 2024-06-26T07:03:39.9403623Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:03:39.9404230Z }, 2024-06-26T07:03:39.9404475Z { 2024-06-26T07:03:39.9404882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9405417Z "size": 120, 2024-06-26T07:03:39.9405950Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:03:39.9406541Z }, 2024-06-26T07:03:39.9406783Z { 2024-06-26T07:03:39.9407204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9407742Z "size": 1904157495, 2024-06-26T07:03:39.9408285Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:03:39.9408889Z }, 2024-06-26T07:03:39.9409131Z { 2024-06-26T07:03:39.9409546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9410074Z "size": 172, 2024-06-26T07:03:39.9410598Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:03:39.9411202Z }, 2024-06-26T07:03:39.9411436Z { 2024-06-26T07:03:39.9411853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9412392Z "size": 908, 2024-06-26T07:03:39.9412928Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:03:39.9413605Z }, 2024-06-26T07:03:39.9413854Z { 2024-06-26T07:03:39.9414272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9414818Z "size": 702, 2024-06-26T07:03:39.9415333Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:03:39.9415935Z }, 2024-06-26T07:03:39.9416176Z { 2024-06-26T07:03:39.9416592Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9417122Z "size": 134, 2024-06-26T07:03:39.9417637Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:03:39.9418231Z }, 2024-06-26T07:03:39.9418466Z { 2024-06-26T07:03:39.9418882Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9419415Z "size": 32, 2024-06-26T07:03:39.9419943Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:39.9420534Z }, 2024-06-26T07:03:39.9420784Z { 2024-06-26T07:03:39.9421292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9421923Z "size": 156, 2024-06-26T07:03:39.9422457Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:03:39.9423072Z }, 2024-06-26T07:03:39.9423313Z { 2024-06-26T07:03:39.9423723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9424259Z "size": 1841, 2024-06-26T07:03:39.9424790Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:03:39.9425389Z }, 2024-06-26T07:03:39.9425623Z { 2024-06-26T07:03:39.9426041Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9426581Z "size": 7529784, 2024-06-26T07:03:39.9427117Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:03:39.9427704Z }, 2024-06-26T07:03:39.9427945Z { 2024-06-26T07:03:39.9428367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9428906Z "size": 165, 2024-06-26T07:03:39.9429432Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:03:39.9430024Z }, 2024-06-26T07:03:39.9430274Z { 2024-06-26T07:03:39.9430681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9431218Z "size": 7944, 2024-06-26T07:03:39.9431931Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:03:39.9432541Z }, 2024-06-26T07:03:39.9432775Z { 2024-06-26T07:03:39.9433194Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9433732Z "size": 8070, 2024-06-26T07:03:39.9434251Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:03:39.9434831Z }, 2024-06-26T07:03:39.9435073Z { 2024-06-26T07:03:39.9435494Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9436021Z "size": 303, 2024-06-26T07:03:39.9436548Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:03:39.9437153Z }, 2024-06-26T07:03:39.9437396Z { 2024-06-26T07:03:39.9437808Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9438345Z "size": 7629071, 2024-06-26T07:03:39.9438856Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:03:39.9439423Z }, 2024-06-26T07:03:39.9439657Z { 2024-06-26T07:03:39.9440071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9440606Z "size": 108, 2024-06-26T07:03:39.9441131Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:03:39.9441721Z }, 2024-06-26T07:03:39.9441962Z { 2024-06-26T07:03:39.9442382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9443005Z "size": 54145775, 2024-06-26T07:03:39.9443555Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:03:39.9444154Z }, 2024-06-26T07:03:39.9444393Z { 2024-06-26T07:03:39.9444803Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9445339Z "size": 474, 2024-06-26T07:03:39.9445857Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:03:39.9446451Z }, 2024-06-26T07:03:39.9446689Z { 2024-06-26T07:03:39.9447106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9447643Z "size": 1374858747, 2024-06-26T07:03:39.9448174Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:03:39.9448751Z }, 2024-06-26T07:03:39.9448991Z { 2024-06-26T07:03:39.9449409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9449935Z "size": 106, 2024-06-26T07:03:39.9450475Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:03:39.9451155Z }, 2024-06-26T07:03:39.9451397Z { 2024-06-26T07:03:39.9451804Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9452336Z "size": 558, 2024-06-26T07:03:39.9452870Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:03:39.9453472Z }, 2024-06-26T07:03:39.9453706Z { 2024-06-26T07:03:39.9454126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9454662Z "size": 46248513, 2024-06-26T07:03:39.9455190Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:03:39.9455773Z }, 2024-06-26T07:03:39.9456019Z { 2024-06-26T07:03:39.9456437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9456968Z "size": 111, 2024-06-26T07:03:39.9457499Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:03:39.9458092Z }, 2024-06-26T07:03:39.9458343Z { 2024-06-26T07:03:39.9458755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9459288Z "size": 32, 2024-06-26T07:03:39.9459814Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:39.9460413Z }, 2024-06-26T07:03:39.9460648Z { 2024-06-26T07:03:39.9461137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9461697Z "size": 32, 2024-06-26T07:03:39.9462210Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:39.9462804Z }, 2024-06-26T07:03:39.9463044Z { 2024-06-26T07:03:39.9463464Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9463991Z "size": 32, 2024-06-26T07:03:39.9464517Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:39.9465114Z }, 2024-06-26T07:03:39.9465356Z { 2024-06-26T07:03:39.9465771Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:03:39.9466310Z "size": 32, 2024-06-26T07:03:39.9466832Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:03:39.9467416Z } 2024-06-26T07:03:39.9467660Z ] 2024-06-26T07:03:39.9467898Z } 2024-06-26T07:03:39.9586993Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:03:39.9587506Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:03:39.9588046Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:03:39.9596486Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.9596971Z env: 2024-06-26T07:03:39.9597249Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.9598364Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.9599504Z ##[endgroup] 2024-06-26T07:03:39.9624587Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.9674680Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:03:39.9675309Z with: 2024-06-26T07:03:39.9676513Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.9677996Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.9678515Z env: 2024-06-26T07:03:39.9678800Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.9679157Z ##[endgroup] 2024-06-26T07:03:39.9705304Z ##[group]Run set -x 2024-06-26T07:03:39.9705651Z set -x 2024-06-26T07:03:39.9705958Z set +e 2024-06-26T07:03:39.9706255Z  2024-06-26T07:03:39.9706538Z login() { 2024-06-26T07:03:39.9707205Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:03:39.9708025Z } 2024-06-26T07:03:39.9708303Z  2024-06-26T07:03:39.9708603Z retry () { 2024-06-26T07:03:39.9708994Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:03:39.9709438Z } 2024-06-26T07:03:39.9709712Z  2024-06-26T07:03:39.9710027Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:03:39.9710435Z  2024-06-26T07:03:39.9710703Z set -e 2024-06-26T07:03:39.9711174Z # ignore output since only exit code is used for conditional 2024-06-26T07:03:39.9712034Z # only pull docker image if it's not available locally 2024-06-26T07:03:39.9712783Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:03:39.9713461Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:03:39.9713895Z fi 2024-06-26T07:03:39.9721466Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:03:39.9721966Z env: 2024-06-26T07:03:39.9722249Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:03:39.9723375Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:39.9724660Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.9725174Z ##[endgroup] 2024-06-26T07:03:39.9743444Z + set +e 2024-06-26T07:03:39.9744101Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.9744768Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:39.9745873Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:03:39.9746810Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:03:40.3843466Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:03:40.3844253Z Configure a credential helper to remove this warning. See 2024-06-26T07:03:40.3845074Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:03:40.3845605Z 2024-06-26T07:03:40.3845733Z Login Succeeded 2024-06-26T07:03:40.3855242Z + set -e 2024-06-26T07:03:40.3856839Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:40.3982020Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:40.3984115Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:03:40.5933807Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:03:40.5935198Z 7a2c55901189: Pulling fs layer 2024-06-26T07:03:40.5935751Z 224fe954d725: Pulling fs layer 2024-06-26T07:03:40.5936526Z 75722010b82e: Pulling fs layer 2024-06-26T07:03:40.5937079Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:03:40.5937622Z b57676e46aee: Pulling fs layer 2024-06-26T07:03:40.5938145Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:03:40.5938581Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:03:40.5939048Z 0c1227890755: Pulling fs layer 2024-06-26T07:03:40.5939419Z d8d1234baab3: Pulling fs layer 2024-06-26T07:03:40.5939780Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:03:40.5940154Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:03:40.5940557Z 309b872da805: Pulling fs layer 2024-06-26T07:03:40.5940968Z e82932cb002c: Pulling fs layer 2024-06-26T07:03:40.5941450Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:03:40.5941822Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:03:40.5942192Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:03:40.5942564Z f4863801e30c: Pulling fs layer 2024-06-26T07:03:40.5943062Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:03:40.5943444Z cf2f34f225db: Pulling fs layer 2024-06-26T07:03:40.5943822Z df546d7b3e34: Pulling fs layer 2024-06-26T07:03:40.5944234Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:03:40.5944591Z 96d0e2032184: Pulling fs layer 2024-06-26T07:03:40.5944954Z 06e12ce59790: Pulling fs layer 2024-06-26T07:03:40.5945319Z ed712f63cde7: Pulling fs layer 2024-06-26T07:03:40.5945688Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:03:40.5946049Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:03:40.5946420Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:03:40.5946786Z de45a14c9b22: Pulling fs layer 2024-06-26T07:03:40.5947150Z e99241e77209: Pulling fs layer 2024-06-26T07:03:40.5947508Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:03:40.5947879Z a823ffa4c541: Pulling fs layer 2024-06-26T07:03:40.5948247Z 44ba272eae20: Pulling fs layer 2024-06-26T07:03:40.5948618Z da02d278e524: Pulling fs layer 2024-06-26T07:03:40.5949005Z d527cbbb87e3: Waiting 2024-06-26T07:03:40.5949398Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:03:40.5949774Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:03:40.5950140Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:03:40.5950507Z b2950a036534: Pulling fs layer 2024-06-26T07:03:40.5950896Z b57676e46aee: Waiting 2024-06-26T07:03:40.5951219Z a8c1e85b5e14: Waiting 2024-06-26T07:03:40.5951581Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:03:40.5952263Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:03:40.5952634Z 7a868239227f: Pulling fs layer 2024-06-26T07:03:40.5952979Z f4863801e30c: Waiting 2024-06-26T07:03:40.5953296Z ca9dd73e3e93: Waiting 2024-06-26T07:03:40.5953616Z cf2f34f225db: Waiting 2024-06-26T07:03:40.5953947Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:03:40.5954311Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:03:40.5954670Z d37dd9ed8948: Waiting 2024-06-26T07:03:40.5954984Z 16d76a79d5d6: Waiting 2024-06-26T07:03:40.5955308Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:03:40.5955668Z 3d914950f8ad: Waiting 2024-06-26T07:03:40.5955983Z a97a71ef8a4e: Waiting 2024-06-26T07:03:40.5956368Z c1068d03a811: Pulling fs layer 2024-06-26T07:03:40.5956740Z 671b4a301d81: Pulling fs layer 2024-06-26T07:03:40.5957146Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:03:40.5957553Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:03:40.5957912Z df546d7b3e34: Waiting 2024-06-26T07:03:40.5958234Z fb2e97768560: Pulling fs layer 2024-06-26T07:03:40.5958590Z 7ed32bc8e469: Waiting 2024-06-26T07:03:40.5958922Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:03:40.5959274Z de45a14c9b22: Waiting 2024-06-26T07:03:40.5959603Z e08e88c13668: Pulling fs layer 2024-06-26T07:03:40.5959957Z ec1e7978c1fe: Waiting 2024-06-26T07:03:40.5960272Z 96d0e2032184: Waiting 2024-06-26T07:03:40.5960576Z e99241e77209: Waiting 2024-06-26T07:03:40.5960889Z 4bdf41c9ecbe: Waiting 2024-06-26T07:03:40.5961206Z 35e8f4e2a42f: Waiting 2024-06-26T07:03:40.5961519Z e82932cb002c: Waiting 2024-06-26T07:03:40.5961823Z 9bf94bc45658: Waiting 2024-06-26T07:03:40.5962135Z 9cd4c745f8e4: Waiting 2024-06-26T07:03:40.5962458Z 84b2c7a89982: Waiting 2024-06-26T07:03:40.5962779Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:03:40.5963135Z c1068d03a811: Waiting 2024-06-26T07:03:40.5963598Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:03:40.5964028Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:03:40.5964383Z a41a8d1c11c8: Waiting 2024-06-26T07:03:40.5964714Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:03:40.5965068Z b3c5d66c679f: Waiting 2024-06-26T07:03:40.5965391Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:03:40.5965752Z 25cd0c482ad6: Waiting 2024-06-26T07:03:40.5966063Z 671b4a301d81: Waiting 2024-06-26T07:03:40.5966392Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:03:40.5966741Z 06e12ce59790: Waiting 2024-06-26T07:03:40.5967047Z 7a868239227f: Waiting 2024-06-26T07:03:40.5967371Z e797fe86caef: Pulling fs layer 2024-06-26T07:03:40.5967730Z ed712f63cde7: Waiting 2024-06-26T07:03:40.5968035Z d8d1234baab3: Waiting 2024-06-26T07:03:40.5968343Z b2950a036534: Waiting 2024-06-26T07:03:40.5968658Z 73a0ce681c29: Waiting 2024-06-26T07:03:40.5969053Z da02d278e524: Waiting 2024-06-26T07:03:40.5969364Z 44ba272eae20: Waiting 2024-06-26T07:03:40.5969675Z 49737bd5ae70: Waiting 2024-06-26T07:03:40.5969990Z 25e19d81d9fc: Waiting 2024-06-26T07:03:40.5970295Z 67760cf5ec46: Waiting 2024-06-26T07:03:40.5970606Z cf7c2cb896cb: Waiting 2024-06-26T07:03:40.5970921Z ddc9740bcea2: Waiting 2024-06-26T07:03:40.5971229Z fb2e97768560: Waiting 2024-06-26T07:03:40.5971556Z 967190076fc0: Pulling fs layer 2024-06-26T07:03:40.5971940Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:03:40.5972315Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:03:40.5972668Z 60156b40fcbe: Waiting 2024-06-26T07:03:40.5972996Z cc91933e985a: Pulling fs layer 2024-06-26T07:03:40.5973371Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:03:40.5973742Z 526083994857: Pulling fs layer 2024-06-26T07:03:40.5974107Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:03:40.5974482Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:03:40.5974855Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:03:40.5975216Z 95237e7311a1: Pulling fs layer 2024-06-26T07:03:40.5975587Z dda1ce302b9a: Waiting 2024-06-26T07:03:40.5975923Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:03:40.5976300Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:03:40.5976663Z 2060028f7488: Pulling fs layer 2024-06-26T07:03:40.5977035Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:03:40.5977396Z 789eff6c51aa: Waiting 2024-06-26T07:03:40.5977714Z de0f8a8c45c6: Waiting 2024-06-26T07:03:40.5978022Z 2060028f7488: Waiting 2024-06-26T07:03:40.5978336Z 0c1227890755: Waiting 2024-06-26T07:03:40.5978649Z e412c7f3b2de: Waiting 2024-06-26T07:03:40.5978956Z 236c8f0245f4: Waiting 2024-06-26T07:03:40.5979265Z e08e88c13668: Waiting 2024-06-26T07:03:40.5979577Z 526083994857: Waiting 2024-06-26T07:03:40.5979883Z 9f3187916e7d: Waiting 2024-06-26T07:03:40.5980195Z 05a6f6dcc1c0: Waiting 2024-06-26T07:03:40.5980509Z 4fb0a83f0685: Waiting 2024-06-26T07:03:40.5980821Z 6bc9dc8315b0: Waiting 2024-06-26T07:03:40.5981213Z 95237e7311a1: Waiting 2024-06-26T07:03:40.5981529Z 1b0e27c35f07: Waiting 2024-06-26T07:03:40.5981890Z 68af2ec4e30a: Waiting 2024-06-26T07:03:40.5982202Z 12ba7431ff38: Waiting 2024-06-26T07:03:40.5982507Z cc91933e985a: Waiting 2024-06-26T07:03:40.5982810Z 83911d7c7ecf: Waiting 2024-06-26T07:03:40.5983116Z e797fe86caef: Waiting 2024-06-26T07:03:40.5983424Z 3a778ddfac2a: Waiting 2024-06-26T07:03:40.7262402Z 224fe954d725: Verifying Checksum 2024-06-26T07:03:40.7262829Z 224fe954d725: Download complete 2024-06-26T07:03:40.8081723Z d527cbbb87e3: Download complete 2024-06-26T07:03:40.8897681Z b57676e46aee: Download complete 2024-06-26T07:03:40.9623780Z 7a2c55901189: Download complete 2024-06-26T07:03:41.0306684Z a41a8d1c11c8: Download complete 2024-06-26T07:03:41.1491884Z 0c1227890755: Verifying Checksum 2024-06-26T07:03:41.1492674Z 0c1227890755: Download complete 2024-06-26T07:03:41.1960970Z 75722010b82e: Verifying Checksum 2024-06-26T07:03:41.1961461Z 75722010b82e: Download complete 2024-06-26T07:03:41.2494933Z d8d1234baab3: Verifying Checksum 2024-06-26T07:03:41.2495524Z d8d1234baab3: Download complete 2024-06-26T07:03:41.3229803Z ec1e7978c1fe: Download complete 2024-06-26T07:03:41.3929513Z 309b872da805: Download complete 2024-06-26T07:03:42.0116966Z 7a2c55901189: Pull complete 2024-06-26T07:03:42.3287952Z 224fe954d725: Pull complete 2024-06-26T07:03:43.2773107Z 75722010b82e: Pull complete 2024-06-26T07:03:43.3188547Z d527cbbb87e3: Pull complete 2024-06-26T07:03:43.3587467Z b57676e46aee: Pull complete 2024-06-26T07:03:43.9119989Z e82932cb002c: Verifying Checksum 2024-06-26T07:03:43.9120424Z e82932cb002c: Download complete 2024-06-26T07:03:43.9873811Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:03:43.9874287Z ca9dd73e3e93: Download complete 2024-06-26T07:03:44.0557217Z 16d76a79d5d6: Download complete 2024-06-26T07:03:44.1368040Z 4bdf41c9ecbe: Download complete 2024-06-26T07:03:45.1078714Z f4863801e30c: Verifying Checksum 2024-06-26T07:03:45.1079373Z f4863801e30c: Download complete 2024-06-26T07:03:45.1879279Z d37dd9ed8948: Verifying Checksum 2024-06-26T07:03:45.1879800Z d37dd9ed8948: Download complete 2024-06-26T07:03:45.2595166Z cf2f34f225db: Verifying Checksum 2024-06-26T07:03:45.2595656Z cf2f34f225db: Download complete 2024-06-26T07:03:45.3373605Z df546d7b3e34: Verifying Checksum 2024-06-26T07:03:54.2366557Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:03:54.2367102Z a8c1e85b5e14: Download complete 2024-06-26T07:03:54.3146734Z 96d0e2032184: Verifying Checksum 2024-06-26T07:03:54.3147398Z 96d0e2032184: Download complete 2024-06-26T07:03:54.3822284Z 06e12ce59790: Download complete 2024-06-26T07:03:54.5324451Z ed712f63cde7: Verifying Checksum 2024-06-26T07:03:54.5325434Z ed712f63cde7: Download complete 2024-06-26T07:03:54.6094178Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:03:54.6095212Z a97a71ef8a4e: Download complete 2024-06-26T07:03:54.6885940Z 25cd0c482ad6: Download complete 2024-06-26T07:03:56.0779754Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:03:56.0780478Z 9bf94bc45658: Download complete 2024-06-26T07:03:56.1528301Z de45a14c9b22: Verifying Checksum 2024-06-26T07:03:56.1528895Z de45a14c9b22: Download complete 2024-06-26T07:03:56.2242106Z e99241e77209: Verifying Checksum 2024-06-26T07:03:56.2242676Z e99241e77209: Download complete 2024-06-26T07:03:56.2955632Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:03:56.2956369Z 35e8f4e2a42f: Download complete 2024-06-26T07:03:56.3757378Z a823ffa4c541: Verifying Checksum 2024-06-26T07:03:56.3758062Z a823ffa4c541: Download complete 2024-06-26T07:03:56.4783246Z 44ba272eae20: Verifying Checksum 2024-06-26T07:03:56.4783796Z 44ba272eae20: Download complete 2024-06-26T07:04:01.0104636Z da02d278e524: Verifying Checksum 2024-06-26T07:04:01.0105216Z da02d278e524: Download complete 2024-06-26T07:04:01.0966704Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:04:01.0967177Z 73a0ce681c29: Download complete 2024-06-26T07:04:01.1805847Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:04:01.1806457Z 49737bd5ae70: Download complete 2024-06-26T07:04:01.6008827Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:04:01.6009473Z 9cd4c745f8e4: Download complete 2024-06-26T07:04:01.7018970Z b2950a036534: Download complete 2024-06-26T07:04:01.7912239Z 25e19d81d9fc: Download complete 2024-06-26T07:04:02.1160821Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:04:02.1161534Z 60156b40fcbe: Download complete 2024-06-26T07:04:02.2104730Z 7a868239227f: Verifying Checksum 2024-06-26T07:04:02.2105316Z 7a868239227f: Download complete 2024-06-26T07:04:02.2852528Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:04:02.2852990Z 83911d7c7ecf: Download complete 2024-06-26T07:04:02.3738504Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:04:02.3739113Z b3c5d66c679f: Download complete 2024-06-26T07:04:06.5247368Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:04:06.5248001Z 7ed32bc8e469: Download complete 2024-06-26T07:04:06.6062741Z c1068d03a811: Verifying Checksum 2024-06-26T07:04:06.6063334Z c1068d03a811: Download complete 2024-06-26T07:04:06.7077955Z 671b4a301d81: Verifying Checksum 2024-06-26T07:04:06.7078606Z 671b4a301d81: Download complete 2024-06-26T07:04:07.0356330Z a8c1e85b5e14: Pull complete 2024-06-26T07:04:07.2031406Z 67760cf5ec46: Download complete 2024-06-26T07:04:07.2710151Z a41a8d1c11c8: Pull complete 2024-06-26T07:04:07.2802380Z ddc9740bcea2: Download complete 2024-06-26T07:04:07.3829506Z fb2e97768560: Verifying Checksum 2024-06-26T07:04:07.3830048Z fb2e97768560: Download complete 2024-06-26T07:04:07.4841000Z b599c5a8c7cb: Download complete 2024-06-26T07:04:07.5159530Z 0c1227890755: Pull complete 2024-06-26T07:04:07.5607672Z e08e88c13668: Download complete 2024-06-26T07:04:07.7467243Z d8d1234baab3: Pull complete 2024-06-26T07:04:13.2447809Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:04:13.2448455Z 3d914950f8ad: Download complete 2024-06-26T07:04:13.3170885Z cf7c2cb896cb: Verifying Checksum 2024-06-26T07:04:13.3171657Z cf7c2cb896cb: Download complete 2024-06-26T07:04:13.3993290Z 3a778ddfac2a: Download complete 2024-06-26T07:04:13.4725555Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:04:13.4726161Z 236c8f0245f4: Download complete 2024-06-26T07:04:13.4810840Z 4f4fb700ef54: Verifying Checksum 2024-06-26T07:04:13.4811650Z 4f4fb700ef54: Download complete 2024-06-26T07:04:13.5539748Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:04:13.5540410Z dda1ce302b9a: Download complete 2024-06-26T07:04:13.6302556Z e797fe86caef: Verifying Checksum 2024-06-26T07:04:13.6303162Z e797fe86caef: Download complete 2024-06-26T07:04:13.7652872Z 967190076fc0: Verifying Checksum 2024-06-26T07:04:13.7653423Z 967190076fc0: Download complete 2024-06-26T07:04:13.8398412Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:04:13.8398944Z 4fb0a83f0685: Download complete 2024-06-26T07:04:13.9144460Z 68af2ec4e30a: Verifying Checksum 2024-06-26T07:04:13.9145119Z 68af2ec4e30a: Download complete 2024-06-26T07:04:13.9934386Z cc91933e985a: Download complete 2024-06-26T07:04:14.0844099Z 9f3187916e7d: Verifying Checksum 2024-06-26T07:04:14.0844759Z 9f3187916e7d: Download complete 2024-06-26T07:04:14.2079289Z 526083994857: Verifying Checksum 2024-06-26T07:04:14.2079832Z 526083994857: Download complete 2024-06-26T07:04:14.2854937Z 05a6f6dcc1c0: Download complete 2024-06-26T07:04:14.8899311Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:04:14.8899889Z 6bc9dc8315b0: Download complete 2024-06-26T07:04:14.9711299Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:04:14.9712399Z 1b0e27c35f07: Download complete 2024-06-26T07:04:37.4777259Z 12ba7431ff38: Download complete 2024-06-26T07:04:37.5560596Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:04:37.5561079Z 789eff6c51aa: Download complete 2024-06-26T07:04:37.6330976Z de0f8a8c45c6: Download complete 2024-06-26T07:04:39.1727731Z 2060028f7488: Verifying Checksum 2024-06-26T07:04:39.1728305Z 2060028f7488: Download complete 2024-06-26T07:04:39.2489214Z e412c7f3b2de: Download complete 2024-06-26T07:04:42.8492736Z 95237e7311a1: Download complete 2024-06-26T07:05:05.8636555Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:05:23.2429432Z 7ed32bc8e469: Pull complete 2024-06-26T07:05:23.4716902Z ec1e7978c1fe: Pull complete 2024-06-26T07:05:23.6960656Z 309b872da805: Pull complete 2024-06-26T07:05:30.9649069Z e82932cb002c: Pull complete 2024-06-26T07:05:31.1712671Z ca9dd73e3e93: Pull complete 2024-06-26T07:05:31.4043907Z 16d76a79d5d6: Pull complete 2024-06-26T07:05:31.6287327Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:05:34.0056797Z f4863801e30c: Pull complete 2024-06-26T07:05:34.2299435Z d37dd9ed8948: Pull complete 2024-06-26T07:05:34.4600271Z cf2f34f225db: Pull complete 2024-06-26T07:05:34.6844757Z df546d7b3e34: Pull complete 2024-06-26T07:06:23.7274720Z 3d914950f8ad: Pull complete 2024-06-26T07:06:23.9378293Z 96d0e2032184: Pull complete 2024-06-26T07:06:24.1521679Z 06e12ce59790: Pull complete 2024-06-26T07:06:24.3011997Z ed712f63cde7: Pull complete 2024-06-26T07:06:24.4718382Z a97a71ef8a4e: Pull complete 2024-06-26T07:06:24.5786054Z 25cd0c482ad6: Pull complete 2024-06-26T07:06:27.9865043Z 9bf94bc45658: Pull complete 2024-06-26T07:06:28.1179931Z de45a14c9b22: Pull complete 2024-06-26T07:06:28.2861090Z e99241e77209: Pull complete 2024-06-26T07:06:28.4538177Z 35e8f4e2a42f: Pull complete 2024-06-26T07:06:28.5698386Z a823ffa4c541: Pull complete 2024-06-26T07:06:28.7311224Z 44ba272eae20: Pull complete 2024-06-26T07:06:36.9573259Z da02d278e524: Pull complete 2024-06-26T07:06:37.1795240Z 73a0ce681c29: Pull complete 2024-06-26T07:06:37.4056724Z 49737bd5ae70: Pull complete 2024-06-26T07:06:38.3964116Z 9cd4c745f8e4: Pull complete 2024-06-26T07:06:38.6002143Z b2950a036534: Pull complete 2024-06-26T07:06:38.7595468Z 25e19d81d9fc: Pull complete 2024-06-26T07:06:39.2266006Z 60156b40fcbe: Pull complete 2024-06-26T07:06:39.4367076Z 7a868239227f: Pull complete 2024-06-26T07:06:39.7779442Z 83911d7c7ecf: Pull complete 2024-06-26T07:06:39.9934963Z b3c5d66c679f: Pull complete 2024-06-26T07:07:19.1887328Z 84b2c7a89982: Pull complete 2024-06-26T07:07:19.3688526Z c1068d03a811: Pull complete 2024-06-26T07:07:19.4069981Z 671b4a301d81: Pull complete 2024-06-26T07:07:21.3314075Z 67760cf5ec46: Pull complete 2024-06-26T07:07:21.3883040Z ddc9740bcea2: Pull complete 2024-06-26T07:07:21.4389862Z fb2e97768560: Pull complete 2024-06-26T07:07:21.5356966Z b599c5a8c7cb: Pull complete 2024-06-26T07:07:21.5728583Z e08e88c13668: Pull complete 2024-06-26T07:07:55.6698562Z 12ba7431ff38: Pull complete 2024-06-26T07:07:55.8212734Z cf7c2cb896cb: Pull complete 2024-06-26T07:07:56.0047139Z 3a778ddfac2a: Pull complete 2024-06-26T07:07:56.4629780Z 236c8f0245f4: Pull complete 2024-06-26T07:07:56.6958449Z 4f4fb700ef54: Pull complete 2024-06-26T07:07:56.9141711Z dda1ce302b9a: Pull complete 2024-06-26T07:07:57.1415046Z e797fe86caef: Pull complete 2024-06-26T07:07:57.5145149Z 967190076fc0: Pull complete 2024-06-26T07:07:57.7160369Z 4fb0a83f0685: Pull complete 2024-06-26T07:07:57.9456342Z 68af2ec4e30a: Pull complete 2024-06-26T07:07:58.1626773Z cc91933e985a: Pull complete 2024-06-26T07:07:58.4031144Z 9f3187916e7d: Pull complete 2024-06-26T07:07:59.4801648Z 526083994857: Pull complete 2024-06-26T07:07:59.6357498Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:08:01.6202362Z 6bc9dc8315b0: Pull complete 2024-06-26T07:08:01.8339767Z 1b0e27c35f07: Pull complete 2024-06-26T07:08:15.2110903Z 95237e7311a1: Pull complete 2024-06-26T07:08:15.2723947Z 789eff6c51aa: Pull complete 2024-06-26T07:08:15.3136022Z de0f8a8c45c6: Pull complete 2024-06-26T07:08:16.0144828Z 2060028f7488: Pull complete 2024-06-26T07:08:16.0510561Z e412c7f3b2de: Pull complete 2024-06-26T07:08:16.2690320Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:08:16.2722672Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:16.2752625Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:16.2791450Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:08:16.2792793Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:08:16.2800821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:16.2801312Z env: 2024-06-26T07:08:16.2801593Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:16.2801940Z ##[endgroup] 2024-06-26T07:08:16.2953836Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:08:16.2954432Z with: 2024-06-26T07:08:16.2954711Z driver-version: 550.54.15 2024-06-26T07:08:16.2955045Z env: 2024-06-26T07:08:16.2955318Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:16.2955657Z ##[endgroup] 2024-06-26T07:08:16.2994188Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:08:16.2994787Z with: 2024-06-26T07:08:16.2995061Z timeout_minutes: 10 2024-06-26T07:08:16.2995381Z max_attempts: 3 2024-06-26T07:08:16.3025994Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:08:16.3056815Z retry_wait_seconds: 10 2024-06-26T07:08:16.3057255Z polling_interval_seconds: 1 2024-06-26T07:08:16.3057621Z warning_on_retry: true 2024-06-26T07:08:16.3057969Z continue_on_error: false 2024-06-26T07:08:16.3058301Z env: 2024-06-26T07:08:16.3058571Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:16.3058918Z DRIVER_VERSION: 550.54.15 2024-06-26T07:08:16.3059259Z ##[endgroup] 2024-06-26T07:08:16.3576983Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:08:16.3577954Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:08:16.3578804Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:08:16.6562738Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:16.6949412Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:08:16.7187401Z No Packages marked for removal 2024-06-26T07:08:16.7307981Z + install_nvidia_driver_common 2024-06-26T07:08:16.7310136Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:08:16.7310648Z + lspci 2024-06-26T07:08:16.7311019Z Before installing NVIDIA driver 2024-06-26T07:08:16.7388705Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:08:16.7389538Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:08:16.7390709Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:08:16.7392098Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:08:16.7393031Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:08:16.7393861Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:08:16.7394704Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:08:16.7395553Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:08:16.7396212Z + lsmod 2024-06-26T07:08:16.7403764Z Module Size Used by 2024-06-26T07:08:16.7404423Z nvidia_modeset 1339392 0 2024-06-26T07:08:16.7404979Z veth 16384 0 2024-06-26T07:08:16.7405422Z nvidia_uvm 4599808 0 2024-06-26T07:08:16.7405951Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:08:16.7406501Z drm 421888 1 nvidia 2024-06-26T07:08:16.7407199Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:08:16.7407742Z backlight 16384 1 nvidia_modeset 2024-06-26T07:08:16.7408252Z xt_conntrack 16384 1 2024-06-26T07:08:16.7408707Z ipt_MASQUERADE 16384 1 2024-06-26T07:08:16.7409209Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:08:16.7409733Z nf_conntrack_netlink 49152 0 2024-06-26T07:08:16.7410247Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:08:16.7410775Z xfrm_user 45056 1 2024-06-26T07:08:16.7411353Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:08:16.7412048Z iptable_nat 16384 1 2024-06-26T07:08:16.7412624Z nf_conntrack_ipv4 16384 3 2024-06-26T07:08:16.7413128Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:08:16.7413674Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:08:16.7414526Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:08:16.7415678Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:08:16.7417127Z xt_addrtype 16384 2 2024-06-26T07:08:16.7417797Z iptable_filter 16384 1 2024-06-26T07:08:16.7418422Z br_netfilter 24576 0 2024-06-26T07:08:16.7419089Z bridge 172032 1 br_netfilter 2024-06-26T07:08:16.7419821Z stp 16384 1 bridge 2024-06-26T07:08:16.7420464Z llc 16384 2 bridge,stp 2024-06-26T07:08:16.7421047Z overlay 86016 0 2024-06-26T07:08:16.7421502Z sunrpc 393216 1 2024-06-26T07:08:16.7421954Z dm_mirror 28672 0 2024-06-26T07:08:16.7422432Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:08:16.7423169Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:08:16.7423757Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:08:16.7424331Z dax 69632 1 dm_mod 2024-06-26T07:08:16.7424817Z crc32_pclmul 16384 0 2024-06-26T07:08:16.7425383Z ghash_clmulni_intel 16384 0 2024-06-26T07:08:16.7425849Z pcbc 16384 0 2024-06-26T07:08:16.7426345Z aesni_intel 188416 0 2024-06-26T07:08:16.7426806Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:08:16.7427329Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:08:16.7427852Z glue_helper 16384 1 aesni_intel 2024-06-26T07:08:16.7428481Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:08:16.7429083Z mousedev 24576 0 2024-06-26T07:08:16.7429525Z evdev 20480 3 2024-06-26T07:08:16.7429965Z psmouse 32768 0 2024-06-26T07:08:16.7430409Z button 16384 0 2024-06-26T07:08:16.7430848Z ena 139264 0 2024-06-26T07:08:16.7431290Z ptp 20480 1 ena 2024-06-26T07:08:16.7432029Z pps_core 20480 1 ptp 2024-06-26T07:08:16.7432511Z crc32c_intel 24576 0 2024-06-26T07:08:16.7432950Z autofs4 49152 2 2024-06-26T07:08:16.7433385Z + modinfo nvidia 2024-06-26T07:08:16.7434115Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:08:16.7434866Z alias: char-major-195-* 2024-06-26T07:08:16.7435319Z version: 550.54.15 2024-06-26T07:08:16.7435739Z supported: external 2024-06-26T07:08:16.7436162Z license: NVIDIA 2024-06-26T07:08:16.7436607Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:08:16.7437160Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:08:16.7437686Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:08:16.7438224Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:08:16.7438795Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:08:16.7439358Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:08:16.7439918Z depends: i2c-core,drm 2024-06-26T07:08:16.7440476Z retpoline: Y 2024-06-26T07:08:16.7440868Z name: nvidia 2024-06-26T07:08:16.7441498Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:08:16.7442223Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:08:16.7442936Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:08:16.7443598Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:08:16.7444111Z parm: NVreg_RmLogonRC:int 2024-06-26T07:08:16.7444620Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:08:16.7445148Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:08:16.7445654Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:08:16.7446163Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:08:16.7446772Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:08:16.7447396Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:08:16.7447942Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:08:16.7448446Z parm: NVreg_EnableMSI:int 2024-06-26T07:08:16.7448936Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:08:16.7449470Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:08:16.7450067Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:08:16.7450723Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:08:16.7451355Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:08:16.7452029Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:16.7452698Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:08:16.7453382Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:16.7454046Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:08:16.7454784Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:08:16.7455388Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:08:16.7456008Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:08:16.7456572Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:08:16.7457113Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:08:16.7457673Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:08:16.7458214Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:08:16.7458726Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:08:16.7459304Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:08:16.7459901Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:08:16.7460450Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:08:16.7461171Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:08:16.7461748Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:08:16.7462321Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:08:16.7462902Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:08:16.7463459Z parm: NVreg_RmMsg:charp 2024-06-26T07:08:16.7463944Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:08:16.7464480Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:08:16.7465023Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:08:16.7465542Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:08:16.7466090Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:08:16.7466630Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:08:16.7467146Z parm: rm_firmware_active:charp 2024-06-26T07:08:16.7467627Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:08:16.7468079Z ++ command -v nvidia-smi 2024-06-26T07:08:16.7468549Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:08:16.7469012Z + set +e 2024-06-26T07:08:16.7469589Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:08:18.7247011Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:08:18.7247800Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:08:18.7248494Z + '[' 0 -ne 0 ']' 2024-06-26T07:08:18.7249031Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:08:18.7249851Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:08:18.7250755Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:08:18.7251527Z + set -e 2024-06-26T07:08:18.7251891Z + '[' 1 -eq 0 ']' 2024-06-26T07:08:18.7252538Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:08:18.7253288Z + post_install_nvidia_driver_common 2024-06-26T07:08:18.7253771Z + sudo modprobe nvidia 2024-06-26T07:08:18.7335410Z + echo 'After installing NVIDIA driver' 2024-06-26T07:08:18.7336203Z + lspci 2024-06-26T07:08:18.7336716Z After installing NVIDIA driver 2024-06-26T07:08:18.7412514Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:08:18.7413418Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:08:18.7414423Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:08:18.7415292Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:08:18.7416111Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:08:18.7416922Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:08:18.7417708Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:08:18.7418556Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:08:18.7419212Z + lsmod 2024-06-26T07:08:18.7425633Z Module Size Used by 2024-06-26T07:08:18.7426275Z nvidia_modeset 1339392 0 2024-06-26T07:08:18.7426771Z veth 16384 0 2024-06-26T07:08:18.7427224Z nvidia_uvm 4599808 0 2024-06-26T07:08:18.7427742Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:08:18.7428461Z drm 421888 1 nvidia 2024-06-26T07:08:18.7428957Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:08:18.7429496Z backlight 16384 1 nvidia_modeset 2024-06-26T07:08:18.7430007Z xt_conntrack 16384 1 2024-06-26T07:08:18.7430489Z ipt_MASQUERADE 16384 1 2024-06-26T07:08:18.7430998Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:08:18.7431540Z nf_conntrack_netlink 49152 0 2024-06-26T07:08:18.7432345Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:08:18.7432876Z xfrm_user 45056 1 2024-06-26T07:08:18.7433336Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:08:18.7433821Z iptable_nat 16384 1 2024-06-26T07:08:18.7434264Z nf_conntrack_ipv4 16384 3 2024-06-26T07:08:18.7434759Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:08:18.7435334Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:08:18.7435957Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:08:18.7436964Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:08:18.7437863Z xt_addrtype 16384 2 2024-06-26T07:08:18.7438318Z iptable_filter 16384 1 2024-06-26T07:08:18.7438767Z br_netfilter 24576 0 2024-06-26T07:08:18.7439236Z bridge 172032 1 br_netfilter 2024-06-26T07:08:18.7439745Z stp 16384 1 bridge 2024-06-26T07:08:18.7440236Z llc 16384 2 bridge,stp 2024-06-26T07:08:18.7440714Z overlay 86016 0 2024-06-26T07:08:18.7441159Z sunrpc 393216 1 2024-06-26T07:08:18.7441603Z dm_mirror 28672 0 2024-06-26T07:08:18.7442074Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:08:18.7442711Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:08:18.7443290Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:08:18.7443857Z dax 69632 1 dm_mod 2024-06-26T07:08:18.7444326Z crc32_pclmul 16384 0 2024-06-26T07:08:18.7444772Z ghash_clmulni_intel 16384 0 2024-06-26T07:08:18.7445340Z pcbc 16384 0 2024-06-26T07:08:18.7445787Z aesni_intel 188416 0 2024-06-26T07:08:18.7446250Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:08:18.7446761Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:08:18.7447280Z glue_helper 16384 1 aesni_intel 2024-06-26T07:08:18.7447904Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:08:18.7448501Z mousedev 24576 0 2024-06-26T07:08:18.7448944Z evdev 20480 3 2024-06-26T07:08:18.7449383Z psmouse 32768 0 2024-06-26T07:08:18.7449821Z button 16384 0 2024-06-26T07:08:18.7450247Z ena 139264 0 2024-06-26T07:08:18.7450686Z ptp 20480 1 ena 2024-06-26T07:08:18.7451154Z pps_core 20480 1 ptp 2024-06-26T07:08:18.7451618Z crc32c_intel 24576 0 2024-06-26T07:08:18.7452053Z autofs4 49152 2 2024-06-26T07:08:18.7452483Z + modinfo nvidia 2024-06-26T07:08:18.7453251Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:08:18.7454020Z alias: char-major-195-* 2024-06-26T07:08:18.7454472Z version: 550.54.15 2024-06-26T07:08:18.7454889Z supported: external 2024-06-26T07:08:18.7455296Z license: NVIDIA 2024-06-26T07:08:18.7455738Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:08:18.7456282Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:08:18.7456811Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:08:18.7457346Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:08:18.7457911Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:08:18.7458467Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:08:18.7459112Z depends: i2c-core,drm 2024-06-26T07:08:18.7459541Z retpoline: Y 2024-06-26T07:08:18.7459911Z name: nvidia 2024-06-26T07:08:18.7460537Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:08:18.7461361Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:08:18.7462076Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:08:18.7462725Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:08:18.7463240Z parm: NVreg_RmLogonRC:int 2024-06-26T07:08:18.7463743Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:08:18.7464266Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:08:18.7464765Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:08:18.7465313Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:08:18.7465922Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:08:18.7466549Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:08:18.7467103Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:08:18.7467603Z parm: NVreg_EnableMSI:int 2024-06-26T07:08:18.7468093Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:08:18.7468630Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:08:18.7469223Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:08:18.7469875Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:08:18.7470502Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:08:18.7471187Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:18.7472059Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:08:18.7472755Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:08:18.7473430Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:08:18.7474004Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:08:18.7474610Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:08:18.7475273Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:08:18.7475851Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:08:18.7476471Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:08:18.7477029Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:08:18.7477567Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:08:18.7478081Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:08:18.7478652Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:08:18.7479259Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:08:18.7479818Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:08:18.7480388Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:08:18.7480957Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:08:18.7481519Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:08:18.7482092Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:08:18.7482642Z parm: NVreg_RmMsg:charp 2024-06-26T07:08:18.7483115Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:08:18.7483655Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:08:18.7484201Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:08:18.7484728Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:08:18.7485295Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:08:18.7485868Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:08:18.7486396Z parm: rm_firmware_active:charp 2024-06-26T07:08:18.7486855Z + set +e 2024-06-26T07:08:18.7487228Z + nvidia-smi 2024-06-26T07:08:20.2950668Z Wed Jun 26 07:08:20 2024 2024-06-26T07:08:20.2951518Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:20.2952561Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:08:20.2953340Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:08:20.2954364Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:08:20.2955260Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:08:20.2955954Z | | | MIG M. | 2024-06-26T07:08:20.2956499Z |=========================================+========================+======================| 2024-06-26T07:08:20.3057591Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:08:20.3058309Z | 0% 32C P0 57W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:08:20.3058937Z | | | N/A | 2024-06-26T07:08:20.3059717Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:08:20.3060326Z 2024-06-26T07:08:20.3061073Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:20.3061705Z | Processes: | 2024-06-26T07:08:20.3062401Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:08:20.3063079Z | ID ID Usage | 2024-06-26T07:08:20.3063634Z |=========================================================================================| 2024-06-26T07:08:20.3064273Z | No running processes found | 2024-06-26T07:08:20.3065014Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:08:20.9136538Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:08:22.4722384Z NVIDIA A10G 2024-06-26T07:08:22.9254969Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:08:22.9255512Z + '[' 0 -eq 0 ']' 2024-06-26T07:08:22.9255920Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:08:22.9256648Z + set -e 2024-06-26T07:08:22.9256957Z INFO: Ignoring allowed status 0 2024-06-26T07:08:22.9259177Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:08:22.9261664Z + sudo yum install -y yum-utils 2024-06-26T07:08:23.2283297Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:24.7366402Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:08:24.7367224Z Nothing to do 2024-06-26T07:08:24.8852167Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:08:24.8853029Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:24.8854065Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:25.1848612Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:25.2117731Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:08:25.2118925Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:08:25.2119864Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:08:25.2225200Z + sudo yum install -y nvidia-docker2 2024-06-26T07:08:25.5054579Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:08:26.9541269Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:08:26.9541936Z Nothing to do 2024-06-26T07:08:27.1024716Z + sudo systemctl restart docker 2024-06-26T07:09:07.4093379Z Command completed after 1 attempt(s). 2024-06-26T07:09:07.4153629Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:07.4154406Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:09:07.4155219Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:09:07.4155871Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:07.4163647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:07.4164141Z env: 2024-06-26T07:09:07.4164417Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:07.4164856Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:07.4165342Z ##[endgroup] 2024-06-26T07:09:07.6393005Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:09:07.6554397Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:09:07.6654927Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:09:07.7608720Z Prepare all required actions 2024-06-26T07:09:07.7609272Z Getting action download info 2024-06-26T07:09:07.8908881Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:09:08.0861788Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:09:08.2633303Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:09:08.2633778Z with: 2024-06-26T07:09:08.2634118Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:08.2634574Z s3-bucket: gha-artifacts 2024-06-26T07:09:08.2634912Z env: 2024-06-26T07:09:08.2635179Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:08.2635624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:08.2636100Z ##[endgroup] 2024-06-26T07:09:08.2666909Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:09:08.2667346Z with: 2024-06-26T07:09:08.2667715Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:08.2668168Z s3-bucket: gha-artifacts 2024-06-26T07:09:08.2668522Z region: us-east-1 2024-06-26T07:09:08.2668819Z env: 2024-06-26T07:09:08.2669101Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:08.2669543Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:08.2670024Z ##[endgroup] 2024-06-26T07:09:08.6763811Z (node:30376) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:09:08.6764507Z 2024-06-26T07:09:08.6764767Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:09:08.6765499Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:09:08.6766478Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:09:08.7564538Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:09:08.7565634Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:09:16.2470104Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:09:16.2476259Z Artifact download has finished successfully 2024-06-26T07:09:16.2619836Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:09:16.2620276Z unzip -o artifacts.zip 2024-06-26T07:09:16.2628179Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:16.2628687Z env: 2024-06-26T07:09:16.2628958Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:16.2629403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:16.2629879Z ##[endgroup] 2024-06-26T07:09:16.2656432Z Archive: artifacts.zip 2024-06-26T07:09:16.2658230Z creating: dist/ 2024-06-26T07:09:18.2341661Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:09:18.2342782Z creating: build/custom_test_artifacts/ 2024-06-26T07:09:18.2343362Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:09:18.2344086Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:09:18.2345132Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:18.2348187Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:18.2349138Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:18.2350070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:18.2351042Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:18.2352377Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:18.2353523Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:18.2354742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:18.2355737Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:18.2356715Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:18.2357834Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:18.2358975Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:18.2360093Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:18.2361208Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:18.2362341Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:18.2363489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:18.2364490Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:18.2365481Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:18.2402322Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:18.2440158Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:18.2441574Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:18.2484507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:18.2485904Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:18.2487312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:18.2488766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:18.2490370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:18.2491747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:18.2493117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:18.2494493Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:18.2495826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:18.2497239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:18.2498457Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:18.2499650Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:18.2500990Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:18.2502166Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:18.2503373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:18.2554685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:18.2612213Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:18.2613389Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:18.2614358Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:18.2615208Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:18.2616095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:18.2617003Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:09:18.2618010Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:09:18.2619138Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:09:18.2620226Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:09:18.2621310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:09:18.2622368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:09:18.2623479Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:09:18.2624536Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:09:18.2625592Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:09:18.2626625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:09:18.2639991Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:09:18.2767638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:09:18.2768627Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:09:18.2769702Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:09:18.2770880Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:09:18.2772021Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:09:18.2773137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:09:18.2774237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:09:18.2775342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:09:18.2776452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:09:18.2777673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:09:18.2778770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:09:18.2793055Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:09:18.2872365Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:09:18.2873637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:18.2874695Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:18.2875764Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:09:18.2876667Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:09:18.2877541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:18.2878419Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:09:18.2879242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:09:18.2880004Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:09:18.2880799Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:09:18.2987587Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:09:18.3048018Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:09:18.3048830Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:09:18.3049520Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:09:18.3050311Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:18.3055597Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:18.3056551Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:18.3057459Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:18.3058418Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:18.3059353Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:18.3060481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:18.3061649Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:18.3062629Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:18.3063601Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:18.3064706Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:18.3065822Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:18.3066920Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:18.3068020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:18.3069147Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:18.3070268Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:18.3071270Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:18.3072667Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:18.3109072Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:18.3146770Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:18.3148168Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:18.3192568Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:18.3194084Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:18.3195584Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:18.3197034Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:18.3198409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:18.3199751Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:18.3201111Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:18.3202458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:18.3203778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:18.3205023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:18.3206225Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:18.3207410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:18.3208600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:18.3209756Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:18.3210929Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:18.3259352Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:18.3317176Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:18.3318333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:18.3319293Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:18.3320125Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:18.3320998Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:18.3321922Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:09:18.3322964Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:09:18.3324124Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:09:18.3325236Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:09:18.3326281Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:09:18.3327512Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:09:18.3328590Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:09:18.3329675Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:09:18.3330742Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:09:18.3331814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:09:18.3343737Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:09:18.3405103Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:09:18.3406284Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:18.3407329Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:18.3408284Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:09:18.3409171Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:09:18.3410043Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:18.3410911Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:09:18.3411734Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:09:18.3412488Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:09:18.3413272Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:09:18.3459315Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:09:18.3460117Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:09:18.3481275Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:09:18.3482506Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:09:18.3483642Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:09:18.3484618Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:09:18.3485598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:09:18.3486631Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:09:18.3487644Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:09:18.3488782Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:09:18.3489932Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:09:18.3490972Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:09:18.3492009Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:09:18.3493188Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:09:18.3494375Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:09:18.3495543Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:09:18.3496717Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:09:18.3498050Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:09:18.3499241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:09:18.3500311Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:09:18.3501462Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:09:18.3516958Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:09:18.3554712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:09:18.3556288Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:09:18.3598630Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:09:18.3600088Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:09:18.3601560Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:09:18.3603068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:09:18.3604507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:09:18.3616380Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:09:18.3617863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:09:18.3619305Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:09:18.3620706Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:09:18.3622127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:09:18.3623408Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:09:18.3624653Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:09:18.3625917Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:09:18.3627155Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:09:18.3628401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:09:18.3681459Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:09:18.3740593Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:09:18.3742833Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:09:18.3743884Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:09:18.3744800Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:09:18.3745745Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:09:18.3746908Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:09:18.3748023Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:09:18.3749253Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:09:18.3750430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:09:18.3751552Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:09:18.3752936Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:09:18.3754185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:09:18.3755356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:09:18.3756518Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:09:18.3757677Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:09:18.3758894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:09:18.3870134Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:09:18.3871295Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:09:18.3872660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:09:18.3873959Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:09:18.3875209Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:09:18.3876388Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:09:18.3877610Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:09:18.3878823Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:09:18.3880038Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:09:18.3881252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:09:18.3882454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:09:18.3894466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:09:18.3947226Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:09:18.3948495Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:09:18.3949617Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:09:18.3950630Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:09:18.3951578Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:09:18.3952715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:09:18.3953659Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:09:18.3954677Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:09:18.3955510Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:09:18.3956352Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:09:18.4054231Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:09:18.4094717Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:09:18.4095344Z creating: build/lib/ 2024-06-26T07:09:18.4181090Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:09:18.4618413Z inflating: build/lib/libprotobuf.a 2024-06-26T07:09:18.4627672Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:09:18.4635791Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:09:18.4643438Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:09:18.4644066Z inflating: build/lib/libclog.a 2024-06-26T07:09:18.4661587Z inflating: build/lib/libnnpack.a 2024-06-26T07:09:18.4663658Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:09:18.4726088Z inflating: build/lib/libgtest.a 2024-06-26T07:09:18.4797962Z inflating: build/lib/libbenchmark.a 2024-06-26T07:09:18.4859678Z inflating: build/lib/libasmjit.a 2024-06-26T07:09:18.4866783Z inflating: build/lib/libittnotify.a 2024-06-26T07:09:18.4893550Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:09:18.5015793Z inflating: build/lib/libgloo.a 2024-06-26T07:09:18.5037549Z inflating: build/lib/libfmt.a 2024-06-26T07:09:18.5126073Z inflating: build/lib/libc10.so 2024-06-26T07:09:18.5127705Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:09:18.5128297Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:09:18.5129862Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:09:18.5149166Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:09:18.5166520Z inflating: build/lib/libgmock.a 2024-06-26T07:09:18.5167241Z inflating: build/lib/libgtest_main.a 2024-06-26T07:09:18.5167929Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:09:18.5652244Z inflating: build/lib/libprotoc.a 2024-06-26T07:09:18.6020981Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:09:19.5732257Z inflating: build/lib/libdnnl.a 2024-06-26T07:09:19.6282689Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:09:19.6336122Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:09:19.6337066Z inflating: build/lib/libgmock_main.a 2024-06-26T07:09:19.7536803Z inflating: build/lib/libfbgemm.a 2024-06-26T07:09:19.7782036Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:09:19.8278109Z inflating: build/lib/libkineto.a 2024-06-26T07:09:19.8455914Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:09:19.8496190Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:09:19.9159666Z inflating: build/lib/libonnx.a 2024-06-26T07:09:22.2402592Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:09:22.2405928Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:09:22.2409753Z inflating: build/lib/libshm.so 2024-06-26T07:09:24.2238489Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:09:24.2239222Z inflating: build/lib/libtorch.so 2024-06-26T07:09:24.2241740Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:09:25.0169768Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:09:25.2059814Z inflating: build/lib/libtorch_python.so 2024-06-26T07:09:25.2125944Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:09:25.2145934Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:09:25.2170961Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:09:25.2203689Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:09:25.2204156Z creating: build/bin/ 2024-06-26T07:09:25.2252710Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:09:25.2302668Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:09:25.2351136Z inflating: build/bin/c10_Device_test 2024-06-26T07:09:25.2408833Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:09:25.2460491Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:09:25.2507539Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:09:25.2556950Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:09:25.2610139Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:09:25.2663539Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:09:25.2718091Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:09:25.2786792Z inflating: build/bin/c10_cow_test 2024-06-26T07:09:25.2838076Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:09:25.2885472Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:09:25.2934357Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:09:25.2982761Z inflating: build/bin/c10_Half_test 2024-06-26T07:09:25.3037099Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:09:25.3090715Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:09:25.3138733Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:09:25.3192696Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:09:25.3242541Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:09:25.3291913Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:09:25.3339728Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:09:25.3389369Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:09:25.3442823Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:09:25.3492055Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:09:25.3546093Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:09:25.3599957Z inflating: build/bin/c10_complex_test 2024-06-26T07:09:25.3650865Z inflating: build/bin/c10_exception_test 2024-06-26T07:09:25.3700753Z inflating: build/bin/c10_flags_test 2024-06-26T07:09:25.3749090Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:09:25.3907401Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:09:25.3956833Z inflating: build/bin/c10_irange_test 2024-06-26T07:09:25.4008536Z inflating: build/bin/c10_lazy_test 2024-06-26T07:09:25.4063682Z inflating: build/bin/c10_logging_test 2024-06-26T07:09:25.4136047Z inflating: build/bin/c10_optional_test 2024-06-26T07:09:25.4196404Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:09:25.4249115Z inflating: build/bin/c10_registry_test 2024-06-26T07:09:25.4394025Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:09:25.4443756Z inflating: build/bin/c10_ssize_test 2024-06-26T07:09:25.4494034Z inflating: build/bin/c10_string_util_test 2024-06-26T07:09:25.4550740Z inflating: build/bin/c10_string_view_test 2024-06-26T07:09:25.4599457Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:09:25.4646503Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:09:25.4702246Z inflating: build/bin/c10_typeid_test 2024-06-26T07:09:25.5135872Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:09:25.5568122Z inflating: build/bin/protoc 2024-06-26T07:09:25.5619747Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:09:25.5671056Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:09:25.5722051Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:09:25.5772288Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:09:25.5823280Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:09:25.5874553Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:09:25.5921873Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:09:25.5973864Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:09:25.6291388Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:09:25.6624090Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:09:25.6968733Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:09:25.7019814Z inflating: build/bin/FileStoreTest 2024-06-26T07:09:25.7073692Z inflating: build/bin/TCPStoreTest 2024-06-26T07:09:25.7125309Z inflating: build/bin/HashStoreTest 2024-06-26T07:09:25.7138266Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:09:25.7190463Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:09:25.7194496Z inflating: build/bin/torch_shm_manager 2024-06-26T07:09:25.7197135Z inflating: build/bin/example_allreduce 2024-06-26T07:09:25.7250356Z inflating: build/bin/test_dist_autograd 2024-06-26T07:09:25.7317529Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:09:25.7319784Z inflating: build/bin/parallel_benchmark 2024-06-26T07:09:25.7385186Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:09:25.7393536Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:09:25.7726778Z inflating: build/bin/test_lazy 2024-06-26T07:09:25.8868906Z inflating: build/bin/test_api 2024-06-26T07:09:25.8940333Z inflating: build/bin/Dict_test 2024-06-26T07:09:25.8990585Z inflating: build/bin/Dimname_test 2024-06-26T07:09:25.9053364Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:09:25.9108660Z inflating: build/bin/NamedTensor_test 2024-06-26T07:09:25.9165841Z inflating: build/bin/apply_utils_test 2024-06-26T07:09:25.9223046Z inflating: build/bin/atest 2024-06-26T07:09:25.9284169Z inflating: build/bin/basic 2024-06-26T07:09:25.9337552Z inflating: build/bin/broadcast_test 2024-06-26T07:09:25.9386355Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:09:25.9442849Z inflating: build/bin/cpu_generator_test 2024-06-26T07:09:25.9494684Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:09:25.9584479Z inflating: build/bin/cpu_rng_test 2024-06-26T07:09:25.9633008Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:09:25.9681791Z inflating: build/bin/dlconvertor_test 2024-06-26T07:09:25.9738156Z inflating: build/bin/extension_backend_test 2024-06-26T07:09:25.9791822Z inflating: build/bin/half_test 2024-06-26T07:09:25.9884498Z inflating: build/bin/ivalue_test 2024-06-26T07:09:25.9932439Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:09:25.9985287Z inflating: build/bin/math_kernel_test 2024-06-26T07:09:26.0037922Z inflating: build/bin/memory_format_test 2024-06-26T07:09:26.0089805Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:09:26.0140653Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:09:26.0194884Z inflating: build/bin/native_test 2024-06-26T07:09:26.0243565Z inflating: build/bin/operator_name_test 2024-06-26T07:09:26.0294001Z inflating: build/bin/operators_test 2024-06-26T07:09:26.0343690Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:09:26.0409245Z inflating: build/bin/pow_test 2024-06-26T07:09:26.0464188Z inflating: build/bin/quantized_test 2024-06-26T07:09:26.0512751Z inflating: build/bin/reduce_ops_test 2024-06-26T07:09:26.0562108Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:09:26.0616716Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:09:26.0673543Z inflating: build/bin/scalar_test 2024-06-26T07:09:26.0723570Z inflating: build/bin/StorageUtils_test 2024-06-26T07:09:26.0774622Z inflating: build/bin/stride_properties_test 2024-06-26T07:09:26.0850579Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:09:26.0902866Z inflating: build/bin/test_parallel 2024-06-26T07:09:26.0905630Z inflating: build/bin/thread_init_test 2024-06-26T07:09:26.0959585Z inflating: build/bin/type_ptr_test 2024-06-26T07:09:26.1017627Z inflating: build/bin/type_test 2024-06-26T07:09:26.1068559Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:09:26.1069926Z inflating: build/bin/verify_api_visibility 2024-06-26T07:09:26.1137204Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:09:26.1187128Z inflating: build/bin/weakref_test 2024-06-26T07:09:26.1237057Z inflating: build/bin/wrapdim_test 2024-06-26T07:09:26.1287033Z inflating: build/bin/xla_tensor_test 2024-06-26T07:09:26.1345580Z inflating: build/bin/IListRef_test 2024-06-26T07:09:26.1448957Z inflating: build/bin/List_test 2024-06-26T07:09:26.1513688Z inflating: build/bin/KernelFunction_test 2024-06-26T07:09:26.1631812Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:09:26.1725275Z inflating: build/bin/kernel_function_test 2024-06-26T07:09:26.1849564Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:09:26.1949015Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:09:26.2008683Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:09:26.2101777Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:09:26.2150492Z inflating: build/bin/CppSignature_test 2024-06-26T07:09:26.2204717Z inflating: build/bin/backend_fallback_test 2024-06-26T07:09:26.2252519Z inflating: build/bin/op_allowlist_test 2024-06-26T07:09:26.2313991Z inflating: build/bin/inline_container_test 2024-06-26T07:09:26.2606627Z inflating: build/bin/op_registration_test 2024-06-26T07:09:26.2657638Z inflating: build/bin/cuda_apply_test 2024-06-26T07:09:26.2706788Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:09:26.2764105Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:09:26.2817225Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:09:26.2884572Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:09:26.2941810Z inflating: build/bin/cuda_complex_test 2024-06-26T07:09:26.2989787Z inflating: build/bin/cuda_device_test 2024-06-26T07:09:26.3045586Z inflating: build/bin/cuda_cub_test 2024-06-26T07:09:26.3094867Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:09:26.3156970Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:09:26.3211839Z inflating: build/bin/cuda_generator_test 2024-06-26T07:09:26.3260195Z inflating: build/bin/cuda_half_test 2024-06-26T07:09:26.3309759Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:09:26.3357393Z inflating: build/bin/cuda_optional_test 2024-06-26T07:09:26.3407326Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:09:26.3458724Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:09:26.3506623Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:09:26.3565744Z inflating: build/bin/cuda_stream_test 2024-06-26T07:09:26.3613577Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:09:26.3663485Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:09:26.3678009Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:09:26.3741847Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:09:26.3797422Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:09:26.3858640Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:09:26.3918828Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:09:26.4725035Z inflating: build/bin/test_tensorexpr 2024-06-26T07:09:26.5283883Z inflating: build/bin/test_jit 2024-06-26T07:09:26.5284366Z creating: .additional_ci_files/ 2024-06-26T07:09:26.5337464Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:09:26.5548054Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:09:26.5575501Z ##[group]Run rm artifacts.zip 2024-06-26T07:09:26.5575891Z rm artifacts.zip 2024-06-26T07:09:26.5583742Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:26.5584245Z env: 2024-06-26T07:09:26.5584538Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:26.5584986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:26.5585576Z ##[endgroup] 2024-06-26T07:09:26.6292205Z ##[group]Run df -H 2024-06-26T07:09:26.6292536Z df -H 2024-06-26T07:09:26.6300207Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:26.6300711Z env: 2024-06-26T07:09:26.6301089Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:26.6301542Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:26.6302024Z ##[endgroup] 2024-06-26T07:09:26.6328202Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:09:26.6328774Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:09:26.6329290Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:09:26.6329798Z tmpfs 34G 512k 34G 1% /run 2024-06-26T07:09:26.6330432Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:09:26.6330905Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-26T07:09:26.6331513Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-06-26T07:09:26.6368868Z Prepare all required actions 2024-06-26T07:09:26.6369310Z Getting action download info 2024-06-26T07:09:26.7535010Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:09:26.7535465Z with: 2024-06-26T07:09:26.7535718Z env: 2024-06-26T07:09:26.7535994Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:26.7536440Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:26.7536920Z ##[endgroup] 2024-06-26T07:09:26.7567100Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:09:26.7567539Z with: 2024-06-26T07:09:26.7567801Z name: td_results 2024-06-26T07:09:26.7568116Z s3-bucket: gha-artifacts 2024-06-26T07:09:26.7568463Z region: us-east-1 2024-06-26T07:09:26.7568757Z env: 2024-06-26T07:09:26.7569022Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:26.7569471Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:26.7569949Z ##[endgroup] 2024-06-26T07:09:27.1588435Z (node:30399) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:09:27.1589288Z 2024-06-26T07:09:27.1589551Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:09:27.1590280Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:09:27.1591118Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:09:27.2418936Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:09:27.2424739Z Artifact download has finished successfully 2024-06-26T07:09:27.2550522Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:09:27.2551001Z mkdir -p .additional_ci_files 2024-06-26T07:09:27.2551550Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:09:27.2559783Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:27.2560271Z env: 2024-06-26T07:09:27.2560548Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:27.2561007Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:27.2561495Z ##[endgroup] 2024-06-26T07:09:27.2596509Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:09:27.2606278Z ##[error]Process completed with exit code 1. 2024-06-26T07:09:27.2649855Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:09:27.2650338Z .github/scripts/parse_ref.py 2024-06-26T07:09:27.2657163Z shell: /usr/bin/bash -e {0} 2024-06-26T07:09:27.2657512Z env: 2024-06-26T07:09:27.2657797Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:27.2658277Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:27.2658792Z ##[endgroup] 2024-06-26T07:09:27.2885672Z Prepare all required actions 2024-06-26T07:09:27.2921834Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:09:27.2922288Z with: 2024-06-26T07:09:27.2922836Z github-token: *** 2024-06-26T07:09:27.2923141Z env: 2024-06-26T07:09:27.2923413Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:27.2923869Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:27.2924485Z ##[endgroup] 2024-06-26T07:09:27.2939052Z ##[group]Run set -eux 2024-06-26T07:09:27.2939389Z set -eux 2024-06-26T07:09:27.2939969Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:09:27.2947601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:27.2948096Z env: 2024-06-26T07:09:27.2948382Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:27.2948887Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:27.2949526Z GITHUB_TOKEN: *** 2024-06-26T07:09:27.2949836Z ##[endgroup] 2024-06-26T07:09:27.2969113Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-0189c918a23a6d1e2 2024-06-26T07:09:28.6530817Z setting job-id=26688912112 2024-06-26T07:09:28.6531850Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:28.6722755Z Prepare all required actions 2024-06-26T07:09:28.6723194Z Getting action download info 2024-06-26T07:09:28.7877459Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:09:28.7877921Z with: 2024-06-26T07:09:28.7878386Z github-token: *** 2024-06-26T07:09:28.7886418Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:09:28.7895079Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:28.7895814Z env: 2024-06-26T07:09:28.7896090Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:28.7896535Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:28.7897010Z ##[endgroup] 2024-06-26T07:09:28.7936658Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:09:28.7937205Z with: 2024-06-26T07:09:28.7937471Z shell: bash 2024-06-26T07:09:28.7937769Z timeout_minutes: 10 2024-06-26T07:09:28.7938093Z max_attempts: 5 2024-06-26T07:09:28.7938404Z retry_wait_seconds: 30 2024-06-26T07:09:28.7939509Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:09:28.7940771Z polling_interval_seconds: 1 2024-06-26T07:09:28.7941230Z warning_on_retry: true 2024-06-26T07:09:28.7941569Z continue_on_error: false 2024-06-26T07:09:28.7941900Z env: 2024-06-26T07:09:28.7942173Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:28.7942609Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:28.7943251Z GITHUB_TOKEN: *** 2024-06-26T07:09:28.7943564Z ##[endgroup] 2024-06-26T07:09:28.8428899Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:09:29.0590519Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:09:29.0743791Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:09:29.0892018Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:09:29.0900599Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:09:29.0922230Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:09:29.1127340Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:09:29.1139025Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:09:29.8425236Z Command completed after 1 attempt(s). 2024-06-26T07:09:29.8467011Z ##[group]Run set -x 2024-06-26T07:09:29.8467351Z set -x 2024-06-26T07:09:29.8467646Z  2024-06-26T07:09:29.8468199Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:09:29.8468878Z # in runner workspace 2024-06-26T07:09:29.8469398Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:09:29.8477416Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:29.8477914Z env: 2024-06-26T07:09:29.8478188Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:29.8478638Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:29.8479120Z ##[endgroup] 2024-06-26T07:09:29.8502779Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:09:29.8721282Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:09:29.8721860Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:09:29.8722334Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:09:29.8722737Z  2024-06-26T07:09:29.8723286Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:09:29.8723951Z # in runner workspace 2024-06-26T07:09:29.8724529Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:09:29.8725162Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:09:29.8725616Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:09:29.8733815Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:09:29.8742441Z  --selected-test-configs "" \ 2024-06-26T07:09:29.8742898Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:09:29.8743310Z  --tag "${TAG}" \ 2024-06-26T07:09:29.8743702Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:09:29.8744146Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:09:29.8744574Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:09:29.8752346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:29.8752849Z env: 2024-06-26T07:09:29.8753130Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:29.8753590Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:29.8754259Z GITHUB_TOKEN: *** 2024-06-26T07:09:29.8754937Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:29.8755683Z PR_NUMBER: 2024-06-26T07:09:29.8755991Z TAG: ciflow/inductor/129470 2024-06-26T07:09:29.8756346Z EVENT_NAME: push 2024-06-26T07:09:29.8756658Z SCHEDULE: 2024-06-26T07:09:29.8756946Z HEAD_BRANCH: 2024-06-26T07:09:29.8757237Z ##[endgroup] 2024-06-26T07:09:29.8775937Z Workflow: inductor 2024-06-26T07:09:29.8776837Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:30.1154430Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:09:30.2564767Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:09:30.2565232Z echo "Filtered matrix:" 2024-06-26T07:09:30.2573342Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:09:30.2582023Z  2024-06-26T07:09:30.2582307Z echo 2024-06-26T07:09:30.2582688Z echo "Is the current job unstable? False" 2024-06-26T07:09:30.2583150Z  2024-06-26T07:09:30.2583416Z echo 2024-06-26T07:09:30.2583774Z echo "Is keep-going label set? False" 2024-06-26T07:09:30.2584210Z  2024-06-26T07:09:30.2584471Z echo 2024-06-26T07:09:30.2584782Z echo "Renabled issues? " 2024-06-26T07:09:30.2592661Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:30.2593166Z env: 2024-06-26T07:09:30.2593435Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:30.2593884Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:30.2594365Z ##[endgroup] 2024-06-26T07:09:30.2615224Z Filtered matrix: 2024-06-26T07:09:30.2627383Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:09:30.2635823Z 2024-06-26T07:09:30.2635976Z Is the current job unstable? False 2024-06-26T07:09:30.2636252Z 2024-06-26T07:09:30.2636523Z Is keep-going label set? False 2024-06-26T07:09:30.2636783Z 2024-06-26T07:09:30.2636911Z Renabled issues? 2024-06-26T07:09:30.2689430Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:30.2690226Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:09:30.2697064Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:09:30.2697685Z env: 2024-06-26T07:09:30.2697966Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:30.2698434Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:30.2698946Z JOB_TIMEOUT: 240 2024-06-26T07:09:30.2699249Z ##[endgroup] 2024-06-26T07:09:30.2769815Z ##[group]Run set -x 2024-06-26T07:09:30.2770263Z set -x 2024-06-26T07:09:30.2770582Z  2024-06-26T07:09:30.2770947Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:09:30.2771522Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:09:30.2772121Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:09:30.2772668Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:09:30.2773103Z else 2024-06-26T07:09:30.2773448Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:09:30.2773896Z fi 2024-06-26T07:09:30.2774178Z  2024-06-26T07:09:30.2774669Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:09:30.2775497Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:09:30.2776229Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:09:30.2776856Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:09:30.2777311Z container_name=$(docker run \ 2024-06-26T07:09:30.2777728Z  ${GPU_FLAG:-} \ 2024-06-26T07:09:30.2778096Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:09:30.2778496Z  -e PR_NUMBER \ 2024-06-26T07:09:30.2778854Z  -e GITHUB_ACTIONS \ 2024-06-26T07:09:30.2779244Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:09:30.2779649Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:09:30.2780028Z  -e GITHUB_JOB \ 2024-06-26T07:09:30.2780380Z  -e GITHUB_RUN_ID \ 2024-06-26T07:09:30.2780759Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:09:30.2781238Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:09:30.2781624Z  -e JOB_ID \ 2024-06-26T07:09:30.2781967Z  -e JOB_NAME \ 2024-06-26T07:09:30.2782312Z  -e BASE_SHA \ 2024-06-26T07:09:30.2782658Z  -e BRANCH \ 2024-06-26T07:09:30.2782982Z  -e SHA1 \ 2024-06-26T07:09:30.2783326Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:09:30.2783726Z  -e IN_WHEEL_TEST \ 2024-06-26T07:09:30.2784102Z  -e SHARD_NUMBER \ 2024-06-26T07:09:30.2784464Z  -e TEST_CONFIG \ 2024-06-26T07:09:30.2784831Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:09:30.2785221Z  -e REENABLED_ISSUES \ 2024-06-26T07:09:30.2785632Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:09:30.2786046Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:09:30.2786443Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:09:30.2786817Z  -e NO_TD \ 2024-06-26T07:09:30.2787155Z  -e TD_DISTRIBUTED \ 2024-06-26T07:09:30.2787523Z  -e PR_LABELS \ 2024-06-26T07:09:30.2787926Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:09:30.2788384Z  -e SCCACHE_BUCKET \ 2024-06-26T07:09:30.2788770Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:09:30.2789176Z  -e XLA_CUDA \ 2024-06-26T07:09:30.2789572Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:09:30.2790082Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:09:30.2790591Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:09:30.2791095Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:09:30.2791556Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:09:30.2792117Z  -e DASHBOARD_TAG \ 2024-06-26T07:09:30.2792586Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:09:30.2793137Z  --security-opt seccomp=unconfined \ 2024-06-26T07:09:30.2793599Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:09:30.2793987Z  --ipc=host \ 2024-06-26T07:09:30.2794336Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:09:30.2794840Z  --tty \ 2024-06-26T07:09:30.2795152Z  --detach \ 2024-06-26T07:09:30.2795505Z  --name="${container_name}" \ 2024-06-26T07:09:30.2795911Z  --user jenkins \ 2024-06-26T07:09:30.2796464Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:09:30.2797024Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:09:30.2797457Z  "${DOCKER_IMAGE}" 2024-06-26T07:09:30.2797805Z ) 2024-06-26T07:09:30.2798206Z # Propagate download.pytorch.org IP to container 2024-06-26T07:09:30.2799115Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:09:30.2800068Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:09:30.2801004Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:09:30.2808949Z shell: /usr/bin/bash -e {0} 2024-06-26T07:09:30.2809301Z env: 2024-06-26T07:09:30.2809574Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:09:30.2810018Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:30.2810609Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:30.2811110Z PR_NUMBER: 2024-06-26T07:09:30.2811425Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:09:30.2811835Z GITHUB_WORKFLOW: inductor 2024-06-26T07:09:30.2812174Z GITHUB_JOB: test 2024-06-26T07:09:30.2812488Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:09:30.2812838Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:09:30.2813174Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:09:30.2813501Z JOB_ID: 26688912112 2024-06-26T07:09:30.2814176Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:30.2814905Z BRANCH: 2024-06-26T07:09:30.2815225Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:30.2815729Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:30.2816218Z TEST_CONFIG: dynamic_inductor_torchbench 2024-06-26T07:09:30.2816635Z SHARD_NUMBER: 2 2024-06-26T07:09:30.2816930Z NUM_TEST_SHARDS: 2 2024-06-26T07:09:30.2817246Z REENABLED_ISSUES: 2024-06-26T07:09:30.2817582Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:09:30.2817963Z VERBOSE_TEST_LOGS: False 2024-06-26T07:09:30.2818305Z NO_TEST_TIMEOUT: False 2024-06-26T07:09:30.2818636Z NO_TD: False 2024-06-26T07:09:30.2818929Z TD_DISTRIBUTED: False 2024-06-26T07:09:30.2819336Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:09:30.2819808Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:09:30.2820178Z SHM_SIZE: 2g 2024-06-26T07:09:30.2821286Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:30.2822421Z XLA_CUDA: 2024-06-26T07:09:30.2822885Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:30.2823487Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:09:30.2823916Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:09:30.2824316Z DASHBOARD_TAG: 2024-06-26T07:09:30.2824813Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:09:30.2825172Z ##[endgroup] 2024-06-26T07:09:30.2845577Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:09:30.2846291Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:09:30.2846801Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:09:30.2852107Z +++ nproc --ignore=2 2024-06-26T07:09:30.2864297Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:09:41.5525311Z + container_name=068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T07:09:41.5526488Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:09:41.5528044Z + docker exec -i 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:09:41.6149037Z + echo DOCKER_CONTAINER_ID=068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T07:09:41.6152389Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:09:41.6154284Z + docker exec -t 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:09:42.0013513Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:09:42.3134795Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:09:42.3139045Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:09:42.3141493Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:09:42.3145396Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:09:42.3147429Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:09:42.3151314Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:09:42.3177803Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:09:42.3236333Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:09:42.3671518Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:09:42.3830880Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:09:43.7014299Z Installing collected packages: torch 2024-06-26T07:09:52.4464705Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:09:52.4466023Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:09:52.4466691Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:09:52.5323151Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:09:52.5328889Z + source .ci/pytorch/common.sh 2024-06-26T07:09:52.5331539Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:09:52.5337117Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:09:52.5338669Z +++ declare -f -t trap_add 2024-06-26T07:09:52.5344365Z ++ set -ex 2024-06-26T07:09:52.5344948Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:09:52.5345521Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:09:52.5347040Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:09:52.5348147Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:09:52.5356044Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:09:52.5356600Z + trap_add cleanup_workspace EXIT 2024-06-26T07:09:52.5357017Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:09:52.5357855Z + shift 2024-06-26T07:09:52.5358237Z + for trap_add_name in "$@" 2024-06-26T07:09:52.5362579Z +++ trap -p EXIT 2024-06-26T07:09:52.5364737Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:09:52.5365226Z +++ extract_trap_cmd 2024-06-26T07:09:52.5365578Z +++ printf '%s\n' '' 2024-06-26T07:09:52.5365957Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:09:52.5366510Z + trap -- ' 2024-06-26T07:09:52.5366906Z cleanup_workspace' EXIT 2024-06-26T07:09:52.5367368Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:09:52.9897592Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:09:52.9926296Z + echo 'Environment variables:' 2024-06-26T07:09:52.9926749Z Environment variables: 2024-06-26T07:09:52.9927085Z + env 2024-06-26T07:09:52.9932283Z INSTALLED_DB=yes 2024-06-26T07:09:52.9932840Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:09:52.9933325Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:09:52.9933799Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:09:52.9934548Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:09:52.9935373Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:09:52.9935979Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:09:52.9936713Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:52.9937500Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:52.9938198Z HOSTNAME=068cf6bbc92f 2024-06-26T07:09:52.9939367Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:52.9940517Z GITHUB_ACTION=__self 2024-06-26T07:09:52.9941101Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:09:52.9945875Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:09:52.9949669Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:09:52.9950168Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:09:52.9950519Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:09:52.9950886Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-26T07:09:52.9951307Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:09:52.9951927Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:09:52.9952381Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:09:52.9952813Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:09:52.9953220Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:09:52.9953654Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:52.9954139Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:09:52.9954826Z GITHUB_REF_TYPE=tag 2024-06-26T07:09:52.9955148Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:09:52.9955512Z NCCL_VERSION=2.17.1-1 2024-06-26T07:09:52.9955891Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:52.9956418Z XLA_CUDA= 2024-06-26T07:09:52.9957025Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:09:52.9958734Z *** 2024-06-26T07:09:52.9959038Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:09:52.9959445Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:09:52.9959815Z GITHUB_ACTIONS=true 2024-06-26T07:09:52.9960151Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:52.9960660Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:09:52.9961189Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:09:52.9961650Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:52.9962150Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:09:52.9962635Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:52.9963426Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:09:52.9964123Z UCC_HOME=/usr 2024-06-26T07:09:52.9964452Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:09:52.9964845Z VERBOSE_TEST_LOGS=False 2024-06-26T07:09:52.9965189Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:09:52.9965626Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:09:52.9966117Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:09:52.9966574Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:09:52.9966947Z SHARD_NUMBER=2 2024-06-26T07:09:52.9967247Z GITHUB_REF_PROTECTED=false 2024-06-26T07:09:52.9967597Z HOME=/var/lib/jenkins 2024-06-26T07:09:52.9967955Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:09:52.9968381Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:09:52.9968855Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:09:52.9969331Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:09:52.9969696Z CUDA_VERSION=12.1.1 2024-06-26T07:09:52.9970103Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:09:52.9970535Z NUM_TEST_SHARDS=2 2024-06-26T07:09:52.9970835Z UCX_HOME=/usr 2024-06-26T07:09:52.9971335Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:09:52.9972377Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:52.9973605Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:52.9974867Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:52.9975942Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:09:52.9976579Z GITHUB_EVENT_NAME=push 2024-06-26T07:09:52.9976906Z DASHBOARD_TAG= 2024-06-26T07:09:52.9977208Z GITHUB_RUN_ID=9673646808 2024-06-26T07:09:52.9977652Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:09:52.9978175Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:09:52.9979142Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:52.9980023Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:52.9980428Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:09:52.9980798Z PR_NUMBER= 2024-06-26T07:09:52.9981215Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:09:52.9981548Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:09:52.9981973Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:09:52.9982426Z TERM=xterm 2024-06-26T07:09:52.9982753Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:09:52.9983162Z INSTALLED_VISION=yes 2024-06-26T07:09:52.9983462Z BRANCH= 2024-06-26T07:09:52.9983741Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:09:52.9984134Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:09:52.9984553Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:09:52.9985274Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:09:52.9986093Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:09:52.9986577Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:09:52.9987038Z REENABLED_ISSUES= 2024-06-26T07:09:52.9989058Z SHLVL=1 2024-06-26T07:09:52.9989321Z MAX_JOBS=14 2024-06-26T07:09:52.9989634Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:09:52.9989975Z NVARCH=x86_64 2024-06-26T07:09:52.9990269Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:09:52.9990715Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:52.9991233Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:09:52.9991997Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:09:52.9992647Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:52.9993209Z GITHUB_JOB=test 2024-06-26T07:09:52.9993596Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:09:52.9994136Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:09:52.9994702Z NO_TEST_TIMEOUT=False 2024-06-26T07:09:52.9995025Z TD_DISTRIBUTED=False 2024-06-26T07:09:52.9995387Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:09:52.9995816Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:09:52.9996231Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:09:52.9996594Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:09:52.9996933Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:09:52.9997279Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:09:52.9998291Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:52.9999327Z GITHUB_BASE_REF= 2024-06-26T07:09:52.9999642Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:09:53.0000004Z CI=true 2024-06-26T07:09:53.0000310Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:53.0000711Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:09:53.0001082Z JOB_ID=26688912112 2024-06-26T07:09:53.0001390Z INSTALLED_PROTOBUF=yes 2024-06-26T07:09:53.0001721Z GITHUB_HEAD_REF= 2024-06-26T07:09:53.0002017Z GITHUB_ACTION_REF= 2024-06-26T07:09:53.0002438Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:09:53.0002901Z GITHUB_WORKFLOW=inductor 2024-06-26T07:09:53.0003259Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:09:53.0004129Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0004928Z NO_TD=False 2024-06-26T07:09:53.0005231Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:09:53.0005588Z _=/usr/bin/env 2024-06-26T07:09:53.0006044Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:09:53.0080597Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:09:53.0081780Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:09:53.0082749Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:09:53.0083613Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:09:53.0084215Z + BUILD_DIR=build 2024-06-26T07:09:53.0084541Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:09:53.0084933Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:09:53.0085269Z + SHARD_NUMBER=2 2024-06-26T07:09:53.0085572Z + NUM_TEST_SHARDS=2 2024-06-26T07:09:53.0085887Z + export VALGRIND=ON 2024-06-26T07:09:53.0086198Z + VALGRIND=ON 2024-06-26T07:09:53.0086662Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:09:53.0087156Z + [[ 0 == \1 ]] 2024-06-26T07:09:53.0087449Z + [[ False == \1 ]] 2024-06-26T07:09:53.0087902Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:09:53.0088423Z ++ realpath build/custom_test_artifacts 2024-06-26T07:09:53.0091978Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:09:53.0092749Z + [[ -n '' ]] 2024-06-26T07:09:53.0093094Z + echo 'Environment variables' 2024-06-26T07:09:53.0093639Z Environment variables 2024-06-26T07:09:53.0093959Z + env 2024-06-26T07:09:53.0097377Z INSTALLED_DB=yes 2024-06-26T07:09:53.0098008Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:09:53.0098562Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:09:53.0099265Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:09:53.0099873Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:09:53.0100585Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:09:53.0101361Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:09:53.0101989Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:53.0102733Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:09:53.0103465Z HOSTNAME=068cf6bbc92f 2024-06-26T07:09:53.0104303Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0105096Z GITHUB_ACTION=__self 2024-06-26T07:09:53.0105443Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:09:53.0109478Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:09:53.0113748Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:09:53.0114256Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:09:53.0114599Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:09:53.0114980Z TEST_CONFIG=dynamic_inductor_torchbench 2024-06-26T07:09:53.0115457Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:09:53.0115920Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:09:53.0116378Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:09:53.0116819Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:09:53.0117245Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:09:53.0117678Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:53.0118170Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:09:53.0118636Z GITHUB_REF_TYPE=tag 2024-06-26T07:09:53.0118966Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:09:53.0119336Z NCCL_VERSION=2.17.1-1 2024-06-26T07:09:53.0119715Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:53.0120159Z XLA_CUDA= 2024-06-26T07:09:53.0120637Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:09:53.0121022Z *** 2024-06-26T07:09:53.0121303Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:09:53.0121692Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:09:53.0122057Z GITHUB_ACTIONS=true 2024-06-26T07:09:53.0122388Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:09:53.0122879Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:09:53.0123422Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:09:53.0123891Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:53.0124399Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:09:53.0124891Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:53.0125682Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:09:53.0126389Z UCC_HOME=/usr 2024-06-26T07:09:53.0126725Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:09:53.0127128Z VERBOSE_TEST_LOGS=False 2024-06-26T07:09:53.0127472Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:09:53.0127926Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:09:53.0128427Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:09:53.0128901Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:09:53.0129285Z SHARD_NUMBER=2 2024-06-26T07:09:53.0129719Z GITHUB_REF_PROTECTED=false 2024-06-26T07:09:53.0130078Z HOME=/var/lib/jenkins 2024-06-26T07:09:53.0130449Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:09:53.0130887Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:09:53.0131441Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:09:53.0131932Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:09:53.0132306Z CUDA_VERSION=12.1.1 2024-06-26T07:09:53.0132726Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:09:53.0133171Z NUM_TEST_SHARDS=2 2024-06-26T07:09:53.0133475Z UCX_HOME=/usr 2024-06-26T07:09:53.0133995Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:09:53.0135093Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0136335Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:09:53.0137572Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0138662Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:09:53.0139314Z GITHUB_EVENT_NAME=push 2024-06-26T07:09:53.0139646Z DASHBOARD_TAG= 2024-06-26T07:09:53.0139950Z GITHUB_RUN_ID=9673646808 2024-06-26T07:09:53.0140408Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:09:53.0141058Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:09:53.0142032Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0142919Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:09:53.0143330Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:09:53.0143699Z PR_NUMBER= 2024-06-26T07:09:53.0143983Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:09:53.0144295Z VALGRIND=ON 2024-06-26T07:09:53.0144583Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:09:53.0157199Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:09:53.0157727Z TERM=xterm 2024-06-26T07:09:53.0158105Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:09:53.0158524Z INSTALLED_VISION=yes 2024-06-26T07:09:53.0158854Z BRANCH= 2024-06-26T07:09:53.0159164Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:09:53.0159581Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:09:53.0160017Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:09:53.0160773Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:09:53.0161524Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:09:53.0162009Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:09:53.0162471Z REENABLED_ISSUES= 2024-06-26T07:09:53.0162768Z SHLVL=1 2024-06-26T07:09:53.0163031Z MAX_JOBS=14 2024-06-26T07:09:53.0163353Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:09:53.0163706Z NVARCH=x86_64 2024-06-26T07:09:53.0164006Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:09:53.0164470Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:09:53.0164995Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:09:53.0165479Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:09:53.0166145Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:09:53.0166716Z GITHUB_JOB=test 2024-06-26T07:09:53.0167113Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:09:53.0167667Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:09:53.0168180Z NO_TEST_TIMEOUT=False 2024-06-26T07:09:53.0168505Z TD_DISTRIBUTED=False 2024-06-26T07:09:53.0168869Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:09:53.0169309Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:09:53.0169731Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:09:53.0170093Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:09:53.0170448Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:09:53.0170806Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:09:53.0171829Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:53.0173011Z GITHUB_BASE_REF= 2024-06-26T07:09:53.0173339Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:09:53.0173770Z CI=true 2024-06-26T07:09:53.0174095Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:09:53.0174501Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:09:53.0174875Z JOB_ID=26688912112 2024-06-26T07:09:53.0175190Z INSTALLED_PROTOBUF=yes 2024-06-26T07:09:53.0175521Z GITHUB_HEAD_REF= 2024-06-26T07:09:53.0175818Z GITHUB_ACTION_REF= 2024-06-26T07:09:53.0176248Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:09:53.0176724Z GITHUB_WORKFLOW=inductor 2024-06-26T07:09:53.0177077Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:09:53.0177963Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_12c720be-093c-4a39-8946-d2a620befeab 2024-06-26T07:09:53.0178767Z NO_TD=False 2024-06-26T07:09:53.0179076Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:09:53.0179439Z _=/usr/bin/env 2024-06-26T07:09:53.0179779Z + echo 'Testing pytorch' 2024-06-26T07:09:53.0180118Z Testing pytorch 2024-06-26T07:09:53.0180443Z + export LANG=C.UTF-8 2024-06-26T07:09:53.0180772Z + LANG=C.UTF-8 2024-06-26T07:09:53.0181165Z + PR_NUMBER= 2024-06-26T07:09:53.0181543Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-06-26T07:09:53.0182110Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:09:53.0182663Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-06-26T07:09:53.0183315Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:09:53.0184003Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:09:53.0184537Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:09:53.0185009Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:09:53.0185487Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-06-26T07:09:53.0186085Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:09:53.0186716Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:09:53.0187362Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:53.0187917Z + pip_install --user ninja==1.10.2 2024-06-26T07:09:53.0188450Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:09:53.3722370Z Collecting ninja==1.10.2 2024-06-26T07:09:53.3904563Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:09:53.4056269Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:09:54.7164571Z Installing collected packages: ninja 2024-06-26T07:09:54.7231476Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:09:54.7232906Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:09:54.7271886Z Successfully installed ninja-1.10.2 2024-06-26T07:09:54.8175218Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:54.8177394Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:54.8178790Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:54.8179291Z + install_tlparse 2024-06-26T07:09:54.8179655Z + pip_install --user tlparse==0.3.7 2024-06-26T07:09:54.8180188Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:09:55.1722246Z Collecting tlparse==0.3.7 2024-06-26T07:09:55.1900136Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:09:55.2006937Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:09:56.5281189Z Installing collected packages: tlparse 2024-06-26T07:09:56.5654679Z Successfully installed tlparse-0.3.7 2024-06-26T07:09:56.6551375Z ++ python -m site --user-base 2024-06-26T07:09:56.6705479Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:56.6707201Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:09:56.6707857Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:09:56.6708689Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:56.6709613Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:09:56.6710732Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:09:56.6711447Z + cd test 2024-06-26T07:09:56.6712347Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:09:58.1518996Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:09:58.1519820Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:09:58.1520628Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:58.1521121Z + pushd test 2024-06-26T07:09:58.1521435Z ~/workspace/test ~/workspace 2024-06-26T07:09:58.1521952Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:09:59.6204227Z + CUDA_VERSION=12.1 2024-06-26T07:09:59.6204781Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:09:59.6205135Z + ISCUDA124= 2024-06-26T07:09:59.6205423Z + popd 2024-06-26T07:09:59.6205700Z ~/workspace 2024-06-26T07:09:59.6208700Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:09:59.6209412Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-06-26T07:09:59.6210206Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-06-26T07:09:59.6211022Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T07:09:59.6211802Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-06-26T07:09:59.6212544Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-06-26T07:09:59.6213324Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:09:59.6214005Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-06-26T07:09:59.6214978Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:09:59.6215892Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:09:59.6216694Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:09:59.6240784Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:59.6241481Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:09:59.6242901Z + cd test 2024-06-26T07:09:59.6243390Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:10:00.9705974Z PyTorch built with: 2024-06-26T07:10:00.9706631Z - GCC 9.4 2024-06-26T07:10:00.9706993Z - C++ Version: 201703 2024-06-26T07:10:00.9707816Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:00.9708886Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:00.9709541Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:00.9710051Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:10:00.9710519Z - NNPACK is enabled 2024-06-26T07:10:00.9710900Z - CPU capability usage: AVX2 2024-06-26T07:10:00.9711304Z - CUDA Runtime 12.1 2024-06-26T07:10:00.9712134Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:10:00.9712732Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:10:00.9713166Z - Magma 2.6.1 2024-06-26T07:10:00.9720278Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:10:00.9726740Z 2024-06-26T07:10:01.2456407Z + cd test 2024-06-26T07:10:01.2457616Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:10:02.4546194Z ATen/Parallel: 2024-06-26T07:10:02.4546721Z at::get_num_threads() : 8 2024-06-26T07:10:02.4547234Z at::get_num_interop_threads() : 16 2024-06-26T07:10:02.4547740Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:10:02.4548212Z omp_get_max_threads() : 8 2024-06-26T07:10:02.4549171Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:10:02.4550059Z mkl_get_max_threads() : 8 2024-06-26T07:10:02.4550764Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:10:02.4551445Z std::thread::hardware_concurrency() : 16 2024-06-26T07:10:02.4552232Z Environment variables: 2024-06-26T07:10:02.4552622Z OMP_NUM_THREADS : [not set] 2024-06-26T07:10:02.4552983Z MKL_NUM_THREADS : [not set] 2024-06-26T07:10:02.4553354Z ATen parallel backend: OpenMP 2024-06-26T07:10:02.4553601Z 2024-06-26T07:10:02.6946577Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:10:02.6947306Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-06-26T07:10:02.6947811Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-06-26T07:10:02.6948397Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-06-26T07:10:02.6948999Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:10:02.6949714Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:10:02.6950282Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-06-26T07:10:02.6950901Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-06-26T07:10:02.6951587Z + [[ dynamic_inductor_torchbench == *inductor-halide* ]] 2024-06-26T07:10:02.6952510Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-06-26T07:10:02.6953181Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-06-26T07:10:02.6953736Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-06-26T07:10:02.6954253Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-06-26T07:10:02.6954835Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T07:10:02.6955361Z + install_torchaudio cuda 2024-06-26T07:10:02.6955696Z + local commit 2024-06-26T07:10:02.6956060Z ++ get_pinned_commit audio 2024-06-26T07:10:02.6956445Z ++ cat .github/ci_commit_pins/audio.txt 2024-06-26T07:10:02.6960105Z + commit=b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:02.6960746Z + [[ cuda == \c\u\d\a ]] 2024-06-26T07:10:02.6961209Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-06-26T07:10:02.6962114Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:02.6963509Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:03.0096814Z Collecting git+https://github.com/pytorch/audio.git@b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:03.0101428Z Cloning https://github.com/pytorch/audio.git (to revision b829e936f7cc61b48149f5f957a451a38bf2a178) to /tmp/pip-req-build-pfhj778j 2024-06-26T07:10:03.0115076Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-pfhj778j 2024-06-26T07:10:03.7038887Z Running command git rev-parse -q --verify 'sha^b829e936f7cc61b48149f5f957a451a38bf2a178' 2024-06-26T07:10:03.7054211Z Running command git fetch -q https://github.com/pytorch/audio.git b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:04.0854471Z Running command git checkout -q b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:04.2056705Z Resolved https://github.com/pytorch/audio.git to commit b829e936f7cc61b48149f5f957a451a38bf2a178 2024-06-26T07:10:04.2057716Z Running command git submodule update --init --recursive -q 2024-06-26T07:10:06.3960629Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:10:06.3990895Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b829e93) (2.5.0a0+gitb8c4c54) 2024-06-26T07:10:06.4055080Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.13.1) 2024-06-26T07:10:06.4060732Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (4.12.2) 2024-06-26T07:10:06.4063980Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (1.12.1) 2024-06-26T07:10:06.4066156Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2.8.8) 2024-06-26T07:10:06.4069151Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (3.1.4) 2024-06-26T07:10:06.4072341Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b829e93) (2024.2.0) 2024-06-26T07:10:06.4508698Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b829e93) (2.1.5) 2024-06-26T07:10:06.4665225Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchaudio==2.4.0a0+b829e93) (1.3.0) 2024-06-26T07:10:06.4739200Z Building wheels for collected packages: torchaudio 2024-06-26T07:12:30.9241305Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T07:12:30.9271491Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b829e93-cp310-cp310-linux_x86_64.whl size=2308816 sha256=d60ced588b4d0a3fb776cb72832435ed40f839f728bb53a39f5c47e3908a78ca 2024-06-26T07:12:30.9274537Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/30/b3/9c/882f183f00af2e582825654a92a04400924519922db53ff522 2024-06-26T07:12:30.9299279Z Successfully built torchaudio 2024-06-26T07:12:32.1513115Z Installing collected packages: torchaudio 2024-06-26T07:12:32.3436976Z Successfully installed torchaudio-2.4.0a0+b829e93 2024-06-26T07:12:32.4786085Z + install_torchtext 2024-06-26T07:12:32.4786503Z + local data_commit 2024-06-26T07:12:32.4786842Z + local text_commit 2024-06-26T07:12:32.4789801Z ++ get_pinned_commit data 2024-06-26T07:12:32.4790431Z ++ cat .github/ci_commit_pins/data.txt 2024-06-26T07:12:32.4802517Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:32.4805058Z ++ get_pinned_commit text 2024-06-26T07:12:32.4805429Z ++ cat .github/ci_commit_pins/text.txt 2024-06-26T07:12:32.4815546Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:12:32.4816645Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:32.4818057Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:32.7932648Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:32.7936720Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-404940fl 2024-06-26T07:12:32.7954773Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-404940fl 2024-06-26T07:12:33.2349457Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-06-26T07:12:33.2364417Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:33.4620370Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:33.5902401Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-06-26T07:12:33.5903475Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:45.1117975Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:45.1170107Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-06-26T07:13:45.1173593Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-06-26T07:13:45.1179646Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:45.1241889Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-06-26T07:13:45.1247716Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-06-26T07:13:45.1251215Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.12.1) 2024-06-26T07:13:45.1254487Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-06-26T07:13:45.1257884Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-06-26T07:13:45.1261337Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.2.0) 2024-06-26T07:13:45.1449875Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-06-26T07:13:45.1456563Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-06-26T07:13:45.1463499Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.6.2) 2024-06-26T07:13:45.1931093Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-06-26T07:13:45.2096825Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-06-26T07:13:45.2178956Z Building wheels for collected packages: torchdata 2024-06-26T07:13:46.9507216Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-06-26T07:13:46.9514239Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182956 sha256=c1d0c16bcc799f6203d81b17673815f23746cf9b6eb339ac33504659a19dbceb 2024-06-26T07:13:46.9516246Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-06-26T07:13:46.9541317Z Successfully built torchdata 2024-06-26T07:13:48.1707257Z Installing collected packages: torchdata 2024-06-26T07:13:48.2954586Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-06-26T07:13:50.7926314Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:50.7927791Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:51.1125283Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:51.1129852Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-2x3015u4 2024-06-26T07:13:51.1143737Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-2x3015u4 2024-06-26T07:13:51.8792232Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-06-26T07:13:51.8805891Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:52.3296099Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:52.4790039Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-06-26T07:13:52.4791034Z Running command git submodule update --init --recursive -q 2024-06-26T07:13:59.5315028Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:13:59.5366363Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.4) 2024-06-26T07:13:59.5369634Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-06-26T07:13:59.5372949Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+gitb8c4c54) 2024-06-26T07:13:59.5376160Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.21.2) 2024-06-26T07:13:59.5379606Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-06-26T07:13:59.5441235Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-06-26T07:13:59.5446719Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-06-26T07:13:59.5453774Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-06-26T07:13:59.5459150Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.6.2) 2024-06-26T07:13:59.5516493Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-06-26T07:13:59.5523610Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-06-26T07:13:59.5527009Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.12.1) 2024-06-26T07:13:59.5530317Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-06-26T07:13:59.5533636Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-06-26T07:13:59.5536942Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.2.0) 2024-06-26T07:13:59.6234708Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-06-26T07:13:59.6403494Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-06-26T07:13:59.6487670Z Building wheels for collected packages: torchtext 2024-06-26T07:14:35.8766760Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-06-26T07:14:35.8792607Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045302 sha256=8da39b265ffe1b789dd6869c22461dde95d9e3393c1d51422fa3ddcec23ab04a 2024-06-26T07:14:35.8795170Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-06-26T07:14:35.8818518Z Successfully built torchtext 2024-06-26T07:14:37.0812052Z Installing collected packages: torchtext 2024-06-26T07:14:37.1785811Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-06-26T07:14:37.3000357Z + install_torchvision 2024-06-26T07:14:37.3000713Z + local orig_preload 2024-06-26T07:14:37.3001053Z + local commit 2024-06-26T07:14:37.3003308Z ++ get_pinned_commit vision 2024-06-26T07:14:37.3003735Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:14:37.3015058Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:37.3015501Z + orig_preload= 2024-06-26T07:14:37.3015843Z + '[' -n '' ']' 2024-06-26T07:14:37.3016666Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:37.3018104Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:37.6151116Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:37.6155761Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-z_wcem_0 2024-06-26T07:14:37.6170008Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-z_wcem_0 2024-06-26T07:14:38.9509899Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:14:38.9524732Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:40.1607985Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:40.4035159Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:14:42.7135715Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:14:42.7183289Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:14:42.7185904Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:14:42.7191231Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:14:42.7394257Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:14:42.7399811Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:14:42.7402307Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:14:42.7405327Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:14:42.7407869Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:14:42.7410629Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:14:42.7835323Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:14:42.7989859Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:14:42.8063742Z Building wheels for collected packages: torchvision 2024-06-26T07:15:55.9157475Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:15:55.9183599Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112964 sha256=32309532f717de04f9189c9f9f294ceaf66606a3c5b15d75b44139c3efc02d00 2024-06-26T07:15:55.9186264Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:15:55.9220149Z Successfully built torchvision 2024-06-26T07:15:57.1329106Z Installing collected packages: torchvision 2024-06-26T07:15:57.5157306Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:15:57.6242768Z + '[' -n '' ']' 2024-06-26T07:15:57.6243127Z + id=1 2024-06-26T07:15:57.6245312Z + pip_install opencv-python==4.8.0.74 2024-06-26T07:15:57.6246119Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-06-26T07:15:58.0556279Z Collecting opencv-python==4.8.0.74 2024-06-26T07:15:58.0744071Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-06-26T07:15:58.0909465Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.21.2) 2024-06-26T07:15:58.0990924Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-06-26T07:15:59.8913921Z Installing collected packages: opencv-python 2024-06-26T07:16:00.7092149Z Successfully installed opencv-python-4.8.0.74 2024-06-26T07:16:00.8075715Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-06-26T07:16:00.8076761Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-06-26T07:16:00.8077465Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-06-26T07:16:00.8078265Z + checkout_install_torchbench 2024-06-26T07:16:00.8078622Z + local commit 2024-06-26T07:16:00.8078939Z ++ get_pinned_commit torchbench 2024-06-26T07:16:00.8079357Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-06-26T07:16:00.8088807Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:16:00.8089432Z + git clone https://github.com/pytorch/benchmark torchbench 2024-06-26T07:16:00.8099014Z Cloning into 'torchbench'... 2024-06-26T07:16:00.9482878Z remote: Enumerating objects: 30187, done. 2024-06-26T07:16:00.9484920Z remote: Counting objects: 0% (1/546) 2024-06-26T07:16:00.9485662Z remote: Counting objects: 1% (6/546) 2024-06-26T07:16:00.9486281Z remote: Counting objects: 2% (11/546) 2024-06-26T07:16:00.9486935Z remote: Counting objects: 3% (17/546) 2024-06-26T07:16:00.9487641Z remote: Counting objects: 4% (22/546) 2024-06-26T07:16:00.9488976Z remote: Counting objects: 5% (28/546) 2024-06-26T07:16:00.9489665Z remote: Counting objects: 6% (33/546) 2024-06-26T07:16:00.9490413Z remote: Counting objects: 7% (39/546) 2024-06-26T07:16:00.9491020Z remote: Counting objects: 8% (44/546) 2024-06-26T07:16:00.9491723Z remote: Counting objects: 9% (50/546) 2024-06-26T07:16:00.9492421Z remote: Counting objects: 10% (55/546) 2024-06-26T07:16:00.9492927Z remote: Counting objects: 11% (61/546) 2024-06-26T07:16:00.9493408Z remote: Counting objects: 12% (66/546) 2024-06-26T07:16:00.9493888Z remote: Counting objects: 13% (71/546) 2024-06-26T07:16:00.9497935Z remote: Counting objects: 14% (77/546) 2024-06-26T07:16:00.9499019Z remote: Counting objects: 15% (82/546) 2024-06-26T07:16:00.9499736Z remote: Counting objects: 16% (88/546) 2024-06-26T07:16:00.9500419Z remote: Counting objects: 17% (93/546) 2024-06-26T07:16:00.9501169Z remote: Counting objects: 18% (99/546) 2024-06-26T07:16:00.9501675Z remote: Counting objects: 19% (104/546) 2024-06-26T07:16:00.9502332Z remote: Counting objects: 20% (110/546) 2024-06-26T07:16:00.9502998Z remote: Counting objects: 21% (115/546) 2024-06-26T07:16:00.9503607Z remote: Counting objects: 22% (121/546) 2024-06-26T07:16:00.9504295Z remote: Counting objects: 23% (126/546) 2024-06-26T07:16:00.9504796Z remote: Counting objects: 24% (132/546) 2024-06-26T07:16:00.9505302Z remote: Counting objects: 25% (137/546) 2024-06-26T07:16:00.9505845Z remote: Counting objects: 26% (142/546) 2024-06-26T07:16:00.9506341Z remote: Counting objects: 27% (148/546) 2024-06-26T07:16:00.9506832Z remote: Counting objects: 28% (153/546) 2024-06-26T07:16:00.9507330Z remote: Counting objects: 29% (159/546) 2024-06-26T07:16:00.9507829Z remote: Counting objects: 30% (164/546) 2024-06-26T07:16:00.9508321Z remote: Counting objects: 31% (170/546) 2024-06-26T07:16:00.9508931Z remote: Counting objects: 32% (175/546) 2024-06-26T07:16:00.9509532Z remote: Counting objects: 33% (181/546) 2024-06-26T07:16:00.9510046Z remote: Counting objects: 34% (186/546) 2024-06-26T07:16:00.9510546Z remote: Counting objects: 35% (192/546) 2024-06-26T07:16:00.9511043Z remote: Counting objects: 36% (197/546) 2024-06-26T07:16:00.9511527Z remote: Counting objects: 37% (203/546) 2024-06-26T07:16:00.9512311Z remote: Counting objects: 38% (208/546) 2024-06-26T07:16:00.9512816Z remote: Counting objects: 39% (213/546) 2024-06-26T07:16:00.9513308Z remote: Counting objects: 40% (219/546) 2024-06-26T07:16:00.9513795Z remote: Counting objects: 41% (224/546) 2024-06-26T07:16:00.9514286Z remote: Counting objects: 42% (230/546) 2024-06-26T07:16:00.9514778Z remote: Counting objects: 43% (235/546) 2024-06-26T07:16:00.9515275Z remote: Counting objects: 44% (241/546) 2024-06-26T07:16:00.9515764Z remote: Counting objects: 45% (246/546) 2024-06-26T07:16:00.9516252Z remote: Counting objects: 46% (252/546) 2024-06-26T07:16:00.9516744Z remote: Counting objects: 47% (257/546) 2024-06-26T07:16:00.9517228Z remote: Counting objects: 48% (263/546) 2024-06-26T07:16:00.9517892Z remote: Counting objects: 49% (268/546) 2024-06-26T07:16:00.9518385Z remote: Counting objects: 50% (273/546) 2024-06-26T07:16:00.9518875Z remote: Counting objects: 51% (279/546) 2024-06-26T07:16:00.9519360Z remote: Counting objects: 52% (284/546) 2024-06-26T07:16:00.9519965Z remote: Counting objects: 53% (290/546) 2024-06-26T07:16:00.9520465Z remote: Counting objects: 54% (295/546) 2024-06-26T07:16:00.9520958Z remote: Counting objects: 55% (301/546) 2024-06-26T07:16:00.9521445Z remote: Counting objects: 56% (306/546) 2024-06-26T07:16:00.9521933Z remote: Counting objects: 57% (312/546) 2024-06-26T07:16:00.9522427Z remote: Counting objects: 58% (317/546) 2024-06-26T07:16:00.9522919Z remote: Counting objects: 59% (323/546) 2024-06-26T07:16:00.9523404Z remote: Counting objects: 60% (328/546) 2024-06-26T07:16:00.9523901Z remote: Counting objects: 61% (334/546) 2024-06-26T07:16:00.9524393Z remote: Counting objects: 62% (339/546) 2024-06-26T07:16:00.9524895Z remote: Counting objects: 63% (344/546) 2024-06-26T07:16:00.9525380Z remote: Counting objects: 64% (350/546) 2024-06-26T07:16:00.9525869Z remote: Counting objects: 65% (355/546) 2024-06-26T07:16:00.9526360Z remote: Counting objects: 66% (361/546) 2024-06-26T07:16:00.9526849Z remote: Counting objects: 67% (366/546) 2024-06-26T07:16:00.9527340Z remote: Counting objects: 68% (372/546) 2024-06-26T07:16:00.9527831Z remote: Counting objects: 69% (377/546) 2024-06-26T07:16:00.9528321Z remote: Counting objects: 70% (383/546) 2024-06-26T07:16:00.9528806Z remote: Counting objects: 71% (388/546) 2024-06-26T07:16:00.9529301Z remote: Counting objects: 72% (394/546) 2024-06-26T07:16:00.9529793Z remote: Counting objects: 73% (399/546) 2024-06-26T07:16:00.9530284Z remote: Counting objects: 74% (405/546) 2024-06-26T07:16:00.9530768Z remote: Counting objects: 75% (410/546) 2024-06-26T07:16:00.9531255Z remote: Counting objects: 76% (415/546) 2024-06-26T07:16:00.9531751Z remote: Counting objects: 77% (421/546) 2024-06-26T07:16:00.9532239Z remote: Counting objects: 78% (426/546) 2024-06-26T07:16:00.9532723Z remote: Counting objects: 79% (432/546) 2024-06-26T07:16:00.9533213Z remote: Counting objects: 80% (437/546) 2024-06-26T07:16:00.9533708Z remote: Counting objects: 81% (443/546) 2024-06-26T07:16:00.9534199Z remote: Counting objects: 82% (448/546) 2024-06-26T07:16:00.9534684Z remote: Counting objects: 83% (454/546) 2024-06-26T07:16:00.9535177Z remote: Counting objects: 84% (459/546) 2024-06-26T07:16:00.9535673Z remote: Counting objects: 85% (465/546) 2024-06-26T07:16:00.9536157Z remote: Counting objects: 86% (470/546) 2024-06-26T07:16:00.9536648Z remote: Counting objects: 87% (476/546) 2024-06-26T07:16:00.9537141Z remote: Counting objects: 88% (481/546) 2024-06-26T07:16:00.9537640Z remote: Counting objects: 89% (486/546) 2024-06-26T07:16:00.9538126Z remote: Counting objects: 90% (492/546) 2024-06-26T07:16:00.9538626Z remote: Counting objects: 91% (497/546) 2024-06-26T07:16:00.9539118Z remote: Counting objects: 92% (503/546) 2024-06-26T07:16:00.9539610Z remote: Counting objects: 93% (508/546) 2024-06-26T07:16:00.9540094Z remote: Counting objects: 94% (514/546) 2024-06-26T07:16:00.9540588Z remote: Counting objects: 95% (519/546) 2024-06-26T07:16:00.9541234Z remote: Counting objects: 96% (525/546) 2024-06-26T07:16:00.9541792Z remote: Counting objects: 97% (530/546) 2024-06-26T07:16:00.9542467Z remote: Counting objects: 98% (536/546) 2024-06-26T07:16:00.9543026Z remote: Counting objects: 99% (541/546) 2024-06-26T07:16:00.9543712Z remote: Counting objects: 100% (546/546) 2024-06-26T07:16:00.9544285Z remote: Counting objects: 100% (546/546), done. 2024-06-26T07:16:00.9544838Z remote: Compressing objects: 0% (1/319) 2024-06-26T07:16:00.9545394Z remote: Compressing objects: 1% (4/319) 2024-06-26T07:16:00.9545945Z remote: Compressing objects: 2% (7/319) 2024-06-26T07:16:00.9546560Z remote: Compressing objects: 3% (10/319) 2024-06-26T07:16:00.9547091Z remote: Compressing objects: 4% (13/319) 2024-06-26T07:16:00.9547629Z remote: Compressing objects: 5% (16/319) 2024-06-26T07:16:00.9548197Z remote: Compressing objects: 6% (20/319) 2024-06-26T07:16:00.9548721Z remote: Compressing objects: 7% (23/319) 2024-06-26T07:16:00.9549291Z remote: Compressing objects: 8% (26/319) 2024-06-26T07:16:00.9549907Z remote: Compressing objects: 9% (29/319) 2024-06-26T07:16:00.9550896Z remote: Compressing objects: 10% (32/319) 2024-06-26T07:16:00.9553217Z remote: Compressing objects: 11% (36/319) 2024-06-26T07:16:00.9555430Z remote: Compressing objects: 12% (39/319) 2024-06-26T07:16:00.9557666Z remote: Compressing objects: 13% (42/319) 2024-06-26T07:16:00.9559527Z remote: Compressing objects: 14% (45/319) 2024-06-26T07:16:00.9560218Z remote: Compressing objects: 15% (48/319) 2024-06-26T07:16:00.9561021Z remote: Compressing objects: 16% (52/319) 2024-06-26T07:16:00.9561814Z remote: Compressing objects: 17% (55/319) 2024-06-26T07:16:00.9562520Z remote: Compressing objects: 18% (58/319) 2024-06-26T07:16:00.9563041Z remote: Compressing objects: 19% (61/319) 2024-06-26T07:16:00.9563571Z remote: Compressing objects: 20% (64/319) 2024-06-26T07:16:00.9564325Z remote: Compressing objects: 21% (67/319) 2024-06-26T07:16:00.9564951Z remote: Compressing objects: 22% (71/319) 2024-06-26T07:16:00.9565478Z remote: Compressing objects: 23% (74/319) 2024-06-26T07:16:00.9566118Z remote: Compressing objects: 24% (77/319) 2024-06-26T07:16:00.9566814Z remote: Compressing objects: 25% (80/319) 2024-06-26T07:16:00.9567449Z remote: Compressing objects: 26% (83/319) 2024-06-26T07:16:00.9568118Z remote: Compressing objects: 27% (87/319) 2024-06-26T07:16:00.9568650Z remote: Compressing objects: 28% (90/319) 2024-06-26T07:16:00.9569165Z remote: Compressing objects: 29% (93/319) 2024-06-26T07:16:00.9569787Z remote: Compressing objects: 30% (96/319) 2024-06-26T07:16:00.9570516Z remote: Compressing objects: 31% (99/319) 2024-06-26T07:16:00.9571117Z remote: Compressing objects: 32% (103/319) 2024-06-26T07:16:00.9571664Z remote: Compressing objects: 33% (106/319) 2024-06-26T07:16:00.9572406Z remote: Compressing objects: 34% (109/319) 2024-06-26T07:16:00.9573143Z remote: Compressing objects: 35% (112/319) 2024-06-26T07:16:00.9573842Z remote: Compressing objects: 36% (115/319) 2024-06-26T07:16:00.9574563Z remote: Compressing objects: 37% (119/319) 2024-06-26T07:16:00.9575206Z remote: Compressing objects: 38% (122/319) 2024-06-26T07:16:00.9575916Z remote: Compressing objects: 39% (125/319) 2024-06-26T07:16:00.9576472Z remote: Compressing objects: 40% (128/319) 2024-06-26T07:16:00.9577035Z remote: Compressing objects: 41% (131/319) 2024-06-26T07:16:00.9577565Z remote: Compressing objects: 42% (134/319) 2024-06-26T07:16:00.9578087Z remote: Compressing objects: 43% (138/319) 2024-06-26T07:16:00.9578619Z remote: Compressing objects: 44% (141/319) 2024-06-26T07:16:00.9579136Z remote: Compressing objects: 45% (144/319) 2024-06-26T07:16:00.9579660Z remote: Compressing objects: 46% (147/319) 2024-06-26T07:16:00.9580190Z remote: Compressing objects: 47% (150/319) 2024-06-26T07:16:00.9580753Z remote: Compressing objects: 48% (154/319) 2024-06-26T07:16:00.9581382Z remote: Compressing objects: 49% (157/319) 2024-06-26T07:16:00.9581921Z remote: Compressing objects: 50% (160/319) 2024-06-26T07:16:00.9582439Z remote: Compressing objects: 51% (163/319) 2024-06-26T07:16:00.9582960Z remote: Compressing objects: 52% (166/319) 2024-06-26T07:16:00.9583474Z remote: Compressing objects: 53% (170/319) 2024-06-26T07:16:00.9583995Z remote: Compressing objects: 54% (173/319) 2024-06-26T07:16:00.9584512Z remote: Compressing objects: 55% (176/319) 2024-06-26T07:16:00.9585030Z remote: Compressing objects: 56% (179/319) 2024-06-26T07:16:00.9585541Z remote: Compressing objects: 57% (182/319) 2024-06-26T07:16:00.9586205Z remote: Compressing objects: 58% (186/319) 2024-06-26T07:16:00.9586723Z remote: Compressing objects: 59% (189/319) 2024-06-26T07:16:00.9587241Z remote: Compressing objects: 60% (192/319) 2024-06-26T07:16:00.9587822Z remote: Compressing objects: 61% (195/319) 2024-06-26T07:16:00.9588348Z remote: Compressing objects: 62% (198/319) 2024-06-26T07:16:00.9588866Z remote: Compressing objects: 63% (201/319) 2024-06-26T07:16:00.9589382Z remote: Compressing objects: 64% (205/319) 2024-06-26T07:16:00.9589965Z remote: Compressing objects: 65% (208/319) 2024-06-26T07:16:00.9590490Z remote: Compressing objects: 66% (211/319) 2024-06-26T07:16:00.9591006Z remote: Compressing objects: 67% (214/319) 2024-06-26T07:16:00.9591515Z remote: Compressing objects: 68% (217/319) 2024-06-26T07:16:00.9592272Z remote: Compressing objects: 69% (221/319) 2024-06-26T07:16:00.9592952Z remote: Compressing objects: 70% (224/319) 2024-06-26T07:16:00.9593624Z remote: Compressing objects: 71% (227/319) 2024-06-26T07:16:00.9594144Z remote: Compressing objects: 72% (230/319) 2024-06-26T07:16:00.9594714Z remote: Compressing objects: 73% (233/319) 2024-06-26T07:16:00.9595319Z remote: Compressing objects: 74% (237/319) 2024-06-26T07:16:00.9595838Z remote: Compressing objects: 75% (240/319) 2024-06-26T07:16:00.9596350Z remote: Compressing objects: 76% (243/319) 2024-06-26T07:16:00.9596866Z remote: Compressing objects: 77% (246/319) 2024-06-26T07:16:00.9597484Z remote: Compressing objects: 78% (249/319) 2024-06-26T07:16:00.9598143Z remote: Compressing objects: 79% (253/319) 2024-06-26T07:16:00.9598691Z remote: Compressing objects: 80% (256/319) 2024-06-26T07:16:00.9599248Z remote: Compressing objects: 81% (259/319) 2024-06-26T07:16:00.9599767Z remote: Compressing objects: 82% (262/319) 2024-06-26T07:16:00.9600325Z remote: Compressing objects: 83% (265/319) 2024-06-26T07:16:00.9600876Z remote: Compressing objects: 84% (268/319) 2024-06-26T07:16:00.9601466Z remote: Compressing objects: 85% (272/319) 2024-06-26T07:16:00.9601989Z remote: Compressing objects: 86% (275/319) 2024-06-26T07:16:00.9602544Z remote: Compressing objects: 87% (278/319) 2024-06-26T07:16:00.9603098Z remote: Compressing objects: 88% (281/319) 2024-06-26T07:16:00.9603653Z remote: Compressing objects: 89% (284/319) 2024-06-26T07:16:00.9604207Z remote: Compressing objects: 90% (288/319) 2024-06-26T07:16:00.9604733Z remote: Compressing objects: 91% (291/319) 2024-06-26T07:16:00.9605290Z remote: Compressing objects: 92% (294/319) 2024-06-26T07:16:00.9605856Z remote: Compressing objects: 93% (297/319) 2024-06-26T07:16:00.9606372Z remote: Compressing objects: 94% (300/319) 2024-06-26T07:16:00.9606890Z remote: Compressing objects: 95% (304/319) 2024-06-26T07:16:00.9607411Z remote: Compressing objects: 96% (307/319) 2024-06-26T07:16:00.9607928Z remote: Compressing objects: 97% (310/319) 2024-06-26T07:16:00.9608451Z remote: Compressing objects: 98% (313/319) 2024-06-26T07:16:00.9608962Z remote: Compressing objects: 99% (316/319) 2024-06-26T07:16:00.9609481Z remote: Compressing objects: 100% (319/319) 2024-06-26T07:16:00.9610042Z remote: Compressing objects: 100% (319/319), done. 2024-06-26T07:16:00.9659707Z Receiving objects: 0% (1/30187) 2024-06-26T07:16:00.9696226Z Receiving objects: 1% (302/30187) 2024-06-26T07:16:00.9735551Z Receiving objects: 2% (604/30187) 2024-06-26T07:16:00.9769738Z Receiving objects: 3% (906/30187) 2024-06-26T07:16:00.9803164Z Receiving objects: 4% (1208/30187) 2024-06-26T07:16:00.9837200Z Receiving objects: 5% (1510/30187) 2024-06-26T07:16:00.9868230Z Receiving objects: 6% (1812/30187) 2024-06-26T07:16:00.9906479Z Receiving objects: 7% (2114/30187) 2024-06-26T07:16:00.9937997Z Receiving objects: 8% (2415/30187) 2024-06-26T07:16:00.9968415Z Receiving objects: 9% (2717/30187) 2024-06-26T07:16:01.0004144Z Receiving objects: 10% (3019/30187) 2024-06-26T07:16:01.0064718Z Receiving objects: 11% (3321/30187) 2024-06-26T07:16:01.0128488Z Receiving objects: 12% (3623/30187) 2024-06-26T07:16:01.0189723Z Receiving objects: 13% (3925/30187) 2024-06-26T07:16:01.0359231Z Receiving objects: 14% (4227/30187) 2024-06-26T07:16:01.0460180Z Receiving objects: 15% (4529/30187) 2024-06-26T07:16:01.0552495Z Receiving objects: 16% (4830/30187) 2024-06-26T07:16:01.0608140Z Receiving objects: 17% (5132/30187) 2024-06-26T07:16:01.0649059Z Receiving objects: 18% (5434/30187) 2024-06-26T07:16:01.0673789Z Receiving objects: 19% (5736/30187) 2024-06-26T07:16:01.0736300Z Receiving objects: 20% (6038/30187) 2024-06-26T07:16:01.0755541Z Receiving objects: 21% (6340/30187) 2024-06-26T07:16:01.0774557Z Receiving objects: 22% (6642/30187) 2024-06-26T07:16:01.1880257Z Receiving objects: 23% (6944/30187) 2024-06-26T07:16:01.9662813Z Receiving objects: 24% (7245/30187) 2024-06-26T07:16:02.0614298Z Receiving objects: 24% (7432/30187), 80.57 MiB | 80.56 MiB/s 2024-06-26T07:16:02.5973499Z Receiving objects: 25% (7547/30187), 80.57 MiB | 80.56 MiB/s 2024-06-26T07:16:02.6635158Z Receiving objects: 26% (7849/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:02.7297701Z Receiving objects: 27% (8151/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:02.7951507Z Receiving objects: 28% (8453/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:02.8627347Z Receiving objects: 29% (8755/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:02.9380125Z Receiving objects: 30% (9057/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:02.9615607Z Receiving objects: 31% (9358/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:03.0140377Z Receiving objects: 31% (9451/30187), 116.38 MiB | 77.58 MiB/s 2024-06-26T07:16:03.0903917Z Receiving objects: 32% (9660/30187), 155.66 MiB | 77.83 MiB/s 2024-06-26T07:16:03.3355853Z Receiving objects: 33% (9962/30187), 155.66 MiB | 77.83 MiB/s 2024-06-26T07:16:03.4130633Z Receiving objects: 34% (10264/30187), 155.66 MiB | 77.83 MiB/s 2024-06-26T07:16:03.4261105Z Receiving objects: 35% (10566/30187), 155.66 MiB | 77.83 MiB/s 2024-06-26T07:16:03.6557853Z Receiving objects: 36% (10868/30187), 155.66 MiB | 77.83 MiB/s 2024-06-26T07:16:03.9515848Z Receiving objects: 37% (11170/30187), 185.28 MiB | 74.11 MiB/s 2024-06-26T07:16:03.9616684Z Receiving objects: 38% (11472/30187), 185.28 MiB | 74.11 MiB/s 2024-06-26T07:16:03.9742566Z Receiving objects: 38% (11502/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9757505Z Receiving objects: 39% (11773/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9784179Z Receiving objects: 40% (12075/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9798643Z Receiving objects: 41% (12377/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9812841Z Receiving objects: 42% (12679/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9824294Z Receiving objects: 43% (12981/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9834909Z Receiving objects: 44% (13283/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9855897Z Receiving objects: 45% (13585/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9868346Z Receiving objects: 46% (13887/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9873011Z Receiving objects: 47% (14188/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9879506Z Receiving objects: 48% (14490/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9902154Z Receiving objects: 49% (14792/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9919549Z Receiving objects: 50% (15094/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:03.9930968Z Receiving objects: 51% (15396/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0138559Z Receiving objects: 52% (15698/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0143462Z Receiving objects: 53% (16000/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0151187Z Receiving objects: 54% (16301/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0157906Z Receiving objects: 55% (16603/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0164012Z Receiving objects: 56% (16905/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0181552Z Receiving objects: 57% (17207/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0192696Z Receiving objects: 58% (17509/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0202101Z Receiving objects: 59% (17811/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0215941Z Receiving objects: 60% (18113/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0225483Z Receiving objects: 61% (18415/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0236048Z Receiving objects: 62% (18716/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0388726Z Receiving objects: 63% (19018/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0396055Z Receiving objects: 64% (19320/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0402633Z Receiving objects: 65% (19622/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0409695Z Receiving objects: 66% (19924/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0419404Z Receiving objects: 67% (20226/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0428423Z Receiving objects: 68% (20528/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.0695713Z Receiving objects: 69% (20830/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1112038Z Receiving objects: 70% (21131/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1118558Z Receiving objects: 71% (21433/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1130778Z Receiving objects: 72% (21735/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1144766Z Receiving objects: 73% (22037/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1158694Z Receiving objects: 74% (22339/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1169629Z Receiving objects: 75% (22641/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1179006Z Receiving objects: 76% (22943/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1198586Z Receiving objects: 77% (23244/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1205553Z Receiving objects: 78% (23546/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1214408Z Receiving objects: 79% (23848/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.1238426Z Receiving objects: 80% (24150/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3343417Z Receiving objects: 81% (24452/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3565120Z Receiving objects: 82% (24754/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3599136Z Receiving objects: 83% (25056/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3610039Z Receiving objects: 84% (25358/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3621786Z Receiving objects: 85% (25659/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3653320Z Receiving objects: 86% (25961/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3684026Z Receiving objects: 87% (26263/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3837983Z Receiving objects: 88% (26565/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3845857Z Receiving objects: 89% (26867/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3851720Z Receiving objects: 90% (27169/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3857971Z Receiving objects: 91% (27471/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3865383Z Receiving objects: 92% (27773/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3872244Z Receiving objects: 93% (28074/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.3989148Z Receiving objects: 94% (28376/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4005579Z Receiving objects: 95% (28678/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4016228Z Receiving objects: 96% (28980/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4036087Z Receiving objects: 97% (29282/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4052048Z Receiving objects: 98% (29584/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4094541Z Receiving objects: 99% (29886/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4096491Z remote: Total 30187 (delta 255), reused 486 (delta 215), pack-reused 29641 2024-06-26T07:16:04.4112081Z Receiving objects: 100% (30187/30187), 221.25 MiB | 73.75 MiB/s 2024-06-26T07:16:04.4112893Z Receiving objects: 100% (30187/30187), 245.25 MiB | 71.10 MiB/s, done. 2024-06-26T07:16:04.4139164Z Resolving deltas: 0% (0/16212) 2024-06-26T07:16:04.4153794Z Resolving deltas: 1% (163/16212) 2024-06-26T07:16:04.4163276Z Resolving deltas: 2% (403/16212) 2024-06-26T07:16:04.4182222Z Resolving deltas: 3% (499/16212) 2024-06-26T07:16:04.4202752Z Resolving deltas: 4% (650/16212) 2024-06-26T07:16:04.4210876Z Resolving deltas: 5% (902/16212) 2024-06-26T07:16:04.4219735Z Resolving deltas: 6% (987/16212) 2024-06-26T07:16:04.4230727Z Resolving deltas: 7% (1135/16212) 2024-06-26T07:16:04.4239862Z Resolving deltas: 8% (1297/16212) 2024-06-26T07:16:04.4249231Z Resolving deltas: 9% (1461/16212) 2024-06-26T07:16:04.4260815Z Resolving deltas: 10% (1631/16212) 2024-06-26T07:16:04.4272246Z Resolving deltas: 11% (1791/16212) 2024-06-26T07:16:04.4282512Z Resolving deltas: 12% (1950/16212) 2024-06-26T07:16:04.4295540Z Resolving deltas: 13% (2110/16212) 2024-06-26T07:16:04.4304690Z Resolving deltas: 14% (2281/16212) 2024-06-26T07:16:04.4322799Z Resolving deltas: 15% (2434/16212) 2024-06-26T07:16:04.4340070Z Resolving deltas: 16% (2609/16212) 2024-06-26T07:16:04.4355363Z Resolving deltas: 17% (2758/16212) 2024-06-26T07:16:04.4397031Z Resolving deltas: 18% (2920/16212) 2024-06-26T07:16:04.4415278Z Resolving deltas: 19% (3082/16212) 2024-06-26T07:16:04.4432692Z Resolving deltas: 20% (3254/16212) 2024-06-26T07:16:04.4434769Z Resolving deltas: 21% (3532/16212) 2024-06-26T07:16:04.4443813Z Resolving deltas: 22% (3571/16212) 2024-06-26T07:16:04.4458909Z Resolving deltas: 23% (3737/16212) 2024-06-26T07:16:04.4462165Z Resolving deltas: 24% (4025/16212) 2024-06-26T07:16:04.4474712Z Resolving deltas: 25% (4085/16212) 2024-06-26T07:16:04.4511359Z Resolving deltas: 26% (4243/16212) 2024-06-26T07:16:04.4530015Z Resolving deltas: 27% (4399/16212) 2024-06-26T07:16:04.4543101Z Resolving deltas: 28% (4540/16212) 2024-06-26T07:16:04.4555156Z Resolving deltas: 29% (4709/16212) 2024-06-26T07:16:04.4571404Z Resolving deltas: 30% (4864/16212) 2024-06-26T07:16:04.4586635Z Resolving deltas: 31% (5027/16212) 2024-06-26T07:16:04.4595338Z Resolving deltas: 32% (5192/16212) 2024-06-26T07:16:04.4606093Z Resolving deltas: 33% (5351/16212) 2024-06-26T07:16:04.4616053Z Resolving deltas: 34% (5514/16212) 2024-06-26T07:16:04.4636976Z Resolving deltas: 35% (5678/16212) 2024-06-26T07:16:04.4658105Z Resolving deltas: 36% (5837/16212) 2024-06-26T07:16:04.4669826Z Resolving deltas: 37% (6044/16212) 2024-06-26T07:16:04.4685042Z Resolving deltas: 38% (6190/16212) 2024-06-26T07:16:04.4699464Z Resolving deltas: 39% (6331/16212) 2024-06-26T07:16:04.4710684Z Resolving deltas: 40% (6486/16212) 2024-06-26T07:16:04.4726570Z Resolving deltas: 41% (6659/16212) 2024-06-26T07:16:04.4738313Z Resolving deltas: 42% (6840/16212) 2024-06-26T07:16:04.4746530Z Resolving deltas: 43% (6980/16212) 2024-06-26T07:16:04.4756645Z Resolving deltas: 44% (7173/16212) 2024-06-26T07:16:04.4769485Z Resolving deltas: 45% (7315/16212) 2024-06-26T07:16:04.4793187Z Resolving deltas: 46% (7458/16212) 2024-06-26T07:16:04.4804492Z Resolving deltas: 47% (7621/16212) 2024-06-26T07:16:04.4827139Z Resolving deltas: 48% (7782/16212) 2024-06-26T07:16:04.4840087Z Resolving deltas: 49% (7944/16212) 2024-06-26T07:16:04.4852823Z Resolving deltas: 50% (8112/16212) 2024-06-26T07:16:04.4860180Z Resolving deltas: 51% (8280/16212) 2024-06-26T07:16:04.4872181Z Resolving deltas: 52% (8434/16212) 2024-06-26T07:16:04.4885719Z Resolving deltas: 53% (8618/16212) 2024-06-26T07:16:04.4890235Z Resolving deltas: 54% (8827/16212) 2024-06-26T07:16:04.4896379Z Resolving deltas: 55% (8918/16212) 2024-06-26T07:16:04.4908530Z Resolving deltas: 56% (9080/16212) 2024-06-26T07:16:04.4919062Z Resolving deltas: 57% (9245/16212) 2024-06-26T07:16:04.4925844Z Resolving deltas: 58% (9406/16212) 2024-06-26T07:16:04.4935911Z Resolving deltas: 59% (9591/16212) 2024-06-26T07:16:04.4941269Z Resolving deltas: 60% (9863/16212) 2024-06-26T07:16:04.4945339Z Resolving deltas: 61% (9966/16212) 2024-06-26T07:16:04.4956243Z Resolving deltas: 62% (10055/16212) 2024-06-26T07:16:04.4969869Z Resolving deltas: 63% (10263/16212) 2024-06-26T07:16:04.4972370Z Resolving deltas: 64% (10475/16212) 2024-06-26T07:16:04.4979487Z Resolving deltas: 65% (10539/16212) 2024-06-26T07:16:04.4988352Z Resolving deltas: 66% (10701/16212) 2024-06-26T07:16:04.5008340Z Resolving deltas: 67% (10885/16212) 2024-06-26T07:16:04.5028485Z Resolving deltas: 69% (11191/16212) 2024-06-26T07:16:04.5037378Z Resolving deltas: 70% (11350/16212) 2024-06-26T07:16:04.5049078Z Resolving deltas: 71% (11512/16212) 2024-06-26T07:16:04.5058141Z Resolving deltas: 72% (11700/16212) 2024-06-26T07:16:04.5070821Z Resolving deltas: 73% (11860/16212) 2024-06-26T07:16:04.5084075Z Resolving deltas: 74% (11997/16212) 2024-06-26T07:16:04.5116998Z Resolving deltas: 75% (12180/16212) 2024-06-26T07:16:04.5121358Z Resolving deltas: 78% (12719/16212) 2024-06-26T07:16:04.5133020Z Resolving deltas: 79% (12808/16212) 2024-06-26T07:16:04.5141013Z Resolving deltas: 80% (12987/16212) 2024-06-26T07:16:04.5151951Z Resolving deltas: 81% (13140/16212) 2024-06-26T07:16:04.5161200Z Resolving deltas: 82% (13305/16212) 2024-06-26T07:16:04.5169572Z Resolving deltas: 83% (13457/16212) 2024-06-26T07:16:04.5181051Z Resolving deltas: 84% (13619/16212) 2024-06-26T07:16:04.5201245Z Resolving deltas: 85% (13782/16212) 2024-06-26T07:16:04.5211866Z Resolving deltas: 86% (13944/16212) 2024-06-26T07:16:04.5219632Z Resolving deltas: 87% (14112/16212) 2024-06-26T07:16:04.5226406Z Resolving deltas: 88% (14282/16212) 2024-06-26T07:16:04.5235196Z Resolving deltas: 89% (14431/16212) 2024-06-26T07:16:04.5244114Z Resolving deltas: 90% (14616/16212) 2024-06-26T07:16:04.5251292Z Resolving deltas: 91% (14772/16212) 2024-06-26T07:16:04.5262979Z Resolving deltas: 92% (14923/16212) 2024-06-26T07:16:04.5266845Z Resolving deltas: 93% (15141/16212) 2024-06-26T07:16:04.5274702Z Resolving deltas: 94% (15256/16212) 2024-06-26T07:16:04.5282616Z Resolving deltas: 95% (15417/16212) 2024-06-26T07:16:04.5289530Z Resolving deltas: 96% (15569/16212) 2024-06-26T07:16:04.5295924Z Resolving deltas: 97% (15726/16212) 2024-06-26T07:16:04.5304327Z Resolving deltas: 98% (15891/16212) 2024-06-26T07:16:04.5339465Z Resolving deltas: 99% (16051/16212) 2024-06-26T07:16:04.5340026Z Resolving deltas: 100% (16212/16212) 2024-06-26T07:16:04.5340500Z Resolving deltas: 100% (16212/16212), done. 2024-06-26T07:16:05.5915900Z + pushd torchbench 2024-06-26T07:16:05.5916477Z ~/workspace/torchbench ~/workspace 2024-06-26T07:16:05.5917214Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-06-26T07:16:05.6488998Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-06-26T07:16:05.6489584Z 2024-06-26T07:16:05.6490229Z You are in 'detached HEAD' state. You can look around, make experimental 2024-06-26T07:16:05.6491255Z changes and commit them, and you can discard any commits you make in this 2024-06-26T07:16:05.6492106Z state without impacting any branches by switching back to a branch. 2024-06-26T07:16:05.6492574Z 2024-06-26T07:16:05.6492894Z If you want to create a new branch to retain commits you create, you may 2024-06-26T07:16:05.6493689Z do so (now or later) by using -c with the switch command. Example: 2024-06-26T07:16:05.6494106Z 2024-06-26T07:16:05.6494293Z git switch -c 2024-06-26T07:16:05.6494561Z 2024-06-26T07:16:05.6494696Z Or undo this operation with: 2024-06-26T07:16:05.6494951Z 2024-06-26T07:16:05.6495071Z git switch - 2024-06-26T07:16:05.6495257Z 2024-06-26T07:16:05.6495588Z Turn off this advice by setting config variable advice.detachedHead to false 2024-06-26T07:16:05.6496070Z 2024-06-26T07:16:05.6496244Z HEAD is now at 23512dbe Pin numpy version 2024-06-26T07:16:05.6496691Z + '[' '' ']' 2024-06-26T07:16:05.6497046Z + python install.py --continue_on_fail 2024-06-26T07:16:07.6187619Z checking packages torch, torchvision, torchaudio are installed...OK 2024-06-26T07:16:23.1188525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-06-26T07:16:25.4936744Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-06-26T07:16:28.0900617Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-06-26T07:16:30.4277445Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-06-26T07:16:30.4428193Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-06-26T07:16:39.2966289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-06-26T07:16:42.6894289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-06-26T07:16:46.0650380Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-06-26T07:16:49.4522012Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-06-26T07:16:49.4525663Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-06-26T07:16:51.2820418Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-06-26T07:16:54.8308448Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-06-26T07:16:54.8453022Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-06-26T07:17:49.3890493Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-06-26T07:17:58.3332827Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-06-26T07:18:05.9336550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-06-26T07:18:13.1593051Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-06-26T07:18:21.8368381Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-06-26T07:18:29.2785992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-06-26T07:18:36.2416474Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-06-26T07:18:43.6394839Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-06-26T07:18:51.3957602Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-06-26T07:18:58.9009784Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-06-26T07:19:06.2627188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-06-26T07:19:13.6815689Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-06-26T07:19:17.6610818Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-06-26T07:19:56.5170966Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-06-26T07:20:02.5591347Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-06-26T07:20:09.2717936Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-06-26T07:20:20.6456139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-06-26T07:20:22.8603469Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-06-26T07:20:25.3363320Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-06-26T07:20:33.2750392Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-06-26T07:20:41.7966363Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-06-26T07:20:48.5102049Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-06-26T07:20:57.1823363Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-06-26T07:21:04.2811228Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-06-26T07:21:10.5789044Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-06-26T07:21:17.9406279Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-06-26T07:21:34.2293434Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-06-26T07:21:41.5834899Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-06-26T07:21:46.9398074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-06-26T07:21:53.0047737Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-06-26T07:22:00.8441439Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-06-26T07:22:00.8447137Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-06-26T07:22:14.1904126Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-06-26T07:22:19.5145942Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-06-26T07:22:19.5149637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-06-26T07:22:28.1008351Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-06-26T07:22:30.3997454Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-06-26T07:22:32.7176513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-06-26T07:23:12.5759391Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-06-26T07:24:32.2568760Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-06-26T07:24:32.2727037Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-06-26T07:24:34.8438711Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-06-26T07:24:34.8589121Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-06-26T07:24:34.8743836Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-06-26T07:24:34.8896979Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-06-26T07:24:34.9047138Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-06-26T07:24:34.9200670Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-06-26T07:24:55.2576007Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-06-26T07:24:55.2578538Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-06-26T07:24:57.6271869Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-06-26T07:25:00.9877636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-06-26T07:25:01.0036048Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-06-26T07:25:01.0183800Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-06-26T07:25:01.0342564Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-06-26T07:25:01.0493009Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-06-26T07:25:01.0646281Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-06-26T07:25:08.3425684Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-06-26T07:25:10.9343523Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-06-26T07:25:15.5502682Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-06-26T07:25:15.5654181Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-06-26T07:25:15.5811801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-06-26T07:25:15.5965391Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-06-26T07:25:15.6116887Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-06-26T07:25:15.6271320Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-06-26T07:25:26.3315075Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-06-26T07:25:40.6129569Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-06-26T07:25:40.6281146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-06-26T07:25:40.6284481Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-06-26T07:25:40.6288085Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-06-26T07:25:44.3973691Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-06-26T07:25:47.3129222Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-06-26T07:25:47.3287495Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-06-26T07:26:05.4649060Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-06-26T07:26:12.4146450Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-06-26T07:26:16.7527534Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-06-26T07:26:24.6405934Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-06-26T07:26:24.6565139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-06-26T07:26:24.6724192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-06-26T07:26:24.6881159Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-06-26T07:26:24.7036697Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-06-26T07:26:24.7189340Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-06-26T07:26:24.7339902Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-06-26T07:26:24.7486398Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-06-26T07:26:36.3837341Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-06-26T07:26:49.3440407Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-06-26T07:26:49.3596448Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-06-26T07:26:52.3776661Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-06-26T07:26:55.6082065Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-06-26T07:26:56.0295302Z + popd 2024-06-26T07:26:56.0295641Z ~/workspace 2024-06-26T07:26:56.0296597Z + [[ dynamic_inductor_torchbench != *cpu_inductor* ]] 2024-06-26T07:26:56.0297104Z + install_torchrec_and_fbgemm 2024-06-26T07:26:56.0297525Z + local torchrec_commit 2024-06-26T07:26:56.0301587Z ++ get_pinned_commit torchrec 2024-06-26T07:26:56.0302209Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-06-26T07:26:56.0340442Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T07:26:56.0340974Z + local fbgemm_commit 2024-06-26T07:26:56.0343296Z ++ get_pinned_commit fbgemm 2024-06-26T07:26:56.0343725Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-06-26T07:26:56.0353500Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:26:56.0354162Z + pip_uninstall torchrec-nightly 2024-06-26T07:26:56.0354620Z + pip uninstall -y torchrec-nightly 2024-06-26T07:26:56.3924866Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-06-26T07:26:56.4335583Z + pip_uninstall fbgemm-gpu-nightly 2024-06-26T07:26:56.4336126Z + pip uninstall -y fbgemm-gpu-nightly 2024-06-26T07:26:56.7798841Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-06-26T07:26:56.8215266Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:26:56.8216259Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-06-26T07:26:57.2303602Z Collecting setuptools-git-versioning 2024-06-26T07:26:57.2484528Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-06-26T07:26:57.3249430Z Collecting scikit-build 2024-06-26T07:26:57.3289698Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-06-26T07:26:57.4276374Z Collecting pyre-extensions 2024-06-26T07:26:57.4319324Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-06-26T07:26:57.4730334Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-06-26T07:26:57.4733467Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (69.5.1) 2024-06-26T07:26:57.5213195Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-06-26T07:26:57.5242054Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-06-26T07:26:57.5843918Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-06-26T07:26:57.5855554Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-06-26T07:26:57.5860840Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-06-26T07:26:57.6371036Z Collecting typing-inspect (from pyre-extensions) 2024-06-26T07:26:57.6400698Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-06-26T07:26:57.6837150Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-06-26T07:26:57.7063755Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-06-26T07:26:57.7122130Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-06-26T07:26:57.7673235Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-06-26T07:26:57.8306443Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-06-26T07:26:57.8865860Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-06-26T07:26:57.9467461Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-06-26T07:27:00.1663504Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-06-26T07:27:00.5087534Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-06-26T07:27:00.6240269Z + CUDA_PATH=/usr/local/cuda-12.1 2024-06-26T07:27:00.6241538Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:27:00.6243394Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-06-26T07:27:00.9614900Z Collecting fbgemm-gpu 2024-06-26T07:27:00.9617467Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-w4z4qtme/fbgemm-gpu_3c5dcff0647b4acab1353ef917777cdd 2024-06-26T07:27:00.9631459Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-w4z4qtme/fbgemm-gpu_3c5dcff0647b4acab1353ef917777cdd 2024-06-26T07:27:01.9571859Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-06-26T07:27:01.9586815Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:02.3664668Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:02.9029351Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-06-26T07:27:02.9030466Z Running command git submodule update --init --recursive -q 2024-06-26T07:27:10.0922851Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T07:27:10.0944486Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-06-26T07:27:10.0952121Z Building wheels for collected packages: fbgemm-gpu 2024-06-26T08:09:13.3875332Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-06-26T08:09:13.5673868Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252629564 sha256=696356b2c8b76c9677e8df9613934ea81f64fd530500ab4808c14c6d54dd2402 2024-06-26T08:09:13.5675497Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-06-26T08:09:13.5694175Z Successfully built fbgemm-gpu 2024-06-26T08:09:15.7443786Z Installing collected packages: fbgemm-gpu 2024-06-26T08:09:18.8754288Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-06-26T08:09:19.4942472Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:19.4943976Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:19.8334365Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:19.8339211Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-3e4_qebq 2024-06-26T08:09:19.8357442Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-3e4_qebq 2024-06-26T08:09:20.4319694Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-06-26T08:09:20.4333389Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:20.7059678Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:21.0235810Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-06-26T08:09:21.2536065Z Preparing metadata (setup.py) ... [?25l- done 2024-06-26T08:09:21.2584645Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-06-26T08:09:21.2587332Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-06-26T08:09:21.2591486Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-06-26T08:09:21.3112334Z Collecting torchmetrics (from torchrec==0.3.2) 2024-06-26T08:09:21.3280273Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl.metadata (19 kB) 2024-06-26T08:09:21.3336542Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.4) 2024-06-26T08:09:21.3340841Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-06-26T08:09:21.3353874Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-06-26T08:09:21.3375027Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.0) 2024-06-26T08:09:21.4048975Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-06-26T08:09:21.4054020Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:09:21.4059634Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-06-26T08:09:21.4843855Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-06-26T08:09:21.4849705Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+gitb8c4c54) 2024-06-26T08:09:21.5031395Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-06-26T08:09:21.5062216Z Downloading lightning_utilities-0.11.2-py3-none-any.whl.metadata (4.7 kB) 2024-06-26T08:09:21.5224833Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (69.5.1) 2024-06-26T08:09:21.5227157Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-06-26T08:09:21.5265410Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-06-26T08:09:21.5335001Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-06-26T08:09:21.5338803Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.12.1) 2024-06-26T08:09:21.5342043Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-06-26T08:09:21.5345041Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-06-26T08:09:21.5348329Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.2.0) 2024-06-26T08:09:21.6016612Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-06-26T08:09:21.6326197Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-06-26T08:09:21.6470175Z Downloading torchmetrics-1.4.0.post0-py3-none-any.whl (868 kB) 2024-06-26T08:09:21.6659716Z Downloading lightning_utilities-0.11.2-py3-none-any.whl (26 kB) 2024-06-26T08:09:21.6754061Z Building wheels for collected packages: torchrec 2024-06-26T08:09:22.1177208Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-06-26T08:09:22.1185098Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=ba1c9d2f82ba2627f1d1589980a5525b59222b67ab8cd0a5066e912294482a72 2024-06-26T08:09:22.1186797Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-06-26T08:09:22.1208319Z Successfully built torchrec 2024-06-26T08:09:24.2438618Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-06-26T08:09:24.7993707Z Successfully installed lightning-utilities-0.11.2 torchmetrics-1.4.0.post0 torchrec-0.3.2 2024-06-26T08:09:24.9346906Z ++ pwd 2024-06-26T08:09:24.9349290Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-06-26T08:09:24.9349953Z + test_dynamo_benchmark torchbench 1 2024-06-26T08:09:24.9352006Z ++ pwd 2024-06-26T08:09:24.9354433Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:09:24.9355216Z + local suite=torchbench 2024-06-26T08:09:24.9355693Z + shift 2024-06-26T08:09:24.9356089Z + local shard_id=1 2024-06-26T08:09:24.9356572Z + shift 2024-06-26T08:09:24.9357086Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-26T08:09:24.9357777Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-26T08:09:24.9358284Z + [[ dynamic_inductor_torchbench == *cpu_inductor* ]] 2024-06-26T08:09:24.9359005Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:09:24.9359742Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-06-26T08:09:24.9360323Z ++ pwd 2024-06-26T08:09:24.9360908Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:09:24.9361567Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:09:24.9384243Z + local name=inference 2024-06-26T08:09:24.9384605Z + shift 2024-06-26T08:09:24.9384923Z + local suite=torchbench 2024-06-26T08:09:24.9385383Z + shift 2024-06-26T08:09:24.9385748Z + local shard_id=1 2024-06-26T08:09:24.9386165Z + shift 2024-06-26T08:09:24.9386533Z + partition_flags=() 2024-06-26T08:09:24.9386988Z + local partition_flags 2024-06-26T08:09:24.9387465Z + [[ -n 2 ]] 2024-06-26T08:09:24.9387773Z + [[ -n 1 ]] 2024-06-26T08:09:24.9388354Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:09:24.9389072Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-26T08:09:24.9389593Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-26T08:09:24.9390104Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:09:24.9392153Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-06-26T08:09:30.5828924Z 2024-06-26T08:09:31.0668006Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:09:31.0668816Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:09:31.0705805Z cuda eval llama 2024-06-26T08:10:21.9524398Z pass 2024-06-26T08:10:21.9670455Z TIMING: code_gen:4.65248 inductor_compile:19.148 backend_compile:43.08237 entire_frame_compile:48.9573 2024-06-26T08:10:21.9673192Z STATS: call_* op count: 1154 | FakeTensor.__torch_dispatch__:3218 | FakeTensorMode.__torch_dispatch__:46748 | attempt fast:1274 | fast is_contiguous:1274 | ProxyTorchDispatchMode.__torch_dispatch__:10235 2024-06-26T08:10:21.9674574Z Dynamo produced 1 graphs covering 1154 ops with 0 graph breaks (0 unique) 2024-06-26T08:10:27.9301177Z 2024-06-26T08:10:28.2018926Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:28.2019762Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:28.2020453Z cuda eval llama_v2_7b_16h 2024-06-26T08:10:28.2022978Z Traceback (most recent call last): 2024-06-26T08:10:28.2023650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:10:28.2024300Z ) = runner.load_model( 2024-06-26T08:10:28.2024959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:10:28.2025681Z benchmark = benchmark_cls( 2024-06-26T08:10:28.2026386Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:10:28.2027163Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:10:28.2028054Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-06-26T08:10:28.2029239Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:10:28.2030511Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:10:28.2032143Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:10:28.2033193Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:10:28.2034003Z self._skip_by_device_name() 2024-06-26T08:10:28.2034864Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:10:28.2036176Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:10:28.2037605Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-06-26T08:10:28.2038283Z 2024-06-26T08:10:28.2038526Z model_fail_to_load 2024-06-26T08:10:31.4057213Z 2024-06-26T08:10:31.6816520Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:31.6816993Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:31.6817431Z cuda eval llava 2024-06-26T08:10:31.6822229Z Traceback (most recent call last): 2024-06-26T08:10:31.6822973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:10:31.6823857Z ) = runner.load_model( 2024-06-26T08:10:31.6824514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:10:31.6825375Z benchmark = benchmark_cls( 2024-06-26T08:10:31.6826199Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:10:31.6826992Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:10:31.6828098Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-26T08:10:31.6829367Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:10:31.6830595Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:10:31.6831968Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:10:31.6832991Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:10:31.6833719Z self._skip_by_device_name() 2024-06-26T08:10:31.6834504Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:10:31.6835707Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:10:31.6836759Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-26T08:10:31.6837322Z 2024-06-26T08:10:31.6837445Z model_fail_to_load 2024-06-26T08:10:34.8473103Z 2024-06-26T08:10:35.4014508Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:35.4014998Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:35.4021699Z cuda eval maml 2024-06-26T08:10:35.4024816Z pass_due_to_skip 2024-06-26T08:10:35.4033509Z TIMING: 2024-06-26T08:10:35.4034144Z STATS: call_* op count: 0 2024-06-26T08:10:35.4034960Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T08:10:38.6314893Z 2024-06-26T08:10:39.0883743Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:39.0884215Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:39.0892153Z cuda eval maml_omniglot 2024-06-26T08:10:42.4837372Z pass 2024-06-26T08:10:42.4849104Z TIMING: code_gen:0.78276 inductor_compile:1.18259 backend_compile:2.22406 entire_frame_compile:2.39682 2024-06-26T08:10:42.4850549Z STATS: call_* op count: 28 | FakeTensor.__torch_dispatch__:271 | FakeTensorMode.__torch_dispatch__:2380 | ProxyTorchDispatchMode.__torch_dispatch__:555 2024-06-26T08:10:42.4851633Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-06-26T08:10:45.9174534Z 2024-06-26T08:10:46.1126100Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-06-26T08:10:46.1353653Z 2024-06-26T08:10:46.1353916Z 2024-06-26T08:10:46.1920334Z 0% 0.00/16.9M [00:00 will be ignored 2024-06-26T08:11:56.9208419Z [rank0]:W0626 08:11:56.920000 140320955597440 torch/_dynamo/backends/distributed.py:88] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:11:56.9401152Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:11:56.9406487Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:11:56.9408025Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:11:56.9409539Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:11:56.9411001Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] │ 0 │ 157 │ 44910720 │ 2024-06-26T08:11:56.9412479Z [rank0]:W0626 08:11:56.939000 140320955597440 torch/_dynamo/backends/distributed.py:105] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:12:15.1333456Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 2024-06-26T08:12:15.1334737Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 130, in forward 2024-06-26T08:12:15.1335828Z self._momentum_update_key_encoder() # update the key encoder 2024-06-26T08:12:15.1337066Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-06-26T08:12:15.1338045Z return func(*args, **kwargs) 2024-06-26T08:12:15.1339058Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 50, in _momentum_update_key_encoder 2024-06-26T08:12:15.1340269Z param_k.mul_(self.m).add_(param_q.mul(1. - self.m)) 2024-06-26T08:12:15.1340623Z 2024-06-26T08:12:16.0474058Z [rank0]:W0626 08:12:16.046000 140320955597440 torch/_inductor/utils.py:1234] [3/0_1] DeviceCopy in input program 2024-06-26T08:12:16.0577984Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-06-26T08:12:16.0579481Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-06-26T08:12:16.0580470Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-06-26T08:12:16.0580828Z 2024-06-26T08:12:16.9891974Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:12:16.9893377Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:12:16.9894500Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] or: 2024-06-26T08:12:16.9895589Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:12:16.9896924Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] to include these operations in the captured graph. 2024-06-26T08:12:16.9898026Z [rank0]:W0626 08:12:16.988000 140320955597440 torch/_dynamo/variables/tensor.py:714] [5/0] 2024-06-26T08:12:28.6566868Z pass 2024-06-26T08:12:28.6567777Z TIMING: entire_frame_compile:36.5618 code_gen:8.36266 inductor_compile:16.84237 backend_compile:27.18482 2024-06-26T08:12:28.6569044Z STATS: call_* op count: 877 | FakeTensor.__torch_dispatch__:10622 | FakeTensorMode.__torch_dispatch__:58061 | ProxyTorchDispatchMode.__torch_dispatch__:12396 2024-06-26T08:12:28.6570138Z Dynamo produced 7 graphs covering 877 ops with 5 graph breaks (3 unique) 2024-06-26T08:12:29.5980462Z [rank0]:[W626 08:12:29.642135201 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:12:33.8385238Z 2024-06-26T08:12:34.1137031Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:34.1137897Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:12:34.1138772Z cuda eval moondream 2024-06-26T08:12:34.1142026Z Traceback (most recent call last): 2024-06-26T08:12:34.1142956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:12:34.1143767Z ) = runner.load_model( 2024-06-26T08:12:34.1144543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 327, in load_model 2024-06-26T08:12:34.1145456Z benchmark = benchmark_cls( 2024-06-26T08:12:34.1146388Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:12:34.1147424Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:12:34.1148419Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-06-26T08:12:34.1149567Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:34.1150815Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:12:34.1152236Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:12:34.1153196Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:12:34.1153945Z self._skip_by_device_name() 2024-06-26T08:12:34.1154740Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:12:34.1155951Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:12:34.1157031Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-06-26T08:12:34.1157616Z 2024-06-26T08:12:34.1157737Z model_fail_to_load 2024-06-26T08:12:37.2867327Z 2024-06-26T08:12:39.1654884Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:12:39.3780708Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:12:39.3781705Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:12:39.3785636Z using fused AdamW: True 2024-06-26T08:12:39.6389036Z 2024-06-26T08:12:39.6390008Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:12:39.6430005Z cuda eval nanogpt 2024-06-26T08:13:15.3373216Z pass 2024-06-26T08:13:15.3384120Z TIMING: code_gen:2.45381 inductor_compile:10.39792 backend_compile:28.08485 entire_frame_compile:33.94605 2024-06-26T08:13:15.3387483Z STATS: call_* op count: 834 | FakeTensorMode.__torch_dispatch__:40113 | FakeTensor.__torch_dispatch__:2657 | attempt fast:2346 | fast is_contiguous:2346 | ProxyTorchDispatchMode.__torch_dispatch__:8284 2024-06-26T08:13:15.3388827Z Dynamo produced 1 graphs covering 834 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:20.3602247Z 2024-06-26T08:13:22.6723975Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:22.6724442Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:13:22.6729145Z cuda eval nvidia_deeprecommender 2024-06-26T08:13:26.7100252Z pass 2024-06-26T08:13:26.7110859Z TIMING: code_gen:0.3364 inductor_compile:0.60258 backend_compile:2.75665 entire_frame_compile:2.94952 2024-06-26T08:13:26.7112692Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1790 | ProxyTorchDispatchMode.__torch_dispatch__:579 | FakeTensor.__torch_dispatch__:170 2024-06-26T08:13:26.7113774Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:30.1613425Z 2024-06-26T08:13:30.9646878Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:30.9647391Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:30.9667100Z cuda eval opacus_cifar10 2024-06-26T08:13:41.4010824Z pass 2024-06-26T08:13:41.4012674Z TIMING: code_gen:2.23804 inductor_compile:3.73326 backend_compile:6.0973 entire_frame_compile:9.18852 2024-06-26T08:13:41.4013947Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:7448 | ProxyTorchDispatchMode.__torch_dispatch__:2467 | FakeTensor.__torch_dispatch__:866 2024-06-26T08:13:41.4015054Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-06-26T08:13:45.0785850Z 2024-06-26T08:13:45.6049363Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:45.6050255Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:45.6113549Z cuda eval phlippe_densenet 2024-06-26T08:14:00.5202463Z pass 2024-06-26T08:14:00.5213960Z TIMING: code_gen:3.9823 inductor_compile:7.09013 backend_compile:11.6802 entire_frame_compile:13.59236 2024-06-26T08:14:00.5215233Z STATS: call_* op count: 372 | FakeTensor.__torch_dispatch__:3838 | FakeTensorMode.__torch_dispatch__:28866 | ProxyTorchDispatchMode.__torch_dispatch__:7588 2024-06-26T08:14:00.5216338Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:04.3981595Z 2024-06-26T08:14:04.8764088Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:04.8764551Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:04.8791477Z cuda eval phlippe_resnet 2024-06-26T08:14:11.1321814Z pass 2024-06-26T08:14:11.1333569Z TIMING: code_gen:1.06557 inductor_compile:2.20717 backend_compile:4.48622 entire_frame_compile:5.14044 2024-06-26T08:14:11.1334966Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1447 | FakeTensorMode.__torch_dispatch__:10983 | ProxyTorchDispatchMode.__torch_dispatch__:2856 2024-06-26T08:14:11.1336143Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:14.6763600Z 2024-06-26T08:14:14.9635254Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:14.9635743Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:14.9636200Z cuda eval pyhpc_equation_of_state 2024-06-26T08:14:21.1146615Z pass 2024-06-26T08:14:21.1147727Z TIMING: code_gen:1.54164 inductor_compile:3.23862 backend_compile:4.75198 entire_frame_compile:5.15594 2024-06-26T08:14:21.1149226Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:10370 | ProxyTorchDispatchMode.__torch_dispatch__:3745 | FakeTensor.__torch_dispatch__:1061 2024-06-26T08:14:21.1150336Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:24.6104198Z 2024-06-26T08:14:24.9085629Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:24.9086104Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:24.9089602Z cuda eval pyhpc_isoneutral_mixing 2024-06-26T08:14:50.6924426Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 2024-06-26T08:14:50.6928169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2360, in run_n_iterations 2024-06-26T08:14:50.6929175Z return self.model_iter_fn(mod, inputs, collect_outputs=True) 2024-06-26T08:14:50.6930256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:14:50.6931012Z return mod(*inputs) 2024-06-26T08:14:50.6932226Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:14:50.6933283Z return forward_call(*args, **kwargs) 2024-06-26T08:14:50.6934316Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 96, in forward 2024-06-26T08:14:50.6935335Z return isoneutral_pytorch.isoneutral_diffusion_pre( 2024-06-26T08:14:50.6936454Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 165, in isoneutral_diffusion_pre 2024-06-26T08:14:50.6937848Z K_11[1:-2, 2:-2, :] = sumz / (4.0 * dzt[None, None, :]) 2024-06-26T08:14:50.6938201Z 2024-06-26T08:14:50.8153155Z pass 2024-06-26T08:14:50.8155297Z TIMING: code_gen:5.23254 inductor_compile:15.19595 backend_compile:23.46653 entire_frame_compile:25.08512 2024-06-26T08:14:50.8156971Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:52376 | FakeTensor.__torch_dispatch__:5004 | ProxyTorchDispatchMode.__torch_dispatch__:14005 2024-06-26T08:14:50.8158084Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-06-26T08:14:55.2969516Z 2024-06-26T08:14:56.0426141Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:14:56.1229241Z 2024-06-26T08:14:56.1229608Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:14:56.1230373Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:14:56.1231169Z cuda eval pyhpc_turbulent_kinetic_energy 2024-06-26T08:14:56.1691518Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-06-26T08:15:23.9723554Z pass 2024-06-26T08:15:23.9826873Z TIMING: code_gen:9.03716 inductor_compile:15.5276 backend_compile:24.76268 entire_frame_compile:26.54047 2024-06-26T08:15:23.9828175Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:47021 | FakeTensor.__torch_dispatch__:2561 | ProxyTorchDispatchMode.__torch_dispatch__:14635 2024-06-26T08:15:23.9829280Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:28.2775618Z 2024-06-26T08:15:28.9618754Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:28.9619639Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:28.9631636Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-06-26T08:15:44.7474689Z pass 2024-06-26T08:15:44.7496475Z TIMING: code_gen:2.79724 inductor_compile:4.87056 backend_compile:6.96533 entire_frame_compile:7.53505 2024-06-26T08:15:44.7497754Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:11393 | FakeTensor.__torch_dispatch__:1335 | ProxyTorchDispatchMode.__torch_dispatch__:3743 2024-06-26T08:15:44.7498864Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-06-26T08:15:48.4494442Z 2024-06-26T08:15:49.4617244Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:49.4617712Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:15:49.4640055Z cuda eval pytorch_stargan 2024-06-26T08:16:12.5923142Z pass 2024-06-26T08:16:12.5994625Z TIMING: code_gen:2.07701 inductor_compile:4.27361 backend_compile:6.84314 entire_frame_compile:7.46469 2024-06-26T08:16:12.5996620Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:16402 | FakeTensor.__torch_dispatch__:1566 | ProxyTorchDispatchMode.__torch_dispatch__:4659 2024-06-26T08:16:12.5997951Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:16.3135378Z 2024-06-26T08:16:16.9103430Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:16.9103923Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:16:16.9126778Z cuda eval pytorch_unet 2024-06-26T08:16:37.6746757Z pass 2024-06-26T08:16:37.6880185Z TIMING: code_gen:3.82402 inductor_compile:5.95173 backend_compile:8.05243 entire_frame_compile:8.90231 2024-06-26T08:16:37.6882720Z STATS: call_* op count: 142 | FakeTensor.__torch_dispatch__:1807 | FakeTensorMode.__torch_dispatch__:14774 | ProxyTorchDispatchMode.__torch_dispatch__:4409 2024-06-26T08:16:37.6883856Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-06-26T08:16:41.3891423Z 2024-06-26T08:16:42.1547653Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-06-26T08:16:42.1760324Z 2024-06-26T08:16:42.1760332Z 2024-06-26T08:16:42.2764128Z 0% 0.00/230M [00:00 2024-06-26T08:25:03.6123243Z torchbench_main() 2024-06-26T08:25:03.6123921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torchbench_main 2024-06-26T08:25:03.6124736Z main(TorchBenchmarkRunner(), original_dir) 2024-06-26T08:25:03.6125442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3722, in main 2024-06-26T08:25:03.6127067Z process_entry(0, runner, original_dir, args) 2024-06-26T08:25:03.6127862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3654, in process_entry 2024-06-26T08:25:03.6131575Z return run(runner, args, original_dir) 2024-06-26T08:25:03.6132347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4283, in run 2024-06-26T08:25:03.6136693Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-26T08:25:03.6137392Z AssertionError: nothing in example_inputs had a dim with 4 2024-06-26T08:25:06.5768117Z Run failed with return code: 1 2024-06-26T08:25:06.5768573Z Output: None 2024-06-26T08:25:06.5768870Z Error: None 2024-06-26T08:25:09.1103434Z 2024-06-26T08:25:09.2763889Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth 2024-06-26T08:25:09.2996663Z 2024-06-26T08:25:09.2997015Z 2024-06-26T08:25:09.3230669Z 0% 0.00/8.79M [00:00 2024-06-26T08:30:32.1134447Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-26T08:30:32.1135023Z 2024-06-26T08:30:33.2052928Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:30:33.2054508Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:30:33.2055577Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] or: 2024-06-26T08:30:33.2056621Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:30:33.2057871Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] to include these operations in the captured graph. 2024-06-26T08:30:33.2059144Z W0626 08:30:33.204000 140312881570432 torch/_dynamo/variables/tensor.py:714] [13/0] 2024-06-26T08:31:17.0050040Z pass 2024-06-26T08:31:17.0062787Z TIMING: entire_frame_compile:69.69329 code_gen:9.05923 inductor_compile:20.48188 backend_compile:36.25365 2024-06-26T08:31:17.0064601Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:78417 | FakeTensor.__torch_dispatch__:6024 | ProxyTorchDispatchMode.__torch_dispatch__:18376 | attempt fast:412 | fast is_contiguous:284 | slow both tensors nontrivially broadcast:128 2024-06-26T08:31:17.0066312Z Dynamo produced 25 graphs covering 1358 ops with 16 graph breaks (2 unique) 2024-06-26T08:31:21.8322169Z 2024-06-26T08:31:23.6633004Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:31:23.6633715Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:31:23.6724311Z cuda eval yolov3 2024-06-26T08:31:45.7555001Z W0626 08:31:45.754000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:46.0002027Z W0626 08:31:45.999000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:46.2398197Z W0626 08:31:46.239000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:46.5828448Z W0626 08:31:46.582000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:46.5838341Z W0626 08:31:46.583000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:46.5849445Z W0626 08:31:46.584000 140444468691584 torch/_inductor/utils.py:1234] [0/0_1] DeviceCopy in input program 2024-06-26T08:31:53.1986388Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg225_1). Found from : 2024-06-26T08:31:53.1987534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2359, in run_n_iterations 2024-06-26T08:31:53.1988620Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-06-26T08:31:53.1989507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 447, in forward_pass 2024-06-26T08:31:53.1990221Z return mod(*inputs) 2024-06-26T08:31:53.1991181Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:31:53.1992465Z return forward_call(*args, **kwargs) 2024-06-26T08:31:53.1993420Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-06-26T08:31:53.1994461Z return self.forward_once(x) 2024-06-26T08:31:53.1995581Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-06-26T08:31:53.1996455Z yolo_out.append(module(x, out)) 2024-06-26T08:31:53.1997285Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-26T08:31:53.1998133Z self.create_grids((nx, ny), p.device) 2024-06-26T08:31:53.1999012Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-26T08:31:53.1999922Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-26T08:31:53.2000245Z 2024-06-26T08:31:53.6788309Z pass 2024-06-26T08:31:53.6812497Z TIMING: code_gen:6.39322 inductor_compile:12.20863 backend_compile:19.15426 entire_frame_compile:22.5709 2024-06-26T08:31:53.6813767Z STATS: call_* op count: 649 | FakeTensor.__torch_dispatch__:5435 | FakeTensorMode.__torch_dispatch__:46959 | ProxyTorchDispatchMode.__torch_dispatch__:12537 2024-06-26T08:31:53.6814863Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique) 2024-06-26T08:31:55.4069225Z accuracy pass_rate=82.22% 2024-06-26T08:31:55.4073279Z calls_captured gmean=0.00x mean=528.956x 2024-06-26T08:31:55.4077213Z unique_graphs gmean=0.00x mean=1.667x 2024-06-26T08:31:55.4080968Z graph_breaks gmean=0.00x mean=0.733x 2024-06-26T08:31:55.4084739Z unique_graph_breaks gmean=0.00x mean=0.178x 2024-06-26T08:31:55.4088541Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:31:55.4092210Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:31:55.4095941Z cudagraph_skips gmean=0.00x mean=0.244x 2024-06-26T08:31:56.2424980Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_inference.csv 2024-06-26T08:31:56.5814544Z llama PASS 2024-06-26T08:31:56.5818073Z llama_v2_7b_16h XFAIL 2024-06-26T08:31:56.5822292Z llava XFAIL 2024-06-26T08:31:56.5826655Z maml XFAIL 2024-06-26T08:31:56.5831563Z maml_omniglot PASS 2024-06-26T08:31:56.5835802Z mnasnet1_0 PASS 2024-06-26T08:31:56.5840184Z mobilenet_v2 PASS 2024-06-26T08:31:56.5844843Z mobilenet_v2_quantized_qat XFAIL 2024-06-26T08:31:56.5849307Z mobilenet_v3_large PASS 2024-06-26T08:31:56.5853714Z moco PASS 2024-06-26T08:31:56.5858003Z moondream XFAIL 2024-06-26T08:31:56.5862826Z nanogpt PASS 2024-06-26T08:31:56.5867267Z nvidia_deeprecommender PASS 2024-06-26T08:31:56.5872099Z opacus_cifar10 PASS 2024-06-26T08:31:56.5876475Z phlippe_densenet PASS 2024-06-26T08:31:56.5880825Z phlippe_resnet PASS 2024-06-26T08:31:56.5885264Z pyhpc_equation_of_state PASS 2024-06-26T08:31:56.5889645Z pyhpc_isoneutral_mixing PASS 2024-06-26T08:31:56.5894139Z pyhpc_turbulent_kinetic_energy PASS 2024-06-26T08:31:56.5898434Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:31:56.5902999Z pytorch_stargan PASS 2024-06-26T08:31:56.5907568Z pytorch_unet PASS 2024-06-26T08:31:56.5912038Z resnet152 PASS 2024-06-26T08:31:56.5916531Z resnet18 PASS 2024-06-26T08:31:56.5920861Z resnet50 PASS 2024-06-26T08:31:56.5925304Z resnet50_quantized_qat XFAIL 2024-06-26T08:31:56.5929653Z resnext50_32x4d PASS 2024-06-26T08:31:56.5934107Z sam PASS 2024-06-26T08:31:56.5938405Z shufflenet_v2_x1_0 PASS 2024-06-26T08:31:56.5943216Z soft_actor_critic PASS 2024-06-26T08:31:56.5947543Z speech_transformer PASS 2024-06-26T08:31:56.5952014Z squeezenet1_1 PASS 2024-06-26T08:31:56.5956612Z stable_diffusion_text_encoder PASS 2024-06-26T08:31:56.5960905Z stable_diffusion_unet XFAIL 2024-06-26T08:31:56.5965309Z timm_efficientnet PASS 2024-06-26T08:31:56.5970201Z timm_regnet PASS 2024-06-26T08:31:56.5974068Z timm_resnest PASS 2024-06-26T08:31:56.5978370Z timm_vision_transformer PASS 2024-06-26T08:31:56.5983077Z timm_vision_transformer_large XFAIL 2024-06-26T08:31:56.5987470Z timm_vovnet PASS 2024-06-26T08:31:56.5991902Z torch_multimodal_clip PASS 2024-06-26T08:31:56.5996598Z tts_angular PASS 2024-06-26T08:31:56.6000971Z vgg16 PASS 2024-06-26T08:31:56.6005301Z vision_maskrcnn PASS 2024-06-26T08:31:56.6009726Z yolov3 PASS 2024-06-26T08:31:56.6460166Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_inference.csv 2024-06-26T08:31:56.9565048Z llama PASS 2024-06-26T08:31:56.9569519Z llama_v2_7b_16h PASS 2024-06-26T08:31:56.9573898Z llava PASS 2024-06-26T08:31:56.9578760Z maml PASS 2024-06-26T08:31:56.9583727Z maml_omniglot PASS 2024-06-26T08:31:56.9588667Z mnasnet1_0 PASS 2024-06-26T08:31:56.9593766Z mobilenet_v2 PASS 2024-06-26T08:31:56.9598648Z mobilenet_v2_quantized_qat PASS 2024-06-26T08:31:56.9603374Z mobilenet_v3_large PASS 2024-06-26T08:31:56.9608200Z moco PASS 2024-06-26T08:31:56.9612904Z moondream PASS 2024-06-26T08:31:56.9617835Z nanogpt PASS 2024-06-26T08:31:56.9622709Z nvidia_deeprecommender PASS 2024-06-26T08:31:56.9627540Z opacus_cifar10 PASS 2024-06-26T08:31:56.9632406Z phlippe_densenet PASS 2024-06-26T08:31:56.9637567Z phlippe_resnet PASS 2024-06-26T08:31:56.9642511Z pyhpc_equation_of_state PASS 2024-06-26T08:31:56.9647357Z pyhpc_isoneutral_mixing PASS 2024-06-26T08:31:56.9652308Z pyhpc_turbulent_kinetic_energy PASS 2024-06-26T08:31:56.9657189Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T08:31:56.9662166Z pytorch_stargan PASS 2024-06-26T08:31:56.9667382Z pytorch_unet PASS 2024-06-26T08:31:56.9672049Z resnet152 PASS 2024-06-26T08:31:56.9676902Z resnet18 PASS 2024-06-26T08:31:56.9681824Z resnet50 PASS 2024-06-26T08:31:56.9686740Z resnet50_quantized_qat PASS 2024-06-26T08:31:56.9691791Z resnext50_32x4d PASS 2024-06-26T08:31:56.9696521Z sam PASS 2024-06-26T08:31:56.9701616Z shufflenet_v2_x1_0 PASS 2024-06-26T08:31:56.9706500Z soft_actor_critic PASS 2024-06-26T08:31:56.9711404Z speech_transformer PASS 2024-06-26T08:31:56.9716640Z squeezenet1_1 PASS 2024-06-26T08:31:56.9721551Z stable_diffusion_text_encoder PASS 2024-06-26T08:31:56.9726297Z stable_diffusion_unet PASS 2024-06-26T08:31:56.9731532Z timm_efficientnet PASS 2024-06-26T08:31:56.9736421Z timm_regnet PASS 2024-06-26T08:31:56.9741345Z timm_resnest PASS 2024-06-26T08:31:56.9746648Z timm_vision_transformer PASS 2024-06-26T08:31:56.9751531Z timm_vision_transformer_large PASS 2024-06-26T08:31:56.9756659Z timm_vovnet PASS 2024-06-26T08:31:56.9761558Z torch_multimodal_clip PASS 2024-06-26T08:31:56.9766373Z tts_angular PASS 2024-06-26T08:31:56.9771251Z vgg16 PASS 2024-06-26T08:31:56.9776152Z vision_maskrcnn PASS 2024-06-26T08:31:56.9781027Z yolov3 PASS 2024-06-26T08:31:57.0240437Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-06-26T08:31:57.0242653Z ++ pwd 2024-06-26T08:31:57.0244916Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:31:57.0245613Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T08:31:57.0254534Z + local name=training 2024-06-26T08:31:57.0255722Z + shift 2024-06-26T08:31:57.0256125Z + local suite=torchbench 2024-06-26T08:31:57.0256501Z + shift 2024-06-26T08:31:57.0256838Z + local shard_id=1 2024-06-26T08:31:57.0257138Z + shift 2024-06-26T08:31:57.0257459Z + partition_flags=() 2024-06-26T08:31:57.0257801Z + local partition_flags 2024-06-26T08:31:57.0258327Z + [[ -n 2 ]] 2024-06-26T08:31:57.0258627Z + [[ -n 1 ]] 2024-06-26T08:31:57.0259260Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T08:31:57.0260019Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-06-26T08:31:57.0260538Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-06-26T08:31:57.0261185Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-06-26T08:31:57.0263542Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-06-26T08:32:02.2397973Z 2024-06-26T08:32:02.5103687Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:02.5104356Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:02.5104991Z cuda train llava 2024-06-26T08:32:02.5110577Z Traceback (most recent call last): 2024-06-26T08:32:02.5112002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:32:02.5113023Z ) = runner.load_model( 2024-06-26T08:32:02.5114007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 320, in load_model 2024-06-26T08:32:02.5115077Z benchmark = benchmark_cls( 2024-06-26T08:32:02.5116125Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-06-26T08:32:02.5117347Z obj = type.__call__(cls, *args, **kwargs) 2024-06-26T08:32:02.5118663Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-06-26T08:32:02.5120392Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:32:02.5122381Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-06-26T08:32:02.5124209Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-06-26T08:32:02.5125743Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-06-26T08:32:02.5126887Z self._skip_by_device_name() 2024-06-26T08:32:02.5128150Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-06-26T08:32:02.5130075Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-06-26T08:32:02.5131759Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-06-26T08:32:02.5132677Z 2024-06-26T08:32:02.5132868Z model_fail_to_load 2024-06-26T08:32:05.6775659Z 2024-06-26T08:32:06.2089484Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:06.2090020Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:06.2090470Z cuda train maml_omniglot 2024-06-26T08:32:15.6364936Z W0626 08:32:15.635000 139919450665600 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:21.4873957Z pass 2024-06-26T08:32:21.4875318Z TIMING: entire_frame_compile:13.55324 code_gen:6.80565 inductor_compile:8.91226 backend_compile:12.47222 2024-06-26T08:32:21.4878137Z STATS: call_* op count: 78 | FakeTensor.__torch_dispatch__:1255 | FakeTensorMode.__torch_dispatch__:7719 | attempt fast:88 | fast is_contiguous:88 | ProxyTorchDispatchMode.__torch_dispatch__:1538 2024-06-26T08:32:21.4880669Z Dynamo produced 3 graphs covering 78 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:25.2624047Z 2024-06-26T08:32:27.5008898Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:27.5009511Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:32:27.5010774Z cuda train mnasnet1_0 2024-06-26T08:33:11.7593569Z W0626 08:33:11.758000 139954233926272 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:33:56.8027146Z pass 2024-06-26T08:33:56.8054264Z TIMING: entire_frame_compile:82.76447 code_gen:25.40893 inductor_compile:47.71905 backend_compile:72.0185 2024-06-26T08:33:56.8056620Z STATS: call_* op count: 648 | FakeTensor.__torch_dispatch__:14766 | FakeTensorMode.__torch_dispatch__:84946 | attempt fast:1436 | fast is_contiguous:1436 | ProxyTorchDispatchMode.__torch_dispatch__:17678 2024-06-26T08:33:56.8058136Z Dynamo produced 3 graphs covering 648 ops with 7 graph breaks (5 unique) 2024-06-26T08:34:03.9067977Z 2024-06-26T08:34:05.6829452Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:34:05.6830315Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:34:05.6830768Z cuda train mobilenet_v2 2024-06-26T08:34:59.7360392Z pass 2024-06-26T08:34:59.7408264Z TIMING: entire_frame_compile:49.2877 code_gen:9.47518 inductor_compile:27.32925 backend_compile:43.40297 2024-06-26T08:34:59.7410295Z STATS: call_* op count: 157 | FakeTensor.__torch_dispatch__:7895 | FakeTensorMode.__torch_dispatch__:50372 | attempt fast:1572 | fast is_contiguous:1572 | ProxyTorchDispatchMode.__torch_dispatch__:10584 2024-06-26T08:34:59.7411631Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-06-26T08:35:05.4970936Z 2024-06-26T08:35:07.2605072Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:07.2605693Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:35:07.2606243Z cuda train mobilenet_v2_quantized_qat 2024-06-26T08:35:07.2606721Z Traceback (most recent call last): 2024-06-26T08:35:07.2607492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:35:07.2608402Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:35:07.2610706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:35:07.2611553Z pred = mod(*cloned_inputs) 2024-06-26T08:35:07.2612542Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:35:07.2613402Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:35:07.2614341Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:35:07.2615167Z raise e 2024-06-26T08:35:07.2615972Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:35:07.2617004Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:35:07.2618271Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:35:07.2619236Z return self._call_impl(*args, **kwargs) 2024-06-26T08:35:07.2620270Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:07.2621268Z return forward_call(*args, **kwargs) 2024-06-26T08:35:07.2621731Z File ".3", line 207, in forward 2024-06-26T08:35:07.2622556Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-06-26T08:35:07.2623920Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:35:07.2624864Z return self._call_impl(*args, **kwargs) 2024-06-26T08:35:07.2625846Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:07.2626730Z return forward_call(*args, **kwargs) 2024-06-26T08:35:07.2627760Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:35:07.2628721Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:35:07.2629338Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:35:07.2629746Z 2024-06-26T08:35:07.2630061Z The above exception was the direct cause of the following exception: 2024-06-26T08:35:07.2630530Z 2024-06-26T08:35:07.2630713Z Traceback (most recent call last): 2024-06-26T08:35:07.2631374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:35:07.2632340Z ) = runner.load_model( 2024-06-26T08:35:07.2633027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:35:07.2634131Z self.validate_model(model, example_inputs) 2024-06-26T08:35:07.2634933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:35:07.2635897Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:35:07.2636372Z RuntimeError: Eager run failed 2024-06-26T08:35:07.2636672Z 2024-06-26T08:35:07.2636817Z eager_fail_to_run 2024-06-26T08:35:10.4989063Z 2024-06-26T08:35:12.6513708Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:35:12.6514188Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:35:12.6514630Z cuda train mobilenet_v3_large 2024-06-26T08:36:14.4386993Z W0626 08:36:14.437000 140294265115264 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:37:03.3368040Z E0626 08:37:03.336000 140294265115264 torch/_dynamo/utils.py:1470] RMSE (res-fp64): 0.01441, (ref-fp64): 0.00390 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000 2024-06-26T08:37:03.3369717Z E0626 08:37:03.336000 140294265115264 torch/_dynamo/utils.py:1384] Accuracy failed for key name features.12.block.2.fc1.bias.grad 2024-06-26T08:37:03.3390295Z pass 2024-06-26T08:37:03.3427965Z TIMING: entire_frame_compile:103.22284 code_gen:26.91807 inductor_compile:57.71305 backend_compile:88.78758 2024-06-26T08:37:03.3429479Z STATS: call_* op count: 731 | FakeTensor.__torch_dispatch__:17580 | FakeTensorMode.__torch_dispatch__:100831 | attempt fast:2054 | fast is_contiguous:2054 | ProxyTorchDispatchMode.__torch_dispatch__:19585 2024-06-26T08:37:03.3430767Z Dynamo produced 3 graphs covering 731 ops with 7 graph breaks (5 unique) 2024-06-26T08:37:11.5922936Z 2024-06-26T08:37:13.5803832Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:13.5804438Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:37:13.5804880Z cuda train moco 2024-06-26T08:37:18.8427297Z [rank0]:W0626 08:37:18.841000 139992724193920 torch/_logging/_internal.py:1040] [3/0] Profiler function will be ignored 2024-06-26T08:37:26.3875419Z [rank0]:W0626 08:37:26.386000 139992724193920 torch/_dynamo/backends/distributed.py:88] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-06-26T08:37:26.4070764Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-06-26T08:37:26.4078683Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-06-26T08:37:26.4080277Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-06-26T08:37:26.4081805Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-06-26T08:37:26.4083268Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] │ 0 │ 161 │ 94032128 │ 2024-06-26T08:37:26.4084808Z [rank0]:W0626 08:37:26.406000 139992724193920 torch/_dynamo/backends/distributed.py:105] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-06-26T08:38:19.1223991Z skipping cudagraphs due to mutated inputs (161 instances) 2024-06-26T08:38:21.3967117Z [rank0]:W0626 08:38:21.396000 139992724193920 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T08:38:21.4543602Z skipping cudagraphs due to skipping cudagraphs due to cpu device (randperm). Found from : 2024-06-26T08:38:21.4546465Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 82, in _batch_shuffle_ddp 2024-06-26T08:38:21.4547660Z idx_shuffle = torch.randperm(batch_size_all).cuda() 2024-06-26T08:38:21.4548022Z 2024-06-26T08:38:23.5317572Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T08:38:23.5318982Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T08:38:23.5320178Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] or: 2024-06-26T08:38:23.5321370Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T08:38:23.5322792Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] to include these operations in the captured graph. 2024-06-26T08:38:23.5324024Z [rank0]:W0626 08:38:23.531000 139992724193920 torch/_dynamo/variables/tensor.py:714] [7/0] 2024-06-26T08:38:52.0381429Z pass 2024-06-26T08:38:52.0710416Z TIMING: entire_frame_compile:85.17342 inductor_compile:43.80508 backend_compile:73.22392 code_gen:18.62416 2024-06-26T08:38:52.0714715Z STATS: call_* op count: 907 | FakeTensorMode.__torch_dispatch__:91780 | ProxyTorchDispatchMode.__torch_dispatch__:20700 | FakeTensor.__torch_dispatch__:13447 | attempt fast:2631 | fast is_contiguous:2615 | slow no contiguity match:16 2024-06-26T08:38:52.0717536Z Dynamo produced 10 graphs covering 907 ops with 11 graph breaks (8 unique) 2024-06-26T08:38:54.0862262Z [rank0]:[W626 08:38:54.130211465 ProcessGroupNCCL.cpp:1161] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-06-26T08:38:59.3971880Z 2024-06-26T08:39:01.2801138Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-06-26T08:39:01.4987048Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-06-26T08:39:01.4987887Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-06-26T08:39:01.4993191Z using fused AdamW: True 2024-06-26T08:39:01.8321476Z 2024-06-26T08:39:01.8322738Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:39:01.8323418Z cuda train nanogpt 2024-06-26T08:39:55.0462453Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:39:55.0463898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T08:39:55.0466612Z pred = mod(*cloned_inputs) 2024-06-26T08:39:55.0467871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:39:55.0468747Z return forward_call(*args, **kwargs) 2024-06-26T08:39:55.0469841Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 228, in forward 2024-06-26T08:39:55.0470988Z tok_emb = self.transformer.wte(idx) # token embeddings of shape (b, t, n_embd) 2024-06-26T08:39:55.0471481Z 2024-06-26T08:39:56.3258762Z W0626 08:39:56.325000 140083437859456 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:40:34.3785205Z pass 2024-06-26T08:40:34.4416166Z TIMING: entire_frame_compile:87.00885 code_gen:20.4143 inductor_compile:47.80689 backend_compile:75.68789 2024-06-26T08:40:34.4418500Z STATS: call_* op count: 883 | FakeTensorMode.__torch_dispatch__:90972 | FakeTensor.__torch_dispatch__:13453 | attempt fast:2561 | fast is_contiguous:2561 | ProxyTorchDispatchMode.__torch_dispatch__:19295 2024-06-26T08:40:34.4420352Z Dynamo produced 3 graphs covering 883 ops with 7 graph breaks (5 unique) 2024-06-26T08:40:41.9542019Z 2024-06-26T08:40:44.3160854Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:40:44.3161419Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:40:44.3161868Z cuda train nvidia_deeprecommender 2024-06-26T08:40:52.3574257Z W0626 08:40:52.356000 140121947497088 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:40:57.5801163Z pass 2024-06-26T08:40:57.6091786Z TIMING: entire_frame_compile:10.86072 code_gen:3.59165 inductor_compile:5.78706 backend_compile:9.71764 2024-06-26T08:40:57.6096004Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:7601 | attempt fast:210 | fast is_contiguous:210 | FakeTensor.__torch_dispatch__:1171 | ProxyTorchDispatchMode.__torch_dispatch__:1490 2024-06-26T08:40:57.6097695Z Dynamo produced 3 graphs covering 71 ops with 7 graph breaks (5 unique) 2024-06-26T08:41:01.3222218Z 2024-06-26T08:41:02.1188902Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:02.1189418Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:02.1189982Z cuda train opacus_cifar10 2024-06-26T08:41:02.1216012Z Traceback (most recent call last): 2024-06-26T08:41:02.1216891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:41:02.1217641Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:41:02.1218486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:41:02.1219312Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:41:02.1220570Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:41:02.1221738Z torch.autograd.backward( 2024-06-26T08:41:02.1222659Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:41:02.1223512Z _engine_run_backward( 2024-06-26T08:41:02.1224688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:41:02.1226220Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:41:02.1227487Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 97, in __call__ 2024-06-26T08:41:02.1228336Z return self.hook(module, *args, **kwargs) 2024-06-26T08:41:02.1229460Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook 2024-06-26T08:41:02.1230529Z activations, backprops = self.rearrange_grad_samples( 2024-06-26T08:41:02.1231644Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples 2024-06-26T08:41:02.1232906Z raise ValueError( 2024-06-26T08:41:02.1233743Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-06-26T08:41:02.1234416Z 2024-06-26T08:41:02.1235115Z The above exception was the direct cause of the following exception: 2024-06-26T08:41:02.1235695Z 2024-06-26T08:41:02.1235893Z Traceback (most recent call last): 2024-06-26T08:41:02.1236539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:41:02.1237198Z ) = runner.load_model( 2024-06-26T08:41:02.1237850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:41:02.1238595Z self.validate_model(model, example_inputs) 2024-06-26T08:41:02.1239375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:41:02.1240364Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:41:02.1240839Z RuntimeError: Eager run failed 2024-06-26T08:41:02.1241090Z 2024-06-26T08:41:02.1241217Z eager_fail_to_run 2024-06-26T08:41:05.3450239Z 2024-06-26T08:41:06.0364994Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:06.0365664Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:41:06.0379314Z cuda train phlippe_densenet 2024-06-26T08:41:56.8900326Z pass 2024-06-26T08:41:56.8905182Z TIMING: entire_frame_compile:47.54939 code_gen:10.91743 inductor_compile:25.65124 backend_compile:41.90278 2024-06-26T08:41:56.8907752Z STATS: call_* op count: 190 | FakeTensor.__torch_dispatch__:6768 | FakeTensorMode.__torch_dispatch__:48459 | attempt fast:1414 | fast is_contiguous:1366 | ProxyTorchDispatchMode.__torch_dispatch__:11455 | slow no contiguity match:48 2024-06-26T08:41:56.8909455Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-06-26T08:42:02.3941474Z 2024-06-26T08:42:02.9817495Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:02.9817980Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:02.9818432Z cuda train phlippe_resnet 2024-06-26T08:42:21.9291672Z pass 2024-06-26T08:42:21.9292404Z TIMING: entire_frame_compile:17.02979 code_gen:2.99833 inductor_compile:8.27098 backend_compile:14.99464 2024-06-26T08:42:21.9296468Z STATS: call_* op count: 75 | FakeTensor.__torch_dispatch__:2555 | FakeTensorMode.__torch_dispatch__:18058 | attempt fast:567 | fast is_contiguous:567 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-06-26T08:42:21.9297782Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-06-26T08:42:26.0296779Z 2024-06-26T08:42:27.1509534Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:42:27.1510093Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:42:27.1510653Z cuda train pytorch_CycleGAN_and_pix2pix 2024-06-26T08:42:56.4460769Z pass 2024-06-26T08:42:56.4547867Z TIMING: entire_frame_compile:8.37674 code_gen:8.21426 inductor_compile:12.44928 backend_compile:7.67787 2024-06-26T08:42:56.4550258Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:17458 | FakeTensor.__torch_dispatch__:3096 | ProxyTorchDispatchMode.__torch_dispatch__:5123 2024-06-26T08:42:56.4551372Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-06-26T08:43:00.5509269Z 2024-06-26T08:43:01.9170038Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:43:01.9172008Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:43:01.9172579Z cuda train pytorch_stargan 2024-06-26T08:43:57.1541625Z pass 2024-06-26T08:43:57.1617027Z TIMING: entire_frame_compile:26.94962 code_gen:3.83773 inductor_compile:13.1673 backend_compile:23.9241 2024-06-26T08:43:57.1619130Z STATS: call_* op count: 61 | FakeTensorMode.__torch_dispatch__:26002 | FakeTensor.__torch_dispatch__:2568 | attempt fast:830 | fast is_contiguous:830 | ProxyTorchDispatchMode.__torch_dispatch__:5427 2024-06-26T08:43:57.1620417Z Dynamo produced 2 graphs covering 61 ops with 6 graph breaks (5 unique) 2024-06-26T08:44:01.8812617Z 2024-06-26T08:44:02.6815590Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:02.6816461Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:44:02.6817380Z cuda train pytorch_unet 2024-06-26T08:45:03.6218976Z W0626 08:45:03.621000 140433962099328 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:45:25.4001942Z pass_due_to_skip 2024-06-26T08:45:25.4093795Z TIMING: entire_frame_compile:49.71268 code_gen:19.04641 inductor_compile:32.36481 backend_compile:44.36004 2024-06-26T08:45:25.4096602Z STATS: call_* op count: 315 | FakeTensor.__torch_dispatch__:7154 | FakeTensorMode.__torch_dispatch__:41985 | attempt fast:744 | fast is_contiguous:736 | ProxyTorchDispatchMode.__torch_dispatch__:9107 | slow no contiguity match:8 2024-06-26T08:45:25.4098066Z Dynamo produced 3 graphs covering 315 ops with 7 graph breaks (5 unique) 2024-06-26T08:45:30.7489854Z 2024-06-26T08:45:32.8784085Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:45:32.8784964Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:45:32.8785831Z cuda train resnet152 2024-06-26T08:47:31.6369329Z W0626 08:47:31.636000 139763966497408 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:49:41.8518426Z pass 2024-06-26T08:49:41.8876182Z TIMING: entire_frame_compile:229.03309 code_gen:53.79264 inductor_compile:122.97963 backend_compile:199.03761 2024-06-26T08:49:41.8878576Z STATS: call_* op count: 1938 | FakeTensor.__torch_dispatch__:43363 | FakeTensorMode.__torch_dispatch__:254581 | attempt fast:4290 | fast is_contiguous:4290 | ProxyTorchDispatchMode.__torch_dispatch__:53818 2024-06-26T08:49:41.8879900Z Dynamo produced 3 graphs covering 1938 ops with 7 graph breaks (5 unique) 2024-06-26T08:49:56.3492286Z 2024-06-26T08:49:57.4015939Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:49:57.4016865Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:49:57.4017725Z cuda train resnet18 2024-06-26T08:50:19.3780315Z pass 2024-06-26T08:50:19.3856173Z TIMING: entire_frame_compile:18.78581 code_gen:5.25048 inductor_compile:11.2909 backend_compile:17.27896 2024-06-26T08:50:19.3858021Z STATS: call_* op count: 73 | FakeTensor.__torch_dispatch__:2577 | FakeTensorMode.__torch_dispatch__:17461 | attempt fast:570 | fast is_contiguous:570 | ProxyTorchDispatchMode.__torch_dispatch__:4304 2024-06-26T08:50:19.3859610Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-06-26T08:50:23.5730641Z 2024-06-26T08:50:25.1831101Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:50:25.1832102Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:50:25.1832557Z cuda train resnet50 2024-06-26T08:51:13.4559559Z pass 2024-06-26T08:51:13.4736556Z TIMING: entire_frame_compile:40.86868 code_gen:9.09589 inductor_compile:22.72497 backend_compile:37.47588 2024-06-26T08:51:13.4739197Z STATS: call_* op count: 179 | FakeTensor.__torch_dispatch__:6682 | FakeTensorMode.__torch_dispatch__:44833 | attempt fast:1468 | fast is_contiguous:1468 | ProxyTorchDispatchMode.__torch_dispatch__:11223 2024-06-26T08:51:13.4740693Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-06-26T08:51:18.8297526Z 2024-06-26T08:51:20.8345622Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:51:20.8346190Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:51:20.8346634Z cuda train resnet50_quantized_qat 2024-06-26T08:51:20.8348894Z Traceback (most recent call last): 2024-06-26T08:51:20.8349844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:51:20.8350750Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:51:20.8352105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in forward_and_backward_pass 2024-06-26T08:51:20.8353096Z pred = mod(*cloned_inputs) 2024-06-26T08:51:20.8354296Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped 2024-06-26T08:51:20.8355356Z return self._wrapped_call(self, *args, **kwargs) 2024-06-26T08:51:20.8356349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 316, in __call__ 2024-06-26T08:51:20.8357167Z raise e 2024-06-26T08:51:20.8358179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 303, in __call__ 2024-06-26T08:51:20.8359448Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-06-26T08:51:20.8360946Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:51:20.8361848Z return self._call_impl(*args, **kwargs) 2024-06-26T08:51:20.8362803Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:51:20.8363985Z return forward_call(*args, **kwargs) 2024-06-26T08:51:20.8364499Z File ".3", line 167, in forward 2024-06-26T08:51:20.8365304Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-06-26T08:51:20.8366448Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T08:51:20.8367328Z return self._call_impl(*args, **kwargs) 2024-06-26T08:51:20.8368247Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:51:20.8369064Z return forward_call(*args, **kwargs) 2024-06-26T08:51:20.8370015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 343, in forward 2024-06-26T08:51:20.8370910Z return torch.fused_moving_avg_obs_fake_quant( 2024-06-26T08:51:20.8371468Z RuntimeError: expected scalar type Float but found Half 2024-06-26T08:51:20.8371847Z 2024-06-26T08:51:20.8372150Z The above exception was the direct cause of the following exception: 2024-06-26T08:51:20.8372590Z 2024-06-26T08:51:20.8372735Z Traceback (most recent call last): 2024-06-26T08:51:20.8373388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:51:20.8374044Z ) = runner.load_model( 2024-06-26T08:51:20.8374706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:51:20.8375450Z self.validate_model(model, example_inputs) 2024-06-26T08:51:20.8376202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:51:20.8376958Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:51:20.8377477Z RuntimeError: Eager run failed 2024-06-26T08:51:20.8377728Z 2024-06-26T08:51:20.8377844Z eager_fail_to_run 2024-06-26T08:51:24.1203473Z 2024-06-26T08:51:25.8529467Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:51:25.8530058Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:51:25.8530526Z cuda train resnext50_32x4d 2024-06-26T08:52:14.2586130Z W0626 08:52:14.257000 139985824244352 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:52:58.8375425Z pass 2024-06-26T08:52:58.8490666Z TIMING: entire_frame_compile:81.7499 code_gen:21.95585 inductor_compile:44.93252 backend_compile:71.1196 2024-06-26T08:52:58.8495334Z STATS: call_* op count: 680 | FakeTensor.__torch_dispatch__:14997 | FakeTensorMode.__torch_dispatch__:87903 | attempt fast:1468 | fast is_contiguous:1468 | ProxyTorchDispatchMode.__torch_dispatch__:18492 2024-06-26T08:52:58.8497955Z Dynamo produced 3 graphs covering 680 ops with 7 graph breaks (5 unique) 2024-06-26T08:53:06.1166636Z 2024-06-26T08:53:14.3056051Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:14.3056546Z loading model: 0it [00:08, ?it/s] 2024-06-26T08:53:14.3057882Z cuda train sam 2024-06-26T08:53:14.3065862Z Traceback (most recent call last): 2024-06-26T08:53:14.3066694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T08:53:14.3067692Z self.model_iter_fn(model, example_inputs) 2024-06-26T08:53:14.3068684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 458, in forward_and_backward_pass 2024-06-26T08:53:14.3069632Z self.grad_scaler.scale(loss).backward() 2024-06-26T08:53:14.3070587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-06-26T08:53:14.3071519Z torch.autograd.backward( 2024-06-26T08:53:14.3072683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 288, in backward 2024-06-26T08:53:14.3073571Z _engine_run_backward( 2024-06-26T08:53:14.3074711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 768, in _engine_run_backward 2024-06-26T08:53:14.3076074Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-06-26T08:53:14.3077155Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-06-26T08:53:14.3077678Z 2024-06-26T08:53:14.3077978Z The above exception was the direct cause of the following exception: 2024-06-26T08:53:14.3078479Z 2024-06-26T08:53:14.3078625Z Traceback (most recent call last): 2024-06-26T08:53:14.3079288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T08:53:14.3079952Z ) = runner.load_model( 2024-06-26T08:53:14.3080606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 380, in load_model 2024-06-26T08:53:14.3081362Z self.validate_model(model, example_inputs) 2024-06-26T08:53:14.3082128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T08:53:14.3082899Z raise RuntimeError("Eager run failed") from e 2024-06-26T08:53:14.3083369Z RuntimeError: Eager run failed 2024-06-26T08:53:14.3083633Z 2024-06-26T08:53:14.3083756Z eager_fail_to_run 2024-06-26T08:53:17.6131561Z 2024-06-26T08:53:18.6810303Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:53:18.6810867Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:53:18.6811307Z cuda train shufflenet_v2_x1_0 2024-06-26T08:54:15.0340818Z pass 2024-06-26T08:54:15.0349410Z TIMING: entire_frame_compile:52.58019 code_gen:13.80046 inductor_compile:32.12183 backend_compile:48.55586 2024-06-26T08:54:15.0351222Z STATS: call_* op count: 307 | FakeTensor.__torch_dispatch__:7797 | FakeTensorMode.__torch_dispatch__:51977 | attempt fast:1552 | fast is_contiguous:1552 | ProxyTorchDispatchMode.__torch_dispatch__:12199 2024-06-26T08:54:15.0352790Z Dynamo produced 2 graphs covering 307 ops with 6 graph breaks (5 unique) 2024-06-26T08:54:20.7871279Z 2024-06-26T08:54:21.5024082Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:21.5024676Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:21.5025106Z cuda train soft_actor_critic 2024-06-26T08:54:27.5765277Z W0626 08:54:27.575000 139791460258432 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:54:30.6771205Z pass 2024-06-26T08:54:30.6778099Z TIMING: entire_frame_compile:7.26205 code_gen:2.27511 inductor_compile:3.45081 backend_compile:6.48478 2024-06-26T08:54:30.6779669Z STATS: call_* op count: 59 | FakeTensor.__torch_dispatch__:598 | FakeTensorMode.__torch_dispatch__:4414 | attempt fast:69 | fast is_contiguous:69 | ProxyTorchDispatchMode.__torch_dispatch__:894 2024-06-26T08:54:30.6780996Z Dynamo produced 4 graphs covering 59 ops with 6 graph breaks (5 unique) 2024-06-26T08:54:34.2990658Z 2024-06-26T08:54:35.4938151Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:35.4938757Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:54:35.4939234Z Traceback (most recent call last): 2024-06-26T08:54:35.4940205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 473, in 2024-06-26T08:54:35.4941215Z torchbench_main() 2024-06-26T08:54:35.4942028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torchbench_main 2024-06-26T08:54:35.4942898Z main(TorchBenchmarkRunner(), original_dir) 2024-06-26T08:54:35.4943721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3722, in main 2024-06-26T08:54:35.4944600Z process_entry(0, runner, original_dir, args) 2024-06-26T08:54:35.4945351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3654, in process_entry 2024-06-26T08:54:35.4947650Z return run(runner, args, original_dir) 2024-06-26T08:54:35.4948382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4283, in run 2024-06-26T08:54:35.4952591Z assert marked, f"nothing in example_inputs had a dim with {batch_size}" 2024-06-26T08:54:35.4953617Z AssertionError: nothing in example_inputs had a dim with 32 2024-06-26T08:54:36.1487505Z Run failed with return code: 1 2024-06-26T08:54:36.1487980Z Output: None 2024-06-26T08:54:36.1488281Z Error: None 2024-06-26T08:54:38.6981262Z 2024-06-26T08:54:39.6541486Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:39.6541980Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:54:39.6542413Z cuda train squeezenet1_1 2024-06-26T08:55:02.1143119Z pass 2024-06-26T08:55:02.1163631Z TIMING: entire_frame_compile:19.83401 code_gen:10.33446 inductor_compile:14.54151 backend_compile:18.82705 2024-06-26T08:55:02.1165175Z STATS: call_* op count: 70 | FakeTensor.__torch_dispatch__:1285 | FakeTensorMode.__torch_dispatch__:11158 | ProxyTorchDispatchMode.__torch_dispatch__:2530 | attempt fast:16 | fast is_contiguous:16 2024-06-26T08:55:02.1167509Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-06-26T08:55:06.2429090Z 2024-06-26T08:55:07.6017801Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:55:07.6018345Z 2024-06-26T08:55:08.2233573Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-06-26T08:58:39.0831426Z pass 2024-06-26T08:58:39.0880122Z TIMING: entire_frame_compile:127.36175 code_gen:31.14569 inductor_compile:69.10009 backend_compile:108.17058 2024-06-26T08:58:39.0881653Z STATS: call_* op count: 974 | FakeTensor.__torch_dispatch__:20845 | FakeTensorMode.__torch_dispatch__:120950 | attempt fast:2404 | fast is_contiguous:2404 | ProxyTorchDispatchMode.__torch_dispatch__:24017 2024-06-26T08:58:39.0882969Z Dynamo produced 3 graphs covering 974 ops with 7 graph breaks (5 unique) 2024-06-26T08:58:48.6349916Z 2024-06-26T08:58:51.1783871Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:58:51.1784526Z loading model: 0it [00:02, ?it/s] 2024-06-26T08:58:51.1785202Z cuda train timm_regnet 2024-06-26T09:00:17.2602372Z pass 2024-06-26T09:00:17.3006666Z TIMING: entire_frame_compile:71.63773 code_gen:13.61163 inductor_compile:37.72377 backend_compile:61.81537 2024-06-26T09:00:17.3008722Z STATS: call_* op count: 463 | FakeTensor.__torch_dispatch__:9909 | FakeTensorMode.__torch_dispatch__:71901 | attempt fast:2163 | fast is_contiguous:2163 | ProxyTorchDispatchMode.__torch_dispatch__:17103 2024-06-26T09:00:17.3010043Z Dynamo produced 2 graphs covering 463 ops with 6 graph breaks (5 unique) 2024-06-26T09:00:24.2594037Z 2024-06-26T09:00:26.0376259Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:00:26.0376867Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:00:26.0380357Z cuda train timm_resnest 2024-06-26T09:01:08.4664192Z W0626 09:01:08.465000 140543283126912 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:01:33.0600896Z pass 2024-06-26T09:01:33.0669389Z TIMING: entire_frame_compile:59.25991 code_gen:22.16046 inductor_compile:38.3106 backend_compile:52.8402 2024-06-26T09:01:33.0671452Z STATS: call_* op count: 431 | FakeTensor.__torch_dispatch__:8072 | FakeTensorMode.__torch_dispatch__:48296 | attempt fast:769 | fast is_contiguous:753 | ProxyTorchDispatchMode.__torch_dispatch__:9820 | slow no contiguity match:16 2024-06-26T09:01:33.0673084Z Dynamo produced 3 graphs covering 431 ops with 7 graph breaks (5 unique) 2024-06-26T09:01:38.9389715Z 2024-06-26T09:01:40.3605019Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:01:40.3605666Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:01:40.3606258Z cuda train timm_vision_transformer 2024-06-26T09:02:33.9908651Z pass 2024-06-26T09:02:34.0085632Z TIMING: entire_frame_compile:49.4052 code_gen:10.3488 inductor_compile:26.53 backend_compile:43.84004 2024-06-26T09:02:34.0087833Z STATS: call_* op count: 381 | FakeTensor.__torch_dispatch__:6185 | FakeTensorMode.__torch_dispatch__:48790 | attempt fast:2319 | fast is_contiguous:2319 | ProxyTorchDispatchMode.__torch_dispatch__:10943 2024-06-26T09:02:34.0089510Z Dynamo produced 2 graphs covering 381 ops with 6 graph breaks (5 unique) 2024-06-26T09:02:40.0049438Z 2024-06-26T09:02:52.1212720Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:52.1213303Z loading model: 0it [00:12, ?it/s] 2024-06-26T09:02:52.1227100Z cuda train timm_vision_transformer_large 2024-06-26T09:02:52.1227864Z pass_due_to_skip 2024-06-26T09:02:52.3841377Z TIMING: 2024-06-26T09:02:52.3841830Z STATS: call_* op count: 0 2024-06-26T09:02:52.3842542Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-06-26T09:02:55.7362074Z 2024-06-26T09:02:57.5303453Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:02:57.5304084Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:02:57.5304516Z cuda train timm_vovnet 2024-06-26T09:03:51.5081430Z pass 2024-06-26T09:03:51.5260677Z TIMING: entire_frame_compile:45.26792 code_gen:14.78115 inductor_compile:28.02457 backend_compile:40.22488 2024-06-26T09:03:51.5262871Z STATS: call_* op count: 174 | FakeTensor.__torch_dispatch__:4751 | FakeTensorMode.__torch_dispatch__:34733 | attempt fast:1112 | fast is_contiguous:1052 | ProxyTorchDispatchMode.__torch_dispatch__:8526 | slow no contiguity match:60 2024-06-26T09:03:51.5264559Z Dynamo produced 2 graphs covering 174 ops with 6 graph breaks (5 unique) 2024-06-26T09:03:56.9423687Z 2024-06-26T09:03:58.6785987Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:03:58.6786453Z loading model: 0it [00:01, ?it/s] 2024-06-26T09:03:58.6786906Z cuda train torch_multimodal_clip 2024-06-26T09:05:01.5834706Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_302) 2024-06-26T09:05:27.4901523Z W0626 09:05:27.489000 140712310198912 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:06:42.2090503Z pass 2024-06-26T09:06:42.2900949Z TIMING: entire_frame_compile:154.04721 code_gen:41.09885 inductor_compile:86.64846 backend_compile:135.163 2024-06-26T09:06:42.2904377Z STATS: call_* op count: 963 | FakeTensor.__torch_dispatch__:26756 | FakeTensorMode.__torch_dispatch__:177437 | attempt fast:2217 | fast is_contiguous:2135 | slow no contiguity match:82 | ProxyTorchDispatchMode.__torch_dispatch__:40036 2024-06-26T09:06:42.2905863Z Dynamo produced 3 graphs covering 963 ops with 7 graph breaks (5 unique) 2024-06-26T09:06:53.3145526Z 2024-06-26T09:06:53.8612550Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:06:53.8613019Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:06:53.8613463Z cuda train tts_angular 2024-06-26T09:06:57.1655651Z W0626 09:06:57.164000 140452359025280 torch/_logging/_internal.py:1040] [10/0] Profiler function will be ignored 2024-06-26T09:07:03.4632237Z pass 2024-06-26T09:07:03.4696623Z TIMING: entire_frame_compile:7.5561 inductor_compile:5.73241 backend_compile:6.75594 code_gen:4.29007 2024-06-26T09:07:03.4698517Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4810 | ProxyTorchDispatchMode.__torch_dispatch__:871 | FakeTensor.__torch_dispatch__:920 2024-06-26T09:07:03.4700203Z Dynamo produced 5 graphs covering 71 ops with 9 graph breaks (6 unique) 2024-06-26T09:07:07.1182892Z 2024-06-26T09:07:09.2567196Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:07:09.2567935Z loading model: 0it [00:02, ?it/s] 2024-06-26T09:07:09.2568518Z cuda train vgg16 2024-06-26T09:07:44.8354410Z pass 2024-06-26T09:07:44.8595268Z TIMING: entire_frame_compile:12.49488 code_gen:6.11447 inductor_compile:9.16158 backend_compile:11.84233 2024-06-26T09:07:44.8596880Z STATS: call_* op count: 44 | FakeTensor.__torch_dispatch__:804 | FakeTensorMode.__torch_dispatch__:7375 | attempt fast:34 | fast is_contiguous:34 | ProxyTorchDispatchMode.__torch_dispatch__:1686 2024-06-26T09:07:44.8598134Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-06-26T09:07:48.7542889Z 2024-06-26T09:07:51.8505917Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:07:51.8506874Z loading model: 0it [00:03, ?it/s] 2024-06-26T09:07:51.8507388Z cuda train vision_maskrcnn 2024-06-26T09:08:25.2967528Z W0626 09:08:25.296000 140216806875776 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T09:08:25.2976851Z W0626 09:08:25.297000 140216806875776 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T09:08:25.2986941Z W0626 09:08:25.298000 140216806875776 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T09:08:25.2997519Z W0626 09:08:25.299000 140216806875776 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T09:08:25.3007582Z W0626 09:08:25.300000 140216806875776 torch/_inductor/utils.py:1234] [5/0_1] DeviceCopy in input program 2024-06-26T09:08:27.1716994Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 2024-06-26T09:08:27.1719091Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward 2024-06-26T09:08:27.1720269Z anchors = self.anchor_generator(images, features) 2024-06-26T09:08:27.1721610Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward 2024-06-26T09:08:27.1722881Z self.set_cell_anchors(dtype, device) 2024-06-26T09:08:27.1724274Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors 2024-06-26T09:08:27.1725562Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-26T09:08:27.1726836Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in 2024-06-26T09:08:27.1728021Z self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors] 2024-06-26T09:08:27.1728598Z 2024-06-26T09:08:28.7839440Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-06-26T09:08:28.7840852Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-06-26T09:08:28.7841937Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] or: 2024-06-26T09:08:28.7842983Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-06-26T09:08:28.7844253Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] to include these operations in the captured graph. 2024-06-26T09:08:28.7845304Z W0626 09:08:28.783000 140216806875776 torch/_dynamo/variables/tensor.py:714] [18/0] 2024-06-26T09:08:41.1262031Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:08:41.1263408Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:08:41.1264556Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:08:41.1265139Z 2024-06-26T09:08:46.7475040Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:08:46.7476392Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:08:46.7477504Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:08:46.7478085Z 2024-06-26T09:08:52.8701716Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:08:52.8703433Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:08:52.8705035Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:08:52.8706297Z 2024-06-26T09:09:31.3219381Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:09:31.3221202Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:09:31.3222551Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:09:31.3223135Z 2024-06-26T09:09:40.2342824Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:09:40.2344179Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:09:40.2345351Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:09:40.2345995Z 2024-06-26T09:09:54.1348417Z W0626 09:09:54.133000 140216806875776 torch/fx/experimental/symbolic_shapes.py:4538] [30/6] RecursionError in sympy.xreplace(Eq(Mod(2*s3, s4), 0), {s3: evaluate_static_shape_0 + 1, s4: evaluate_static_shape_1 + 1}) 2024-06-26T09:09:57.6187427Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:09:57.6188740Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:09:57.6190003Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:09:57.6190602Z 2024-06-26T09:10:03.4098637Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:03.4101288Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:10:03.4102444Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:10:03.4103021Z 2024-06-26T09:10:09.6315754Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:09.6317523Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-06-26T09:10:09.6318800Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-06-26T09:10:09.6319291Z 2024-06-26T09:10:10.5133524Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:10.5136051Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407 2024-06-26T09:10:10.5137310Z mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False) 2024-06-26T09:10:10.5137788Z 2024-06-26T09:10:11.5310185Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:11.5312150Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804 2024-06-26T09:10:11.5313290Z masks_probs = maskrcnn_inference(mask_logits, labels) 2024-06-26T09:10:11.5314485Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference 2024-06-26T09:10:11.5315472Z mask_prob = mask_prob[index, labels][:, None] 2024-06-26T09:10:11.5315799Z 2024-06-26T09:10:13.4210332Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:13.4211564Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-06-26T09:10:13.4212690Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-06-26T09:10:13.4213259Z 2024-06-26T09:10:17.0997592Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:17.0999132Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward 2024-06-26T09:10:17.1000803Z features = self.backbone(images.tensors) 2024-06-26T09:10:17.1002058Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward 2024-06-26T09:10:17.1002954Z x = self.fpn(x) 2024-06-26T09:10:17.1003983Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward 2024-06-26T09:10:17.1005037Z inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest") 2024-06-26T09:10:17.1005497Z 2024-06-26T09:10:22.2568394Z W0626 09:10:22.256000 140216806875776 torch/_dynamo/convert_frame.py:762] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-06-26T09:10:22.2570214Z W0626 09:10:22.256000 140216806875776 torch/_dynamo/convert_frame.py:762] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-06-26T09:10:22.2571990Z W0626 09:10:22.256000 140216806875776 torch/_dynamo/convert_frame.py:762] [30/8] last reason: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 623 2024-06-26T09:10:22.2573489Z W0626 09:10:22.256000 140216806875776 torch/_dynamo/convert_frame.py:762] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-06-26T09:10:22.2575066Z W0626 09:10:22.256000 140216806875776 torch/_dynamo/convert_frame.py:762] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-06-26T09:10:30.6885939Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:30.6887266Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:10:30.6888495Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:10:30.6889809Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:10:30.6890787Z v1 = masked_index(y_low, x_low) 2024-06-26T09:10:30.6891767Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:10:30.6892634Z return input[ 2024-06-26T09:10:30.6892895Z 2024-06-26T09:10:36.2814178Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:36.2815554Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:10:36.2816640Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:10:36.2817870Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:10:36.2818737Z v1 = masked_index(y_low, x_low) 2024-06-26T09:10:36.2819619Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:10:36.2820433Z return input[ 2024-06-26T09:10:36.2820631Z 2024-06-26T09:10:43.4485723Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:43.4487098Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:10:43.4488231Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:10:43.4489621Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:10:43.4490600Z v1 = masked_index(y_low, x_low) 2024-06-26T09:10:43.4491600Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:10:43.4492479Z return input[ 2024-06-26T09:10:43.4492684Z 2024-06-26T09:10:56.5005151Z W0626 09:10:56.499000 140216806875776 torch/fx/experimental/symbolic_shapes.py:4538] [54/4] RecursionError in sympy.xreplace(Eq(Mod(2*s4, s3), 0), {s4: evaluate_static_shape_0 + 1, s3: evaluate_static_shape_1 + 1}) 2024-06-26T09:10:58.2595963Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:10:58.2597510Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:10:58.2598749Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:10:58.2599983Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:10:58.2600843Z v1 = masked_index(y_low, x_low) 2024-06-26T09:10:58.2601726Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:10:58.2602532Z return input[ 2024-06-26T09:10:58.2602722Z 2024-06-26T09:11:09.2204865Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:11:09.2206234Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:11:09.2207314Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:11:09.2208554Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:11:09.2209421Z v1 = masked_index(y_low, x_low) 2024-06-26T09:11:09.2210296Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:11:09.2211093Z return input[ 2024-06-26T09:11:09.2211288Z 2024-06-26T09:11:18.2398871Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:11:18.2400137Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:11:18.2401290Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:11:18.2402538Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:11:18.2403418Z v1 = masked_index(y_low, x_low) 2024-06-26T09:11:18.2404303Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:11:18.2405088Z return input[ 2024-06-26T09:11:18.2405286Z 2024-06-26T09:11:23.9164130Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:11:23.9165603Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:11:23.9166673Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:11:23.9167895Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:11:23.9168784Z v1 = masked_index(y_low, x_low) 2024-06-26T09:11:23.9169668Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:11:23.9170457Z return input[ 2024-06-26T09:11:23.9170664Z 2024-06-26T09:11:24.9461070Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:11:24.9463349Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-06-26T09:11:24.9464566Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-06-26T09:11:24.9465864Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-06-26T09:11:24.9466823Z v1 = masked_index(y_low, x_low) 2024-06-26T09:11:24.9467797Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-06-26T09:11:24.9468957Z return input[ 2024-06-26T09:11:24.9469157Z 2024-06-26T09:11:25.2791042Z pass 2024-06-26T09:11:25.3407204Z TIMING: entire_frame_compile:195.24987 inductor_compile:94.92995 backend_compile:152.5546 code_gen:47.46895 2024-06-26T09:11:25.3409543Z STATS: call_* op count: 2515 | FakeTensorMode.__torch_dispatch__:168532 | FakeTensor.__torch_dispatch__:12905 | ProxyTorchDispatchMode.__torch_dispatch__:45349 | attempt fast:2807 | slow no contiguity match:274 | fast is_contiguous:2001 | slow both tensors nontrivially broadcast:532 2024-06-26T09:11:25.3411270Z Dynamo produced 51 graphs covering 2515 ops with 33 graph breaks (8 unique) 2024-06-26T09:11:38.4613100Z 2024-06-26T09:11:41.1113048Z loading model: 0it [00:00, ?it/s] 2024-06-26T09:11:41.1113654Z loading model: 0it [00:02, ?it/s] 2024-06-26T09:11:41.1114090Z cuda train yolov3 2024-06-26T09:12:51.8219134Z W0626 09:12:51.821000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:12:51.8223315Z W0626 09:12:51.821000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:12:52.1398014Z W0626 09:12:52.139000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:12:52.1401884Z W0626 09:12:52.139000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:12:52.4623744Z W0626 09:12:52.461000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:12:52.4629112Z W0626 09:12:52.462000 139906834866816 torch/_inductor/utils.py:1234] [3/0_2] DeviceCopy in input program 2024-06-26T09:13:07.2990300Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_226). Found from : 2024-06-26T09:13:07.2991591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:13:07.2992875Z pred = mod(*cloned_inputs) 2024-06-26T09:13:07.2994030Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:13:07.2994956Z return forward_call(*args, **kwargs) 2024-06-26T09:13:07.2996011Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-06-26T09:13:07.2997101Z return self.forward_once(x) 2024-06-26T09:13:07.2998162Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 290, in forward_once 2024-06-26T09:13:07.2999036Z yolo_out.append(module(x, out)) 2024-06-26T09:13:07.2999970Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 188, in forward 2024-06-26T09:13:07.3001079Z self.create_grids((nx, ny), p.device) 2024-06-26T09:13:07.3002107Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 159, in create_grids 2024-06-26T09:13:07.3003014Z self.anchor_vec = self.anchor_vec.to(device) 2024-06-26T09:13:07.3003346Z 2024-06-26T09:13:52.7345476Z W0626 09:13:52.733000 139906834866816 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T09:16:33.2616094Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T09:16:33.2617266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 456, in torch_dynamo_resume_in_forward_and_backward_pass_at_451 2024-06-26T09:16:33.2618206Z pred = mod(*cloned_inputs) 2024-06-26T09:16:33.2622188Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T09:16:33.2623106Z return forward_call(*args, **kwargs) 2024-06-26T09:16:33.2623982Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 238, in forward 2024-06-26T09:16:33.2624823Z return self.forward_once(x) 2024-06-26T09:16:33.2625977Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 292, in forward_once 2024-06-26T09:16:33.2626834Z x = module(x) 2024-06-26T09:16:33.2627026Z 2024-06-26T09:16:34.1523557Z pass 2024-06-26T09:16:34.1943803Z TIMING: entire_frame_compile:270.53222 code_gen:86.25287 inductor_compile:166.59783 backend_compile:240.22815 2024-06-26T09:16:34.1945422Z STATS: call_* op count: 1244 | FakeTensor.__torch_dispatch__:38404 | FakeTensorMode.__torch_dispatch__:223129 | attempt fast:7250 | fast is_contiguous:7238 | ProxyTorchDispatchMode.__torch_dispatch__:44435 | slow no contiguity match:12 2024-06-26T09:16:34.1946984Z Dynamo produced 4 graphs covering 1244 ops with 8 graph breaks (5 unique) 2024-06-26T09:16:48.3812850Z accuracy pass_rate=78.38% 2024-06-26T09:16:48.3816351Z calls_captured gmean=0.00x mean=421.757x 2024-06-26T09:16:48.3820118Z unique_graphs gmean=0.00x mean=3.649x 2024-06-26T09:16:48.3824204Z graph_breaks gmean=0.00x mean=6.135x 2024-06-26T09:16:48.3828476Z unique_graph_breaks gmean=0.00x mean=4.243x 2024-06-26T09:16:48.3832617Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T09:16:48.3836783Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T09:16:48.3840528Z cudagraph_skips gmean=0.00x mean=0.811x 2024-06-26T09:16:49.4879591Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_training.csv 2024-06-26T09:16:49.8038293Z llava XFAIL 2024-06-26T09:16:49.8042656Z maml_omniglot PASS 2024-06-26T09:16:49.8047558Z mnasnet1_0 PASS 2024-06-26T09:16:49.8051941Z mobilenet_v2 PASS 2024-06-26T09:16:49.8056675Z mobilenet_v2_quantized_qat XFAIL 2024-06-26T09:16:49.8061425Z mobilenet_v3_large PASS 2024-06-26T09:16:49.8066017Z moco PASS 2024-06-26T09:16:49.8071146Z nanogpt PASS 2024-06-26T09:16:49.8075805Z nvidia_deeprecommender PASS 2024-06-26T09:16:49.8080654Z opacus_cifar10 XFAIL 2024-06-26T09:16:49.8085101Z phlippe_densenet PASS 2024-06-26T09:16:49.8090024Z phlippe_resnet PASS 2024-06-26T09:16:49.8094499Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T09:16:49.8099177Z pytorch_stargan PASS 2024-06-26T09:16:49.8104009Z pytorch_unet XFAIL 2024-06-26T09:16:49.8108519Z resnet152 PASS 2024-06-26T09:16:49.8113257Z resnet18 PASS 2024-06-26T09:16:49.8118107Z resnet50 PASS 2024-06-26T09:16:49.8122803Z resnet50_quantized_qat XFAIL 2024-06-26T09:16:49.8127413Z resnext50_32x4d PASS 2024-06-26T09:16:49.8131990Z sam XFAIL 2024-06-26T09:16:49.8136460Z shufflenet_v2_x1_0 PASS 2024-06-26T09:16:49.8141154Z soft_actor_critic PASS 2024-06-26T09:16:49.8145913Z squeezenet1_1 PASS 2024-06-26T09:16:49.8150507Z stable_diffusion_text_encoder PASS 2024-06-26T09:16:49.8155270Z stable_diffusion_unet XFAIL 2024-06-26T09:16:49.8159908Z timm_efficientnet PASS 2024-06-26T09:16:49.8164473Z timm_regnet PASS 2024-06-26T09:16:49.8169130Z timm_resnest PASS 2024-06-26T09:16:49.8173647Z timm_vision_transformer PASS 2024-06-26T09:16:49.8178257Z timm_vision_transformer_large XFAIL 2024-06-26T09:16:49.8182978Z timm_vovnet PASS 2024-06-26T09:16:49.8187666Z torch_multimodal_clip PASS 2024-06-26T09:16:49.8192215Z tts_angular PASS 2024-06-26T09:16:49.8197375Z vgg16 PASS 2024-06-26T09:16:49.8201684Z vision_maskrcnn PASS 2024-06-26T09:16:49.8206400Z yolov3 PASS 2024-06-26T09:16:49.8664148Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_torchbench_training.csv 2024-06-26T09:16:50.1781679Z llava PASS 2024-06-26T09:16:50.1785897Z maml_omniglot PASS 2024-06-26T09:16:50.1789626Z mnasnet1_0 PASS 2024-06-26T09:16:50.1794819Z mobilenet_v2 PASS 2024-06-26T09:16:50.1800000Z mobilenet_v2_quantized_qat PASS 2024-06-26T09:16:50.1805766Z mobilenet_v3_large PASS 2024-06-26T09:16:50.1811112Z moco PASS 2024-06-26T09:16:50.1816395Z nanogpt PASS 2024-06-26T09:16:50.1821932Z nvidia_deeprecommender PASS 2024-06-26T09:16:50.1827138Z opacus_cifar10 PASS 2024-06-26T09:16:50.1831835Z phlippe_densenet PASS 2024-06-26T09:16:50.1836425Z phlippe_resnet PASS 2024-06-26T09:16:50.1841230Z pytorch_CycleGAN_and_pix2pix PASS 2024-06-26T09:16:50.1846022Z pytorch_stargan PASS 2024-06-26T09:16:50.1850760Z pytorch_unet PASS 2024-06-26T09:16:50.1855493Z resnet152 PASS 2024-06-26T09:16:50.1860308Z resnet18 PASS 2024-06-26T09:16:50.1865406Z resnet50 PASS 2024-06-26T09:16:50.1870075Z resnet50_quantized_qat PASS 2024-06-26T09:16:50.1874937Z resnext50_32x4d PASS 2024-06-26T09:16:50.1879851Z sam PASS 2024-06-26T09:16:50.1884719Z shufflenet_v2_x1_0 PASS 2024-06-26T09:16:50.1889253Z soft_actor_critic PASS 2024-06-26T09:16:50.1893973Z squeezenet1_1 PASS 2024-06-26T09:16:50.1898841Z stable_diffusion_text_encoder PASS 2024-06-26T09:16:50.1903737Z stable_diffusion_unet PASS 2024-06-26T09:16:50.1908377Z timm_efficientnet PASS 2024-06-26T09:16:50.1913462Z timm_regnet PASS 2024-06-26T09:16:50.1918112Z timm_resnest PASS 2024-06-26T09:16:50.1922783Z timm_vision_transformer PASS 2024-06-26T09:16:50.1927560Z timm_vision_transformer_large PASS 2024-06-26T09:16:50.1932120Z timm_vovnet PASS 2024-06-26T09:16:50.1936917Z torch_multimodal_clip PASS 2024-06-26T09:16:50.1941753Z tts_angular PASS 2024-06-26T09:16:50.1946722Z vgg16 PASS 2024-06-26T09:16:50.1951307Z vision_maskrcnn PASS 2024-06-26T09:16:50.1956519Z yolov3 PASS 2024-06-26T09:16:50.2419736Z + cleanup_workspace 2024-06-26T09:16:50.2420803Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T09:16:50.2422007Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T09:16:50.2422918Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T09:16:50.2423556Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T09:16:50.2424346Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T09:16:50.2425202Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T09:16:50.2425883Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T09:16:50.7806971Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T09:16:50.7807559Z cat test/**/*_toprint.log || true 2024-06-26T09:16:50.7815649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:50.7816185Z env: 2024-06-26T09:16:50.7816467Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:50.7816915Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:50.7817777Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:50.7818440Z ##[endgroup] 2024-06-26T09:16:50.7881402Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T09:16:50.7934415Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:16:50.7934895Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T09:16:50.7942755Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:50.7943249Z env: 2024-06-26T09:16:50.7943532Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:50.7943981Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:50.7944715Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:50.7945373Z MONITOR_SCRIPT_PID: 30348 2024-06-26T09:16:50.7945726Z ##[endgroup] 2024-06-26T09:16:50.8128570Z Prepare all required actions 2024-06-26T09:16:50.8129016Z Getting action download info 2024-06-26T09:16:50.9839402Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T09:16:51.1416285Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T09:16:51.1416758Z with: 2024-06-26T09:16:51.1417325Z file-suffix: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112 2024-06-26T09:16:51.1418011Z s3-bucket: gha-artifacts 2024-06-26T09:16:51.1418337Z env: 2024-06-26T09:16:51.1418608Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.1419056Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.1419787Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.1420422Z ##[endgroup] 2024-06-26T09:16:51.1445179Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T09:16:51.1445831Z # Remove any previous test jsons if they exist 2024-06-26T09:16:51.1446340Z rm -f test-jsons-*.zip 2024-06-26T09:16:51.1446856Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T09:16:51.1454625Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:51.1455121Z env: 2024-06-26T09:16:51.1455399Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.1455839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.1456574Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.1457506Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112 2024-06-26T09:16:51.1458185Z ##[endgroup] 2024-06-26T09:16:51.1624676Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T09:16:51.1653862Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T09:16:51.1654535Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T09:16:51.1660947Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T09:16:51.1685828Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T09:16:51.1686498Z # Remove any previous test reports if they exist 2024-06-26T09:16:51.1687033Z rm -f test-reports-*.zip 2024-06-26T09:16:51.1687631Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T09:16:51.1695264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:51.1695758Z env: 2024-06-26T09:16:51.1696037Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.1696487Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.1697279Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.1698207Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112 2024-06-26T09:16:51.1698886Z ##[endgroup] 2024-06-26T09:16:51.1868336Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-06-26T09:16:51.1870151Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 93%) 2024-06-26T09:16:51.1871252Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 74%) 2024-06-26T09:16:51.1872207Z adding: test/test-reports/training_torchbench.csv (deflated 66%) 2024-06-26T09:16:51.1878150Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-06-26T09:16:51.1879240Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 80%) 2024-06-26T09:16:51.1905179Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T09:16:51.1905811Z # Remove any previous usage logs if they exist 2024-06-26T09:16:51.1906305Z rm -f logs-*.zip 2024-06-26T09:16:51.1906971Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T09:16:51.1907865Z # so check to see if the file exists first 2024-06-26T09:16:51.1908371Z if [ -f 'usage_log.txt' ]; then 2024-06-26T09:16:51.1908887Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T09:16:51.1909390Z fi 2024-06-26T09:16:51.1909757Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T09:16:51.1910335Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T09:16:51.1910822Z fi 2024-06-26T09:16:51.1918253Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:51.1918747Z env: 2024-06-26T09:16:51.1919021Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.1919463Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.1920195Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.1921117Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112 2024-06-26T09:16:51.1921788Z ##[endgroup] 2024-06-26T09:16:51.2010482Z adding: usage_log.txt (deflated 92%) 2024-06-26T09:16:51.2080104Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T09:16:51.2080843Z # Remove any previous debugging artifacts if they exist 2024-06-26T09:16:51.2081401Z rm -f debug-*.zip 2024-06-26T09:16:51.2081790Z if [ -d 'test/debug' ]; then 2024-06-26T09:16:51.2082294Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T09:16:51.2082779Z fi 2024-06-26T09:16:51.2089692Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:51.2090195Z env: 2024-06-26T09:16:51.2090472Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.2090927Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.2091666Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.2092602Z FILE_SUFFIX: test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112 2024-06-26T09:16:51.2093290Z ##[endgroup] 2024-06-26T09:16:51.2165787Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:16:51.2166224Z with: 2024-06-26T09:16:51.2166523Z s3-bucket: gha-artifacts 2024-06-26T09:16:51.2166960Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:51.2167438Z retention-days: 14 2024-06-26T09:16:51.2167768Z if-no-files-found: warn 2024-06-26T09:16:51.2168131Z path: test-jsons-*.zip 2024-06-26T09:16:51.2168476Z name: artifact 2024-06-26T09:16:51.2168784Z region: us-east-1 2024-06-26T09:16:51.2169080Z env: 2024-06-26T09:16:51.2169366Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.2169823Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.2170573Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.2171239Z ##[endgroup] 2024-06-26T09:16:51.5203462Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:16:51.5204294Z With the provided path, there will be 1 file uploaded 2024-06-26T09:16:51.5204941Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:51.5229908Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:51.6713475Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:51.6854766Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:16:51.6855200Z with: 2024-06-26T09:16:51.6855478Z s3-bucket: gha-artifacts 2024-06-26T09:16:51.6855897Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:51.6856355Z retention-days: 14 2024-06-26T09:16:51.6856707Z if-no-files-found: error 2024-06-26T09:16:51.6857091Z path: test-reports-*.zip 2024-06-26T09:16:51.6857431Z name: artifact 2024-06-26T09:16:51.6857721Z region: us-east-1 2024-06-26T09:16:51.6858024Z env: 2024-06-26T09:16:51.6858297Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:51.6858862Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:51.6859603Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:51.6860263Z ##[endgroup] 2024-06-26T09:16:51.9939889Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:16:51.9940510Z With the provided path, there will be 1 file uploaded 2024-06-26T09:16:51.9941213Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:51.9966745Z Starting upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:52.1404115Z Finished upload of test-reports-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:52.1543526Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:16:52.1543959Z with: 2024-06-26T09:16:52.1544241Z s3-bucket: gha-artifacts 2024-06-26T09:16:52.1544662Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:52.1545135Z retention-days: 14 2024-06-26T09:16:52.1545474Z if-no-files-found: ignore 2024-06-26T09:16:52.1545827Z path: logs-*.zip 2024-06-26T09:16:52.1546130Z name: artifact 2024-06-26T09:16:52.1546435Z region: us-east-1 2024-06-26T09:16:52.1546732Z env: 2024-06-26T09:16:52.1547006Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:52.1547458Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:52.1548196Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:52.1548848Z ##[endgroup] 2024-06-26T09:16:52.4594069Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T09:16:52.4594913Z With the provided path, there will be 1 file uploaded 2024-06-26T09:16:52.4595633Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:52.4620367Z Starting upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:52.5535393Z Finished upload of logs-test-dynamic_inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_26688912112.zip 2024-06-26T09:16:52.5677039Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T09:16:52.5677478Z with: 2024-06-26T09:16:52.5677773Z s3-bucket: gha-artifacts 2024-06-26T09:16:52.5678193Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T09:16:52.5678655Z retention-days: 14 2024-06-26T09:16:52.5678982Z if-no-files-found: ignore 2024-06-26T09:16:52.5679337Z path: debug-*.zip 2024-06-26T09:16:52.5679647Z name: artifact 2024-06-26T09:16:52.5679942Z region: us-east-1 2024-06-26T09:16:52.5680237Z env: 2024-06-26T09:16:52.5680515Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:52.5680968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:52.5681716Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:52.5682370Z ##[endgroup] 2024-06-26T09:16:52.8689490Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T09:16:52.8886906Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T09:16:52.8887466Z # shellcheck disable=SC2156 2024-06-26T09:16:52.8888308Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T09:16:52.8896341Z shell: /usr/bin/bash -e {0} 2024-06-26T09:16:52.8896691Z env: 2024-06-26T09:16:52.8896976Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:52.8897431Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:52.8898167Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:52.8898820Z ##[endgroup] 2024-06-26T09:16:53.0554817Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T09:16:53.0555361Z with: 2024-06-26T09:16:53.0555617Z env: 2024-06-26T09:16:53.0555894Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:53.0556342Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:53.0557082Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:53.0557779Z ##[endgroup] 2024-06-26T09:16:53.0574819Z ##[group]Run set -eou pipefail 2024-06-26T09:16:53.0575235Z set -eou pipefail 2024-06-26T09:16:53.0575582Z  2024-06-26T09:16:53.0576093Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T09:16:53.0576720Z for _ in $(seq 1440); do 2024-06-26T09:16:53.0577195Z  # Break if no ssh session exists anymore 2024-06-26T09:16:53.0577696Z  if [ "$(who)" = "" ]; then 2024-06-26T09:16:53.0578104Z  break 2024-06-26T09:16:53.0578437Z  fi 2024-06-26T09:16:53.0578733Z  echo "." 2024-06-26T09:16:53.0579059Z  sleep 5 2024-06-26T09:16:53.0579376Z done 2024-06-26T09:16:53.0586805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:53.0587317Z env: 2024-06-26T09:16:53.0587634Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:53.0588082Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:53.0588827Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:53.0589478Z ##[endgroup] 2024-06-26T09:16:53.0609768Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T09:16:53.0646278Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:16:53.0647052Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T09:16:53.0647673Z # shellcheck disable=SC2046 2024-06-26T09:16:53.0648167Z docker stop $(docker ps -q) || true 2024-06-26T09:16:53.0648647Z # Prune all of the docker images 2024-06-26T09:16:53.0649097Z docker system prune -af 2024-06-26T09:16:53.0655903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:16:53.0656418Z env: 2024-06-26T09:16:53.0656699Z GIT_DEFAULT_BRANCH: main 2024-06-26T09:16:53.0657151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T09:16:53.0657916Z DOCKER_CONTAINER_ID: 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:53.0658577Z ##[endgroup] 2024-06-26T09:16:53.4236911Z 068cf6bbc92f 2024-06-26T09:16:55.7421615Z Deleted Containers: 2024-06-26T09:16:55.7422183Z 068cf6bbc92f1a6d964ebd0add60c2d5db0c04fe3b90a0407b728c3b93165352 2024-06-26T09:16:55.7422632Z 2024-06-26T09:16:59.6740406Z Deleted Images: 2024-06-26T09:16:59.6742393Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T09:16:59.6745416Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T09:16:59.6747330Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T09:16:59.6748152Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T09:16:59.6748972Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T09:16:59.6749995Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T09:16:59.6750849Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T09:16:59.6751981Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T09:16:59.6752802Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T09:16:59.6753722Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T09:16:59.6754526Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T09:16:59.6755316Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T09:16:59.6756099Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T09:16:59.6756901Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T09:16:59.6757897Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T09:16:59.6758754Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T09:16:59.6759558Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T09:16:59.6760363Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T09:16:59.6761180Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T09:16:59.6761983Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T09:16:59.6762796Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T09:16:59.6763598Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T09:16:59.6764394Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T09:16:59.6765194Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T09:16:59.6765992Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T09:16:59.6766791Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T09:16:59.6767611Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T09:16:59.6768456Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T09:16:59.6769279Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T09:16:59.6770097Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T09:16:59.6770902Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T09:16:59.6771694Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T09:16:59.6772481Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T09:16:59.6773297Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T09:16:59.6774099Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T09:16:59.6774902Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T09:16:59.6775709Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T09:16:59.6776528Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T09:16:59.6777318Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T09:16:59.6778115Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T09:16:59.6778921Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T09:16:59.6779725Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T09:16:59.6780629Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T09:16:59.6781574Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T09:16:59.6782371Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T09:16:59.6783185Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T09:16:59.6784085Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T09:16:59.6784900Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T09:16:59.6785706Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T09:16:59.6786506Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T09:16:59.6787305Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T09:16:59.6788114Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T09:16:59.6788931Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T09:16:59.6789747Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T09:16:59.6790558Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T09:16:59.6791370Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T09:16:59.6792404Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T09:16:59.6793211Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T09:16:59.6794008Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T09:16:59.6794813Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T09:16:59.6795620Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T09:16:59.6796432Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T09:16:59.6797253Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T09:16:59.6798063Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T09:16:59.6798865Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T09:16:59.6799675Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T09:16:59.6800465Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T09:16:59.6801255Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T09:16:59.6802059Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T09:16:59.6802856Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T09:16:59.6803656Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T09:16:59.6804463Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T09:16:59.6805244Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T09:16:59.6806046Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T09:16:59.6806865Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T09:16:59.6807690Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T09:16:59.6808502Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T09:16:59.6809326Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T09:16:59.6810152Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T09:16:59.6811067Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T09:16:59.6811872Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T09:16:59.6812358Z 2024-06-26T09:16:59.6812504Z Total reclaimed space: 41.17GB 2024-06-26T09:16:59.6945315Z Post job cleanup. 2024-06-26T09:16:59.6993968Z Post job cleanup. 2024-06-26T09:16:59.7794311Z [command]/usr/bin/git version 2024-06-26T09:16:59.7835056Z git version 2.40.1 2024-06-26T09:16:59.7875432Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e2d67ad8-ee15-4f7d-8c6b-e2a09297dc0d' before making global git config changes 2024-06-26T09:16:59.7876668Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T09:16:59.7881389Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T09:16:59.7912847Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T09:16:59.7934530Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T09:16:59.8149854Z Entering 'android/libs/fbjni' 2024-06-26T09:16:59.8186072Z Entering 'third_party/FP16' 2024-06-26T09:16:59.8220690Z Entering 'third_party/FXdiv' 2024-06-26T09:16:59.8258632Z Entering 'third_party/NNPACK' 2024-06-26T09:16:59.8295143Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:16:59.8331134Z Entering 'third_party/XNNPACK' 2024-06-26T09:16:59.8378591Z Entering 'third_party/benchmark' 2024-06-26T09:16:59.8416140Z Entering 'third_party/cpp-httplib' 2024-06-26T09:16:59.8450988Z Entering 'third_party/cpuinfo' 2024-06-26T09:16:59.8487586Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:16:59.8523780Z Entering 'third_party/cutlass' 2024-06-26T09:16:59.8566633Z Entering 'third_party/eigen' 2024-06-26T09:16:59.8604287Z Entering 'third_party/fbgemm' 2024-06-26T09:16:59.8640486Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:16:59.8674122Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:16:59.8709130Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:16:59.8748119Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:16:59.8781957Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:16:59.8818014Z Entering 'third_party/flatbuffers' 2024-06-26T09:16:59.8857714Z Entering 'third_party/fmt' 2024-06-26T09:16:59.8890700Z Entering 'third_party/foxi' 2024-06-26T09:16:59.8926272Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:16:59.8962464Z Entering 'third_party/gloo' 2024-06-26T09:16:59.8996138Z Entering 'third_party/googletest' 2024-06-26T09:16:59.9032175Z Entering 'third_party/ideep' 2024-06-26T09:16:59.9065569Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:16:59.9105744Z Entering 'third_party/ittapi' 2024-06-26T09:16:59.9137275Z Entering 'third_party/kineto' 2024-06-26T09:16:59.9169607Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:16:59.9201913Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:16:59.9238279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:16:59.9272130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:16:59.9305305Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:16:59.9337508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:16:59.9376188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:16:59.9413995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:16:59.9451407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:16:59.9484014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:16:59.9520552Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:16:59.9553768Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:16:59.9586314Z Entering 'third_party/mimalloc' 2024-06-26T09:16:59.9618569Z Entering 'third_party/nccl/nccl' 2024-06-26T09:16:59.9649833Z Entering 'third_party/nlohmann' 2024-06-26T09:16:59.9682358Z Entering 'third_party/onnx' 2024-06-26T09:16:59.9726535Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:16:59.9759270Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:16:59.9791674Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:16:59.9825204Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:16:59.9857090Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:16:59.9888524Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:16:59.9920314Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:16:59.9955934Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:16:59.9988522Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:17:00.0021019Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:17:00.0053192Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:17:00.0085789Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:17:00.0118590Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:17:00.0166340Z Entering 'third_party/pocketfft' 2024-06-26T09:17:00.0198912Z Entering 'third_party/protobuf' 2024-06-26T09:17:00.0232326Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:17:00.0264008Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:17:00.0302965Z Entering 'third_party/psimd' 2024-06-26T09:17:00.0337153Z Entering 'third_party/pthreadpool' 2024-06-26T09:17:00.0369564Z Entering 'third_party/pybind11' 2024-06-26T09:17:00.0402222Z Entering 'third_party/python-peachpy' 2024-06-26T09:17:00.0434630Z Entering 'third_party/sleef' 2024-06-26T09:17:00.0466599Z Entering 'third_party/tensorpipe' 2024-06-26T09:17:00.0498971Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:17:00.0530911Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:17:00.0561653Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:17:00.0600842Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:17:00.0633464Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:17:00.0684927Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T09:17:00.0702822Z http.https://github.com/.extraheader 2024-06-26T09:17:00.0710424Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T09:17:00.0734705Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T09:17:00.0939912Z Entering 'android/libs/fbjni' 2024-06-26T09:17:00.0960596Z http.https://github.com/.extraheader 2024-06-26T09:17:00.0982089Z Entering 'third_party/FP16' 2024-06-26T09:17:00.1005202Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1026359Z Entering 'third_party/FXdiv' 2024-06-26T09:17:00.1050263Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1073095Z Entering 'third_party/NNPACK' 2024-06-26T09:17:00.1092615Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1115101Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T09:17:00.1135600Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1157687Z Entering 'third_party/XNNPACK' 2024-06-26T09:17:00.1178977Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1212997Z Entering 'third_party/benchmark' 2024-06-26T09:17:00.1234037Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1256602Z Entering 'third_party/cpp-httplib' 2024-06-26T09:17:00.1277788Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1303005Z Entering 'third_party/cpuinfo' 2024-06-26T09:17:00.1324934Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1348663Z Entering 'third_party/cudnn_frontend' 2024-06-26T09:17:00.1369739Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1390795Z Entering 'third_party/cutlass' 2024-06-26T09:17:00.1412317Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1439715Z Entering 'third_party/eigen' 2024-06-26T09:17:00.1460436Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1483980Z Entering 'third_party/fbgemm' 2024-06-26T09:17:00.1504713Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1528344Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T09:17:00.1549081Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1572391Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T09:17:00.1593887Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1616986Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T09:17:00.1637840Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1663334Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T09:17:00.1685072Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1707929Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T09:17:00.1728127Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1749275Z Entering 'third_party/flatbuffers' 2024-06-26T09:17:00.1771855Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1796845Z Entering 'third_party/fmt' 2024-06-26T09:17:00.1818603Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1842438Z Entering 'third_party/foxi' 2024-06-26T09:17:00.1863578Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1886846Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T09:17:00.1908036Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1932119Z Entering 'third_party/gloo' 2024-06-26T09:17:00.1955263Z http.https://github.com/.extraheader 2024-06-26T09:17:00.1979625Z Entering 'third_party/googletest' 2024-06-26T09:17:00.2002062Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2024927Z Entering 'third_party/ideep' 2024-06-26T09:17:00.2045939Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2067754Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T09:17:00.2089321Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2120310Z Entering 'third_party/ittapi' 2024-06-26T09:17:00.2140917Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2163508Z Entering 'third_party/kineto' 2024-06-26T09:17:00.2184779Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2208387Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T09:17:00.2229956Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2254024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T09:17:00.2275732Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2299972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T09:17:00.2321915Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2345473Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T09:17:00.2366323Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2389375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T09:17:00.2410539Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2432543Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T09:17:00.2453792Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2478012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T09:17:00.2499081Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2523076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T09:17:00.2543293Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2565703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T09:17:00.2586935Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2610875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T09:17:00.2631076Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2657111Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T09:17:00.2678780Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2701760Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T09:17:00.2723974Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2748047Z Entering 'third_party/mimalloc' 2024-06-26T09:17:00.2770936Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2794626Z Entering 'third_party/nccl/nccl' 2024-06-26T09:17:00.2816757Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2838270Z Entering 'third_party/nlohmann' 2024-06-26T09:17:00.2860061Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2883754Z Entering 'third_party/onnx' 2024-06-26T09:17:00.2905071Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2939163Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T09:17:00.2959953Z http.https://github.com/.extraheader 2024-06-26T09:17:00.2983418Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T09:17:00.3004459Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3030170Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T09:17:00.3051160Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3076328Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T09:17:00.3097550Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3120242Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T09:17:00.3140896Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3164150Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T09:17:00.3185117Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3208203Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T09:17:00.3228155Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3250456Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T09:17:00.3271142Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3293460Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T09:17:00.3315071Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3337125Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T09:17:00.3357574Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3380298Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T09:17:00.3401734Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3426650Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T09:17:00.3448628Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3473326Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T09:17:00.3494254Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3530334Z Entering 'third_party/pocketfft' 2024-06-26T09:17:00.3552009Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3575969Z Entering 'third_party/protobuf' 2024-06-26T09:17:00.3597738Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3624488Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T09:17:00.3645757Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3669483Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T09:17:00.3691132Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3714904Z Entering 'third_party/psimd' 2024-06-26T09:17:00.3737218Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3759350Z Entering 'third_party/pthreadpool' 2024-06-26T09:17:00.3779902Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3804236Z Entering 'third_party/pybind11' 2024-06-26T09:17:00.3825851Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3849292Z Entering 'third_party/python-peachpy' 2024-06-26T09:17:00.3871163Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3893911Z Entering 'third_party/sleef' 2024-06-26T09:17:00.3915842Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3938195Z Entering 'third_party/tensorpipe' 2024-06-26T09:17:00.3960072Z http.https://github.com/.extraheader 2024-06-26T09:17:00.3983704Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T09:17:00.4005077Z http.https://github.com/.extraheader 2024-06-26T09:17:00.4028165Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T09:17:00.4048856Z http.https://github.com/.extraheader 2024-06-26T09:17:00.4071940Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T09:17:00.4094541Z http.https://github.com/.extraheader 2024-06-26T09:17:00.4118342Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T09:17:00.4139470Z http.https://github.com/.extraheader 2024-06-26T09:17:00.4162286Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T09:17:00.4183578Z http.https://github.com/.extraheader 2024-06-26T09:17:00.4298723Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T09:17:00.4316312Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T09:17:00.4322903Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T09:17:00.4323418Z ##[endgroup] 2024-06-26T09:17:00.9621975Z Cleaning up orphan processes