2024-08-20T22:04:30.0366793Z Current runner version: '2.319.1'
2024-08-20T22:04:30.0373450Z Runner name: 'i-0e9f01dbcd5a47485'
2024-08-20T22:04:30.0374263Z Runner group name: 'Default'
2024-08-20T22:04:30.0375329Z Machine name: 'ip-10-0-5-191'
2024-08-20T22:04:30.0391677Z Testing runner upgrade compatibility
2024-08-20T22:04:30.1510317Z ##[group]GITHUB_TOKEN Permissions
2024-08-20T22:04:30.1512458Z Actions: read
2024-08-20T22:04:30.1513017Z Attestations: read
2024-08-20T22:04:30.1513597Z Checks: read
2024-08-20T22:04:30.1514155Z Contents: read
2024-08-20T22:04:30.1514702Z Deployments: read
2024-08-20T22:04:30.1515290Z Discussions: read
2024-08-20T22:04:30.1515797Z Issues: read
2024-08-20T22:04:30.1516292Z Metadata: read
2024-08-20T22:04:30.1516858Z Packages: read
2024-08-20T22:04:30.1517360Z Pages: read
2024-08-20T22:04:30.1517865Z PullRequests: read
2024-08-20T22:04:30.1518474Z RepositoryProjects: read
2024-08-20T22:04:30.1519099Z SecurityEvents: read
2024-08-20T22:04:30.1519605Z Statuses: read
2024-08-20T22:04:30.1520469Z ##[endgroup]
2024-08-20T22:04:30.1523810Z Secret source: Actions
2024-08-20T22:04:30.1524737Z Prepare workflow directory
2024-08-20T22:04:30.2619011Z Prepare all required actions
2024-08-20T22:04:30.2794530Z Getting action download info
2024-08-20T22:04:30.4472419Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f)
2024-08-20T22:04:30.9209881Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2)
2024-08-20T22:04:34.9696282Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-08-20T22:04:35.1591243Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-08-20T22:04:35.5034661Z Getting action download info
2024-08-20T22:04:35.5938557Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-08-20T22:04:35.7962821Z Getting action download info
2024-08-20T22:04:35.8896538Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-08-20T22:04:36.0697043Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3)
2024-08-20T22:04:36.0699185Z ##[group] Inputs
2024-08-20T22:04:36.0699656Z   build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:04:36.0706663Z   test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-08-20T22:04:36.0715919Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:04:36.0717258Z   sync-tag: 
2024-08-20T22:04:36.0718094Z   timeout-minutes: 240
2024-08-20T22:04:36.0718431Z   use-gha: 
2024-08-20T22:04:36.0718722Z   dashboard-tag: 
2024-08-20T22:04:36.0719059Z   s3-bucket: gha-artifacts
2024-08-20T22:04:36.0719413Z   aws-role-to-assume: 
2024-08-20T22:04:36.0719756Z ##[endgroup]
2024-08-20T22:04:36.0720834Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:04:36.1387246Z A job started hook has been configured by the self-hosted runner administrator
2024-08-20T22:04:36.1526164Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2024-08-20T22:04:36.1538209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:04:36.1538756Z ##[endgroup]
2024-08-20T22:04:38.2731129Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2024-08-20T22:04:38.2731650Z Instance Type: g5.4xlarge
2024-08-20T22:04:38.2732361Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64
2024-08-20T22:04:38.2732916Z AMI ID: ami-06c68f701d8090592
2024-08-20T22:04:44.0873652Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2024-08-20T22:04:44.0874212Z with:
2024-08-20T22:04:44.0874911Z   github-secret: ***
2024-08-20T22:04:44.0875857Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-08-20T22:04:44.0876846Z   activate-with-label: false
2024-08-20T22:04:44.0877215Z   label: with-ssh
2024-08-20T22:04:44.0877542Z   remove-existing-keys: true
2024-08-20T22:04:44.0877907Z   fail-silently: true
2024-08-20T22:04:44.0878231Z env:
2024-08-20T22:04:44.0878536Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:04:44.0878921Z ##[endgroup]
2024-08-20T22:04:44.1771667Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-08-20T22:04:44.1774001Z ciflow reference detected, attempting to extract PR number
2024-08-20T22:04:44.5611135Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2024-08-20T22:04:44.6444845Z No SSH keys found for user pytorch-bot[bot]
2024-08-20T22:04:44.6445459Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys
2024-08-20T22:04:44.7115240Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2024-08-20T22:04:44.7129767Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2024-08-20T22:04:44.7153675Z Login using: ssh ec2-user@ec2-98-80-233-117.compute-1.amazonaws.com
2024-08-20T22:04:44.7154458Z All testing is done inside the container, to start an interactive session run:
2024-08-20T22:04:44.7155272Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2024-08-20T22:04:44.7286058Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2024-08-20T22:04:44.7286616Z with:
2024-08-20T22:04:44.7286902Z   submodules: recursive
2024-08-20T22:04:44.7287237Z   fetch-depth: 0
2024-08-20T22:04:44.7287535Z env:
2024-08-20T22:04:44.7287814Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:04:44.7288156Z ##[endgroup]
2024-08-20T22:04:44.7476358Z ##[group]Run retry () {
2024-08-20T22:04:44.7476743Z retry () {
2024-08-20T22:04:44.7477254Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2024-08-20T22:04:44.7477828Z }
2024-08-20T22:04:44.7478154Z echo "${GITHUB_WORKSPACE}"
2024-08-20T22:04:44.7478581Z if [ -z "${NO_SUDO}" ]; then
2024-08-20T22:04:44.7479061Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2024-08-20T22:04:44.7479516Z else
2024-08-20T22:04:44.7479859Z   retry rm -rf "${GITHUB_WORKSPACE}"
2024-08-20T22:04:44.7480285Z fi
2024-08-20T22:04:44.7480866Z mkdir "${GITHUB_WORKSPACE}"
2024-08-20T22:04:44.7490148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:04:44.7490719Z env:
2024-08-20T22:04:44.7490996Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:04:44.7491334Z   NO_SUDO: 
2024-08-20T22:04:44.7491607Z ##[endgroup]
2024-08-20T22:04:44.7524574Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:04:49.4206115Z ##[group]Run malfet/checkout@silent-checkout
2024-08-20T22:04:49.4206539Z with:
2024-08-20T22:04:49.4206867Z   ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:04:49.4207318Z   fetch-depth: 0
2024-08-20T22:04:49.4207636Z   submodules: recursive
2024-08-20T22:04:49.4207979Z   quiet-checkout: true
2024-08-20T22:04:49.4208335Z   repository: pytorch/pytorch
2024-08-20T22:04:49.4208817Z   token: ***
2024-08-20T22:04:49.4209109Z   ssh-strict: true
2024-08-20T22:04:49.4209433Z   persist-credentials: true
2024-08-20T22:04:49.4209789Z   clean: true
2024-08-20T22:04:49.4210132Z   sparse-checkout-cone-mode: true
2024-08-20T22:04:49.4210521Z   lfs: false
2024-08-20T22:04:49.4210910Z   set-safe-directory: true
2024-08-20T22:04:49.4211247Z env:
2024-08-20T22:04:49.4211519Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:04:49.4211848Z ##[endgroup]
2024-08-20T22:04:49.5190254Z Syncing repository: pytorch/pytorch
2024-08-20T22:04:49.5191761Z ##[group]Getting Git version info
2024-08-20T22:04:49.5192516Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-08-20T22:04:49.5193370Z [command]/usr/bin/git version
2024-08-20T22:04:49.5193751Z git version 2.40.1
2024-08-20T22:04:49.5198604Z ##[endgroup]
2024-08-20T22:04:49.5212321Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9cb98cb3-6b72-42e1-83b1-2bc0100632d0' before making global git config changes
2024-08-20T22:04:49.5213545Z Adding repository directory to the temporary git global config as a safe directory
2024-08-20T22:04:49.5217555Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:04:49.5270583Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-08-20T22:04:49.5273064Z ##[group]Initializing the repository
2024-08-20T22:04:49.5275847Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:04:49.5321035Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-08-20T22:04:49.5321946Z hint: is subject to change. To configure the initial branch name to use in all
2024-08-20T22:04:49.5322830Z hint: of your new repositories, which will suppress this warning, call:
2024-08-20T22:04:49.5323394Z hint: 
2024-08-20T22:04:49.5323872Z hint: 	git config --global init.defaultBranch <name>
2024-08-20T22:04:49.5324344Z hint: 
2024-08-20T22:04:49.5324864Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-08-20T22:04:49.5325718Z hint: 'development'. The just-created branch can be renamed via this command:
2024-08-20T22:04:49.5326612Z hint: 
2024-08-20T22:04:49.5326954Z hint: 	git branch -m <name>
2024-08-20T22:04:49.5327695Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-08-20T22:04:49.5335137Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-08-20T22:04:49.5378842Z ##[endgroup]
2024-08-20T22:04:49.5379420Z ##[group]Disabling automatic garbage collection
2024-08-20T22:04:49.5381577Z [command]/usr/bin/git config --local gc.auto 0
2024-08-20T22:04:49.5425496Z ##[endgroup]
2024-08-20T22:04:49.5426028Z ##[group]Setting up auth
2024-08-20T22:04:49.5431242Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-08-20T22:04:49.5474980Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-08-20T22:04:49.5824363Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-08-20T22:04:49.5866037Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-08-20T22:04:49.6219668Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-08-20T22:04:49.6277071Z ##[endgroup]
2024-08-20T22:04:49.6277632Z ##[group]Fetching the repository
2024-08-20T22:04:49.6283143Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-08-20T22:04:52.1848081Z remote: Enumerating objects: 1008452        
2024-08-20T22:04:52.1848654Z remote: Enumerating objects: 1010199, done.        
2024-08-20T22:04:52.1849332Z remote: Counting objects:   0% (1/1747)        
2024-08-20T22:04:52.1849838Z remote: Counting objects:   1% (18/1747)        
2024-08-20T22:04:52.1850967Z remote: Counting objects:   2% (35/1747)        
2024-08-20T22:04:52.1851469Z remote: Counting objects:   3% (53/1747)        
2024-08-20T22:04:52.1852512Z remote: Counting objects:   4% (70/1747)        
2024-08-20T22:04:52.1853069Z remote: Counting objects:   5% (88/1747)        
2024-08-20T22:04:52.1853878Z remote: Counting objects:   6% (105/1747)        
2024-08-20T22:04:52.1854501Z remote: Counting objects:   7% (123/1747)        
2024-08-20T22:04:52.1855280Z remote: Counting objects:   8% (140/1747)        
2024-08-20T22:04:52.1856375Z remote: Counting objects:   9% (158/1747)        
2024-08-20T22:04:52.1857339Z remote: Counting objects:  10% (175/1747)        
2024-08-20T22:04:52.1857835Z remote: Counting objects:  11% (193/1747)        
2024-08-20T22:04:52.1858327Z remote: Counting objects:  12% (210/1747)        
2024-08-20T22:04:52.1858823Z remote: Counting objects:  13% (228/1747)        
2024-08-20T22:04:52.1859451Z remote: Counting objects:  14% (245/1747)        
2024-08-20T22:04:52.1860011Z remote: Counting objects:  15% (263/1747)        
2024-08-20T22:04:52.1860515Z remote: Counting objects:  16% (280/1747)        
2024-08-20T22:04:52.1861001Z remote: Counting objects:  17% (297/1747)        
2024-08-20T22:04:52.1861594Z remote: Counting objects:  18% (315/1747)        
2024-08-20T22:04:52.1862219Z remote: Counting objects:  19% (332/1747)        
2024-08-20T22:04:52.1862886Z remote: Counting objects:  20% (350/1747)        
2024-08-20T22:04:52.1863453Z remote: Counting objects:  21% (367/1747)        
2024-08-20T22:04:52.1864066Z remote: Counting objects:  22% (385/1747)        
2024-08-20T22:04:52.1864732Z remote: Counting objects:  23% (402/1747)        
2024-08-20T22:04:52.1865387Z remote: Counting objects:  24% (420/1747)        
2024-08-20T22:04:52.1866036Z remote: Counting objects:  25% (437/1747)        
2024-08-20T22:04:52.1866684Z remote: Counting objects:  26% (455/1747)        
2024-08-20T22:04:52.1867469Z remote: Counting objects:  27% (472/1747)        
2024-08-20T22:04:52.1868552Z remote: Counting objects:  28% (490/1747)        
2024-08-20T22:04:52.1869166Z remote: Counting objects:  29% (507/1747)        
2024-08-20T22:04:52.1869657Z remote: Counting objects:  30% (525/1747)        
2024-08-20T22:04:52.1870141Z remote: Counting objects:  31% (542/1747)        
2024-08-20T22:04:52.1870643Z remote: Counting objects:  32% (560/1747)        
2024-08-20T22:04:52.1871285Z remote: Counting objects:  33% (577/1747)        
2024-08-20T22:04:52.1871937Z remote: Counting objects:  34% (594/1747)        
2024-08-20T22:04:52.1872467Z remote: Counting objects:  35% (612/1747)        
2024-08-20T22:04:52.1873095Z remote: Counting objects:  36% (629/1747)        
2024-08-20T22:04:52.1873749Z remote: Counting objects:  37% (647/1747)        
2024-08-20T22:04:52.1874382Z remote: Counting objects:  38% (664/1747)        
2024-08-20T22:04:52.1875023Z remote: Counting objects:  39% (682/1747)        
2024-08-20T22:04:52.1875639Z remote: Counting objects:  40% (699/1747)        
2024-08-20T22:04:52.1876378Z remote: Counting objects:  41% (717/1747)        
2024-08-20T22:04:52.1876866Z remote: Counting objects:  42% (734/1747)        
2024-08-20T22:04:52.1877358Z remote: Counting objects:  43% (752/1747)        
2024-08-20T22:04:52.1877839Z remote: Counting objects:  44% (769/1747)        
2024-08-20T22:04:52.1878313Z remote: Counting objects:  45% (787/1747)        
2024-08-20T22:04:52.1878800Z remote: Counting objects:  46% (804/1747)        
2024-08-20T22:04:52.1879293Z remote: Counting objects:  47% (822/1747)        
2024-08-20T22:04:52.1879779Z remote: Counting objects:  48% (839/1747)        
2024-08-20T22:04:52.1880301Z remote: Counting objects:  49% (857/1747)        
2024-08-20T22:04:52.1880789Z remote: Counting objects:  50% (874/1747)        
2024-08-20T22:04:52.1881270Z remote: Counting objects:  51% (891/1747)        
2024-08-20T22:04:52.1881755Z remote: Counting objects:  52% (909/1747)        
2024-08-20T22:04:52.1882230Z remote: Counting objects:  53% (926/1747)        
2024-08-20T22:04:52.1882725Z remote: Counting objects:  54% (944/1747)        
2024-08-20T22:04:52.1883204Z remote: Counting objects:  55% (961/1747)        
2024-08-20T22:04:52.1883683Z remote: Counting objects:  56% (979/1747)        
2024-08-20T22:04:52.1884156Z remote: Counting objects:  57% (996/1747)        
2024-08-20T22:04:52.1884651Z remote: Counting objects:  58% (1014/1747)        
2024-08-20T22:04:52.1885162Z remote: Counting objects:  59% (1031/1747)        
2024-08-20T22:04:52.1885670Z remote: Counting objects:  60% (1049/1747)        
2024-08-20T22:04:52.1886170Z remote: Counting objects:  61% (1066/1747)        
2024-08-20T22:04:52.1886669Z remote: Counting objects:  62% (1084/1747)        
2024-08-20T22:04:52.1887189Z remote: Counting objects:  63% (1101/1747)        
2024-08-20T22:04:52.1887690Z remote: Counting objects:  64% (1119/1747)        
2024-08-20T22:04:52.1888192Z remote: Counting objects:  65% (1136/1747)        
2024-08-20T22:04:52.1888688Z remote: Counting objects:  66% (1154/1747)        
2024-08-20T22:04:52.1889191Z remote: Counting objects:  67% (1171/1747)        
2024-08-20T22:04:52.1889688Z remote: Counting objects:  68% (1188/1747)        
2024-08-20T22:04:52.1890237Z remote: Counting objects:  69% (1206/1747)        
2024-08-20T22:04:52.1890862Z remote: Counting objects:  70% (1223/1747)        
2024-08-20T22:04:52.1891361Z remote: Counting objects:  71% (1241/1747)        
2024-08-20T22:04:52.1891859Z remote: Counting objects:  72% (1258/1747)        
2024-08-20T22:04:52.1892354Z remote: Counting objects:  73% (1276/1747)        
2024-08-20T22:04:52.1892846Z remote: Counting objects:  74% (1293/1747)        
2024-08-20T22:04:52.1893340Z remote: Counting objects:  75% (1311/1747)        
2024-08-20T22:04:52.1893833Z remote: Counting objects:  76% (1328/1747)        
2024-08-20T22:04:52.1894329Z remote: Counting objects:  77% (1346/1747)        
2024-08-20T22:04:52.1894818Z remote: Counting objects:  78% (1363/1747)        
2024-08-20T22:04:52.1895407Z remote: Counting objects:  79% (1381/1747)        
2024-08-20T22:04:52.1895912Z remote: Counting objects:  80% (1398/1747)        
2024-08-20T22:04:52.1896406Z remote: Counting objects:  81% (1416/1747)        
2024-08-20T22:04:52.1896895Z remote: Counting objects:  82% (1433/1747)        
2024-08-20T22:04:52.1897394Z remote: Counting objects:  83% (1451/1747)        
2024-08-20T22:04:52.1897887Z remote: Counting objects:  84% (1468/1747)        
2024-08-20T22:04:52.1898375Z remote: Counting objects:  85% (1485/1747)        
2024-08-20T22:04:52.1898870Z remote: Counting objects:  86% (1503/1747)        
2024-08-20T22:04:52.1899362Z remote: Counting objects:  87% (1520/1747)        
2024-08-20T22:04:52.1899856Z remote: Counting objects:  88% (1538/1747)        
2024-08-20T22:04:52.1900342Z remote: Counting objects:  89% (1555/1747)        
2024-08-20T22:04:52.1900835Z remote: Counting objects:  90% (1573/1747)        
2024-08-20T22:04:52.1901330Z remote: Counting objects:  91% (1590/1747)        
2024-08-20T22:04:52.1901923Z remote: Counting objects:  92% (1608/1747)        
2024-08-20T22:04:52.1902419Z remote: Counting objects:  93% (1625/1747)        
2024-08-20T22:04:52.1902912Z remote: Counting objects:  94% (1643/1747)        
2024-08-20T22:04:52.1903403Z remote: Counting objects:  95% (1660/1747)        
2024-08-20T22:04:52.1903897Z remote: Counting objects:  96% (1678/1747)        
2024-08-20T22:04:52.1904384Z remote: Counting objects:  97% (1695/1747)        
2024-08-20T22:04:52.1904875Z remote: Counting objects:  98% (1713/1747)        
2024-08-20T22:04:52.1905367Z remote: Counting objects:  99% (1730/1747)        
2024-08-20T22:04:52.1905859Z remote: Counting objects: 100% (1747/1747)        
2024-08-20T22:04:52.1906383Z remote: Counting objects: 100% (1747/1747), done.        
2024-08-20T22:04:52.2006041Z remote: Compressing objects:   0% (1/828)        
2024-08-20T22:04:52.2266529Z remote: Compressing objects:   1% (9/828)        
2024-08-20T22:04:52.2564598Z remote: Compressing objects:   2% (17/828)        
2024-08-20T22:04:52.2998641Z remote: Compressing objects:   3% (25/828)        
2024-08-20T22:04:52.3712192Z remote: Compressing objects:   4% (34/828)        
2024-08-20T22:04:52.4107514Z remote: Compressing objects:   5% (42/828)        
2024-08-20T22:04:52.4714294Z remote: Compressing objects:   6% (50/828)        
2024-08-20T22:04:52.5117031Z remote: Compressing objects:   7% (58/828)        
2024-08-20T22:04:52.5362519Z remote: Compressing objects:   8% (67/828)        
2024-08-20T22:04:52.5624579Z remote: Compressing objects:   9% (75/828)        
2024-08-20T22:04:52.5939265Z remote: Compressing objects:  10% (83/828)        
2024-08-20T22:04:52.6121907Z remote: Compressing objects:  11% (92/828)        
2024-08-20T22:04:52.6308299Z remote: Compressing objects:  12% (100/828)        
2024-08-20T22:04:52.6448814Z remote: Compressing objects:  13% (108/828)        
2024-08-20T22:04:52.6575599Z remote: Compressing objects:  14% (116/828)        
2024-08-20T22:04:52.6653857Z remote: Compressing objects:  15% (125/828)        
2024-08-20T22:04:52.6737347Z remote: Compressing objects:  16% (133/828)        
2024-08-20T22:04:52.6801553Z remote: Compressing objects:  17% (141/828)        
2024-08-20T22:04:52.6846994Z remote: Compressing objects:  18% (150/828)        
2024-08-20T22:04:52.6885813Z remote: Compressing objects:  19% (158/828)        
2024-08-20T22:04:52.6918730Z remote: Compressing objects:  20% (166/828)        
2024-08-20T22:04:52.6947376Z remote: Compressing objects:  21% (174/828)        
2024-08-20T22:04:52.6948086Z remote: Compressing objects:  22% (183/828)        
2024-08-20T22:04:52.6953587Z remote: Compressing objects:  23% (191/828)        
2024-08-20T22:04:52.6960997Z remote: Compressing objects:  24% (199/828)        
2024-08-20T22:04:52.6969654Z remote: Compressing objects:  25% (207/828)        
2024-08-20T22:04:52.6980309Z remote: Compressing objects:  26% (216/828)        
2024-08-20T22:04:52.6998412Z remote: Compressing objects:  27% (224/828)        
2024-08-20T22:04:52.7013214Z remote: Compressing objects:  28% (232/828)        
2024-08-20T22:04:52.7093161Z remote: Compressing objects:  29% (241/828)        
2024-08-20T22:04:52.7093871Z remote: Compressing objects:  30% (249/828)        
2024-08-20T22:04:52.7094565Z remote: Compressing objects:  31% (257/828)        
2024-08-20T22:04:52.7095213Z remote: Compressing objects:  32% (265/828)        
2024-08-20T22:04:52.7095739Z remote: Compressing objects:  33% (274/828)        
2024-08-20T22:04:52.7096258Z remote: Compressing objects:  34% (282/828)        
2024-08-20T22:04:52.7096768Z remote: Compressing objects:  35% (290/828)        
2024-08-20T22:04:52.7097286Z remote: Compressing objects:  36% (299/828)        
2024-08-20T22:04:52.7097807Z remote: Compressing objects:  37% (307/828)        
2024-08-20T22:04:52.7098734Z remote: Compressing objects:  38% (315/828)        
2024-08-20T22:04:52.7104629Z remote: Compressing objects:  39% (323/828)        
2024-08-20T22:04:52.7110497Z remote: Compressing objects:  40% (332/828)        
2024-08-20T22:04:52.7122562Z remote: Compressing objects:  41% (340/828)        
2024-08-20T22:04:52.7132278Z remote: Compressing objects:  42% (348/828)        
2024-08-20T22:04:52.7140182Z remote: Compressing objects:  43% (357/828)        
2024-08-20T22:04:52.7144330Z remote: Compressing objects:  44% (365/828)        
2024-08-20T22:04:52.7149421Z remote: Compressing objects:  45% (373/828)        
2024-08-20T22:04:52.7155099Z remote: Compressing objects:  46% (381/828)        
2024-08-20T22:04:52.7157687Z remote: Compressing objects:  47% (390/828)        
2024-08-20T22:04:52.7161687Z remote: Compressing objects:  48% (398/828)        
2024-08-20T22:04:52.7164539Z remote: Compressing objects:  49% (406/828)        
2024-08-20T22:04:52.7169356Z remote: Compressing objects:  50% (414/828)        
2024-08-20T22:04:52.7172746Z remote: Compressing objects:  51% (423/828)        
2024-08-20T22:04:52.7174492Z remote: Compressing objects:  52% (431/828)        
2024-08-20T22:04:52.7178964Z remote: Compressing objects:  53% (439/828)        
2024-08-20T22:04:52.7179617Z remote: Compressing objects:  54% (448/828)        
2024-08-20T22:04:52.7180883Z remote: Compressing objects:  55% (456/828)        
2024-08-20T22:04:52.7188022Z remote: Compressing objects:  56% (464/828)        
2024-08-20T22:04:52.7189989Z remote: Compressing objects:  57% (472/828)        
2024-08-20T22:04:52.7190656Z remote: Compressing objects:  58% (481/828)        
2024-08-20T22:04:52.7192242Z remote: Compressing objects:  59% (489/828)        
2024-08-20T22:04:52.7192765Z remote: Compressing objects:  60% (497/828)        
2024-08-20T22:04:52.7194388Z remote: Compressing objects:  61% (506/828)        
2024-08-20T22:04:52.7194991Z remote: Compressing objects:  62% (514/828)        
2024-08-20T22:04:52.7195945Z remote: Compressing objects:  63% (522/828)        
2024-08-20T22:04:52.7202396Z remote: Compressing objects:  64% (530/828)        
2024-08-20T22:04:52.7208447Z remote: Compressing objects:  65% (539/828)        
2024-08-20T22:04:52.7213571Z remote: Compressing objects:  66% (547/828)        
2024-08-20T22:04:52.7218735Z remote: Compressing objects:  67% (555/828)        
2024-08-20T22:04:52.7223512Z remote: Compressing objects:  68% (564/828)        
2024-08-20T22:04:52.7227094Z remote: Compressing objects:  69% (572/828)        
2024-08-20T22:04:52.7231431Z remote: Compressing objects:  70% (580/828)        
2024-08-20T22:04:52.7235485Z remote: Compressing objects:  71% (588/828)        
2024-08-20T22:04:52.7238982Z remote: Compressing objects:  72% (597/828)        
2024-08-20T22:04:52.7243059Z remote: Compressing objects:  73% (605/828)        
2024-08-20T22:04:52.7246752Z remote: Compressing objects:  74% (613/828)        
2024-08-20T22:04:52.7250256Z remote: Compressing objects:  75% (621/828)        
2024-08-20T22:04:52.7254448Z remote: Compressing objects:  76% (630/828)        
2024-08-20T22:04:52.7257619Z remote: Compressing objects:  77% (638/828)        
2024-08-20T22:04:52.7260078Z remote: Compressing objects:  78% (646/828)        
2024-08-20T22:04:52.7262043Z remote: Compressing objects:  79% (655/828)        
2024-08-20T22:04:52.7265354Z remote: Compressing objects:  80% (663/828)        
2024-08-20T22:04:52.7267645Z remote: Compressing objects:  81% (671/828)        
2024-08-20T22:04:52.7270744Z remote: Compressing objects:  82% (679/828)        
2024-08-20T22:04:52.7272420Z remote: Compressing objects:  83% (688/828)        
2024-08-20T22:04:52.7274740Z remote: Compressing objects:  84% (696/828)        
2024-08-20T22:04:52.7276499Z remote: Compressing objects:  85% (704/828)        
2024-08-20T22:04:52.7279627Z remote: Compressing objects:  86% (713/828)        
2024-08-20T22:04:52.7280775Z remote: Compressing objects:  87% (721/828)        
2024-08-20T22:04:52.7284515Z remote: Compressing objects:  88% (729/828)        
2024-08-20T22:04:52.7288267Z remote: Compressing objects:  89% (737/828)        
2024-08-20T22:04:52.7290341Z remote: Compressing objects:  90% (746/828)        
2024-08-20T22:04:52.7292551Z remote: Compressing objects:  91% (754/828)        
2024-08-20T22:04:52.7293400Z remote: Compressing objects:  92% (762/828)        
2024-08-20T22:04:52.7294503Z remote: Compressing objects:  93% (771/828)        
2024-08-20T22:04:52.7296767Z remote: Compressing objects:  94% (779/828)        
2024-08-20T22:04:52.7298450Z remote: Compressing objects:  95% (787/828)        
2024-08-20T22:04:52.7300708Z remote: Compressing objects:  96% (795/828)        
2024-08-20T22:04:52.7302883Z remote: Compressing objects:  97% (804/828)        
2024-08-20T22:04:52.7303705Z remote: Compressing objects:  98% (812/828)        
2024-08-20T22:04:52.7306042Z remote: Compressing objects:  99% (820/828)        
2024-08-20T22:04:52.7306724Z remote: Compressing objects: 100% (828/828)        
2024-08-20T22:04:52.7307316Z remote: Compressing objects: 100% (828/828), done.        
2024-08-20T22:05:16.2498864Z remote: Total 1010199 (delta 1149), reused 1393 (delta 916), pack-reused 1008452 (from 1)        
2024-08-20T22:05:40.0639572Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object}
2024-08-20T22:05:40.0679336Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:05:40.0684432Z ##[endgroup]
2024-08-20T22:05:40.0686298Z ##[group]Determining the checkout info
2024-08-20T22:05:40.0686935Z ##[endgroup]
2024-08-20T22:05:40.0687581Z ##[group]Checking out the ref
2024-08-20T22:05:40.0688805Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:05:41.8043489Z ##[endgroup]
2024-08-20T22:05:41.8044242Z ##[group]Setting up auth for fetching submodules
2024-08-20T22:05:41.8048781Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-08-20T22:05:41.8114574Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-08-20T22:05:41.8155367Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-08-20T22:05:41.8195723Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-08-20T22:05:41.8233475Z ##[endgroup]
2024-08-20T22:05:41.8235349Z ##[group]Fetching submodules
2024-08-20T22:05:41.8237192Z [command]/usr/bin/git submodule sync --recursive
2024-08-20T22:05:41.8616003Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-08-20T22:05:41.8981114Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-08-20T22:05:41.8982751Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-08-20T22:05:41.8986161Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-08-20T22:05:41.8989557Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-08-20T22:05:41.8993703Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-08-20T22:05:41.8996829Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-08-20T22:05:41.9000561Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-08-20T22:05:41.9004493Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-08-20T22:05:41.9008227Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-08-20T22:05:41.9012540Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-08-20T22:05:41.9016459Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-08-20T22:05:41.9022004Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-08-20T22:05:41.9026188Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-08-20T22:05:41.9030587Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-08-20T22:05:41.9034783Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-08-20T22:05:41.9039468Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:05:41.9047276Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-08-20T22:05:41.9051979Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-08-20T22:05:41.9056521Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-08-20T22:05:41.9061477Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-08-20T22:05:41.9066167Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-08-20T22:05:41.9071602Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-08-20T22:05:41.9076292Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-08-20T22:05:41.9081260Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-08-20T22:05:41.9086244Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-08-20T22:05:41.9091764Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-08-20T22:05:41.9096850Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-08-20T22:05:41.9102115Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-08-20T22:05:41.9107429Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-08-20T22:05:41.9112931Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-08-20T22:05:41.9118308Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-08-20T22:05:41.9124433Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-08-20T22:05:41.9129691Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-08-20T22:05:41.9138911Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-08-20T22:05:41.9174329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-08-20T22:05:42.2538572Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-08-20T22:05:42.4583550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-08-20T22:05:42.6348403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-08-20T22:05:42.8765580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-08-20T22:05:44.8317387Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-08-20T22:05:56.1443869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-08-20T22:05:56.5432329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-08-20T22:05:56.9671966Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-08-20T22:05:57.6039597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-08-20T22:05:58.8384987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-08-20T22:06:00.8941679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-08-20T22:06:06.8341634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-08-20T22:06:08.2248731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-08-20T22:06:10.0316350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-08-20T22:06:11.3400818Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-08-20T22:06:11.7498949Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-08-20T22:06:12.0800276Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-08-20T22:06:13.1863593Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-08-20T22:06:13.5265006Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-08-20T22:06:13.8174794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-08-20T22:06:15.4320908Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-08-20T22:06:16.1319018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-08-20T22:06:16.7827282Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-08-20T22:06:22.8079580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-08-20T22:06:24.7514327Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-08-20T22:06:29.0237386Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-08-20T22:06:29.2313266Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-08-20T22:06:38.4572977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-08-20T22:06:38.6124725Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-08-20T22:06:38.8143678Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-08-20T22:06:39.7350012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-08-20T22:06:40.0126412Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-08-20T22:06:40.6520153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-08-20T22:06:41.1244307Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-08-20T22:06:41.1396925Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-08-20T22:06:41.1515990Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-08-20T22:06:41.1830092Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-08-20T22:06:41.2288154Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-08-20T22:06:42.3997222Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2'
2024-08-20T22:06:42.4283553Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-08-20T22:06:42.4809577Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-08-20T22:06:42.5936681Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5'
2024-08-20T22:06:42.6342297Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171'
2024-08-20T22:06:43.2608261Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-08-20T22:06:43.5475336Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-08-20T22:06:43.6416178Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-08-20T22:06:43.6434786Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:06:43.6438228Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:06:43.6441755Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:06:43.6445385Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:06:43.6449706Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:06:43.6484001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-08-20T22:06:44.7078204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-08-20T22:06:45.3089555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-08-20T22:06:47.4237125Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-08-20T22:06:48.4806556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-08-20T22:06:48.7989196Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-08-20T22:06:48.9100476Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-08-20T22:06:49.4178560Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-08-20T22:06:49.4884339Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-08-20T22:06:49.5034302Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-08-20T22:06:49.6548233Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-08-20T22:06:49.7012309Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592'
2024-08-20T22:06:49.7471019Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-08-20T22:06:49.7794077Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-08-20T22:06:49.8326858Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-08-20T22:06:49.8491312Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b'
2024-08-20T22:06:49.8515928Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-08-20T22:06:49.8545750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-08-20T22:07:03.3371666Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67'
2024-08-20T22:07:03.3590299Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-08-20T22:07:03.4595772Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415'
2024-08-20T22:07:03.4614353Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:03.4617526Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:03.4621106Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:03.4652957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-08-20T22:07:04.0306442Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-08-20T22:07:05.2551897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-08-20T22:07:06.3879736Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-08-20T22:07:06.3900225Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:06.3903561Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:06.3907161Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:06.3910821Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:06.3915092Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:06.3918967Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:06.3922768Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:06.3926565Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:06.3958706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-08-20T22:07:07.2179267Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-08-20T22:07:07.5910013Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-08-20T22:07:08.8379496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-08-20T22:07:09.1291780Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-08-20T22:07:09.6880315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-08-20T22:07:10.7301449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-08-20T22:07:18.3147205Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-08-20T22:07:18.8031633Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-08-20T22:07:18.8271201Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-08-20T22:07:18.8704816Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-08-20T22:07:18.8873981Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-08-20T22:07:18.8892788Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:18.8924903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-08-20T22:07:19.1867317Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-08-20T22:07:19.2093330Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-08-20T22:07:19.2573703Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-08-20T22:07:19.3810878Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-08-20T22:07:19.4014025Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-08-20T22:07:19.4465928Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2024-08-20T22:07:19.5131750Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-08-20T22:07:19.5597024Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-08-20T22:07:19.5900185Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-08-20T22:07:19.7161236Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-08-20T22:07:20.2291406Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557'
2024-08-20T22:07:20.2330153Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:20.2334861Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:20.2364174Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2024-08-20T22:07:20.6719585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-08-20T22:07:21.5843227Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6'
2024-08-20T22:07:21.6259853Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242'
2024-08-20T22:07:21.7171714Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-08-20T22:07:21.7193663Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:21.7197001Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:21.7200320Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:21.7203796Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:21.7207639Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:21.7211233Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:21.7214972Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:21.7218454Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:21.7250490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-08-20T22:07:22.1524683Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-08-20T22:07:23.2021049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-08-20T22:07:23.5492184Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-08-20T22:07:29.7352228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-08-20T22:07:29.9941547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-08-20T22:07:30.1829494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-08-20T22:07:30.4852454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-08-20T22:07:37.2565083Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-08-20T22:07:37.3039306Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-08-20T22:07:37.3230276Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-08-20T22:07:37.4459998Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-08-20T22:07:37.4621137Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-08-20T22:07:37.4809496Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-08-20T22:07:37.5014308Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-08-20T22:07:37.5032671Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:37.5036160Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:37.5071469Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-08-20T22:07:39.3718641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-08-20T22:07:40.7416458Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-08-20T22:07:40.7946900Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-08-20T22:07:41.4687972Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-08-20T22:07:41.4832029Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-08-20T22:07:41.8005944Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-08-20T22:07:41.8030033Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:41.8033519Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:41.8065344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-08-20T22:07:42.2671012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-08-20T22:07:43.3419344Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-08-20T22:07:43.4234239Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-08-20T22:07:43.4354031Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-08-20T22:07:43.4511462Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-08-20T22:07:43.4959275Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c'
2024-08-20T22:07:43.5304090Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-08-20T22:07:43.5784404Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-08-20T22:07:43.6125843Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-08-20T22:07:43.6143936Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:43.6147179Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:43.6150838Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:43.6154177Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:43.6187347Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-08-20T22:07:44.6735308Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-08-20T22:07:44.8916728Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-08-20T22:07:46.0854063Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-08-20T22:07:47.0794140Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-08-20T22:07:47.0993464Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-08-20T22:07:47.1781690Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-08-20T22:07:47.2126375Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-08-20T22:07:47.2146456Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:47.2178094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-08-20T22:07:47.4422447Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-08-20T22:07:47.4473703Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-08-20T22:07:47.4850498Z Entering 'android/libs/fbjni'
2024-08-20T22:07:47.4914495Z Entering 'third_party/FP16'
2024-08-20T22:07:47.4968753Z Entering 'third_party/FXdiv'
2024-08-20T22:07:47.5022576Z Entering 'third_party/NNPACK'
2024-08-20T22:07:47.5076595Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:07:47.5129252Z Entering 'third_party/XNNPACK'
2024-08-20T22:07:47.5199431Z Entering 'third_party/benchmark'
2024-08-20T22:07:47.5258726Z Entering 'third_party/cpp-httplib'
2024-08-20T22:07:47.5313028Z Entering 'third_party/cpuinfo'
2024-08-20T22:07:47.5365720Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:07:47.5420103Z Entering 'third_party/cutlass'
2024-08-20T22:07:47.5481314Z Entering 'third_party/eigen'
2024-08-20T22:07:47.5535843Z Entering 'third_party/fbgemm'
2024-08-20T22:07:47.5588189Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:47.5650930Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:47.5713171Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:47.5803312Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:47.5863955Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:47.5924031Z Entering 'third_party/flatbuffers'
2024-08-20T22:07:47.5983941Z Entering 'third_party/fmt'
2024-08-20T22:07:47.6035198Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:07:47.6087739Z Entering 'third_party/gloo'
2024-08-20T22:07:47.6139832Z Entering 'third_party/googletest'
2024-08-20T22:07:47.6192762Z Entering 'third_party/ideep'
2024-08-20T22:07:47.6241961Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:47.6301971Z Entering 'third_party/ittapi'
2024-08-20T22:07:47.6354119Z Entering 'third_party/kineto'
2024-08-20T22:07:47.6405895Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:47.6456035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:47.6508601Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:47.6559266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:47.6610595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:47.6660346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:47.6715988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:47.6766557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:47.6823655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:47.6876840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:47.6930464Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:47.6981077Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:47.7033238Z Entering 'third_party/mimalloc'
2024-08-20T22:07:47.7085298Z Entering 'third_party/nccl/nccl'
2024-08-20T22:07:47.7140547Z Entering 'third_party/nlohmann'
2024-08-20T22:07:47.7194941Z Entering 'third_party/onnx'
2024-08-20T22:07:47.7260452Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:47.7313274Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:47.7369774Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:07:47.7427315Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:47.7479177Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:47.7529611Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:47.7577704Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:47.7627679Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:47.7676576Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:47.7725306Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:47.7773722Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:47.7826257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:47.7882106Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:47.7954583Z Entering 'third_party/pocketfft'
2024-08-20T22:07:47.8008423Z Entering 'third_party/protobuf'
2024-08-20T22:07:47.8064685Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:47.8114880Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:47.8167846Z Entering 'third_party/psimd'
2024-08-20T22:07:47.8222594Z Entering 'third_party/pthreadpool'
2024-08-20T22:07:47.8275331Z Entering 'third_party/pybind11'
2024-08-20T22:07:47.8327186Z Entering 'third_party/python-peachpy'
2024-08-20T22:07:47.8385802Z Entering 'third_party/sleef'
2024-08-20T22:07:47.8437841Z Entering 'third_party/tensorpipe'
2024-08-20T22:07:47.8493277Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:47.8542060Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:47.8593824Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:47.8643780Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:47.8692213Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:47.8766658Z ##[endgroup]
2024-08-20T22:07:47.8769405Z ##[group]Persisting credentials for submodules
2024-08-20T22:07:47.8772528Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-08-20T22:07:47.9138978Z Entering 'android/libs/fbjni'
2024-08-20T22:07:47.9212179Z Entering 'third_party/FP16'
2024-08-20T22:07:47.9282303Z Entering 'third_party/FXdiv'
2024-08-20T22:07:47.9349512Z Entering 'third_party/NNPACK'
2024-08-20T22:07:47.9418336Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:07:47.9486290Z Entering 'third_party/XNNPACK'
2024-08-20T22:07:47.9569418Z Entering 'third_party/benchmark'
2024-08-20T22:07:47.9636607Z Entering 'third_party/cpp-httplib'
2024-08-20T22:07:47.9708884Z Entering 'third_party/cpuinfo'
2024-08-20T22:07:47.9776713Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:07:47.9845198Z Entering 'third_party/cutlass'
2024-08-20T22:07:47.9922827Z Entering 'third_party/eigen'
2024-08-20T22:07:47.9995086Z Entering 'third_party/fbgemm'
2024-08-20T22:07:48.0061150Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:48.0128918Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:48.0195553Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:48.0269329Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:48.0335838Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:48.0403871Z Entering 'third_party/flatbuffers'
2024-08-20T22:07:48.0474572Z Entering 'third_party/fmt'
2024-08-20T22:07:48.0542866Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:07:48.0613776Z Entering 'third_party/gloo'
2024-08-20T22:07:48.0682134Z Entering 'third_party/googletest'
2024-08-20T22:07:48.0749957Z Entering 'third_party/ideep'
2024-08-20T22:07:48.0819711Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:48.0895653Z Entering 'third_party/ittapi'
2024-08-20T22:07:48.0962101Z Entering 'third_party/kineto'
2024-08-20T22:07:48.1027701Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:48.1093792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:48.1162217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:48.1232624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:48.1299279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:48.1363537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:48.1434935Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:48.1502882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:48.1573619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:48.1641067Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:48.1715809Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:48.1782289Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:48.1851790Z Entering 'third_party/mimalloc'
2024-08-20T22:07:48.1919822Z Entering 'third_party/nccl/nccl'
2024-08-20T22:07:48.1994516Z Entering 'third_party/nlohmann'
2024-08-20T22:07:48.2062806Z Entering 'third_party/onnx'
2024-08-20T22:07:48.2143314Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:48.2211942Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:48.2286301Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:07:48.2353420Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:48.2421732Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:48.2488149Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:48.2553852Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:48.2622121Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:48.2689181Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:48.2754167Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:48.2820219Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:48.2889060Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:48.2958941Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:48.3049057Z Entering 'third_party/pocketfft'
2024-08-20T22:07:48.3116354Z Entering 'third_party/protobuf'
2024-08-20T22:07:48.3191533Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:48.3258928Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:48.3332090Z Entering 'third_party/psimd'
2024-08-20T22:07:48.3402945Z Entering 'third_party/pthreadpool'
2024-08-20T22:07:48.3470993Z Entering 'third_party/pybind11'
2024-08-20T22:07:48.3544003Z Entering 'third_party/python-peachpy'
2024-08-20T22:07:48.3611134Z Entering 'third_party/sleef'
2024-08-20T22:07:48.3682878Z Entering 'third_party/tensorpipe'
2024-08-20T22:07:48.3748491Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:48.3815857Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:48.3881987Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:48.3947272Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:48.4011533Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:48.4107110Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-08-20T22:07:48.4477236Z Entering 'android/libs/fbjni'
2024-08-20T22:07:48.4538424Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-08-20T22:07:48.4559634Z Entering 'third_party/FP16'
2024-08-20T22:07:48.4626036Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-08-20T22:07:48.4647378Z Entering 'third_party/FXdiv'
2024-08-20T22:07:48.4708197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-08-20T22:07:48.4730256Z Entering 'third_party/NNPACK'
2024-08-20T22:07:48.4796061Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-08-20T22:07:48.4818449Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:07:48.4880020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-08-20T22:07:48.4901575Z Entering 'third_party/XNNPACK'
2024-08-20T22:07:48.4962470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-08-20T22:07:48.5000382Z Entering 'third_party/benchmark'
2024-08-20T22:07:48.5061668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:07:48.5083828Z Entering 'third_party/cpp-httplib'
2024-08-20T22:07:48.5145437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-08-20T22:07:48.5167435Z Entering 'third_party/cpuinfo'
2024-08-20T22:07:48.5229184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-08-20T22:07:48.5251457Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:07:48.5316624Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-08-20T22:07:48.5338209Z Entering 'third_party/cutlass'
2024-08-20T22:07:48.5399900Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-08-20T22:07:48.5431689Z Entering 'third_party/eigen'
2024-08-20T22:07:48.5496667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-08-20T22:07:48.5520161Z Entering 'third_party/fbgemm'
2024-08-20T22:07:48.5586750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-08-20T22:07:48.5606691Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:48.5668748Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-08-20T22:07:48.5689211Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:48.5755626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-08-20T22:07:48.5775762Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:48.5836756Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-08-20T22:07:48.5864173Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:48.5925862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.5946231Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:48.6008707Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-08-20T22:07:48.6033301Z Entering 'third_party/flatbuffers'
2024-08-20T22:07:48.6097582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-08-20T22:07:48.6121623Z Entering 'third_party/fmt'
2024-08-20T22:07:48.6182763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-08-20T22:07:48.6203919Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:07:48.6264826Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-08-20T22:07:48.6286579Z Entering 'third_party/gloo'
2024-08-20T22:07:48.6347131Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-08-20T22:07:48.6369071Z Entering 'third_party/googletest'
2024-08-20T22:07:48.6430184Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.6452956Z Entering 'third_party/ideep'
2024-08-20T22:07:48.6515003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-08-20T22:07:48.6534109Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:48.6595047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-08-20T22:07:48.6623980Z Entering 'third_party/ittapi'
2024-08-20T22:07:48.6688777Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-08-20T22:07:48.6710662Z Entering 'third_party/kineto'
2024-08-20T22:07:48.6776922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-08-20T22:07:48.6796356Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:48.6864701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-08-20T22:07:48.6884511Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:48.6947030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-08-20T22:07:48.6974194Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:48.7035800Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-08-20T22:07:48.7056747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:48.7119284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-08-20T22:07:48.7139704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:48.7206519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-08-20T22:07:48.7224427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:48.7287188Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-08-20T22:07:48.7311340Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:48.7374869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-08-20T22:07:48.7394891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:48.7457013Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.7477573Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:48.7539906Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-08-20T22:07:48.7561136Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:48.7623790Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-08-20T22:07:48.7647274Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:48.7709351Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-08-20T22:07:48.7730975Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:48.7791129Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.7815675Z Entering 'third_party/mimalloc'
2024-08-20T22:07:48.7881037Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-08-20T22:07:48.7903058Z Entering 'third_party/nccl/nccl'
2024-08-20T22:07:48.7963837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-08-20T22:07:48.7986011Z Entering 'third_party/nlohmann'
2024-08-20T22:07:48.8047375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-08-20T22:07:48.8070452Z Entering 'third_party/onnx'
2024-08-20T22:07:48.8132105Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-08-20T22:07:48.8166008Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:48.8230165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:07:48.8251969Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:48.8315044Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:07:48.8341087Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:07:48.8403612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-08-20T22:07:48.8425300Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:48.8485940Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:07:48.8506202Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:48.8566292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.8587159Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:48.8647568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-08-20T22:07:48.8667859Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:48.8728638Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-08-20T22:07:48.8750346Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:48.8813671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-08-20T22:07:48.8833916Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:48.8895172Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-08-20T22:07:48.8915458Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:48.8977969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-08-20T22:07:48.8996672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:48.9057761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-08-20T22:07:48.9080667Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:48.9141512Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-08-20T22:07:48.9163991Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:48.9225993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-08-20T22:07:48.9268469Z Entering 'third_party/pocketfft'
2024-08-20T22:07:48.9329383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-08-20T22:07:48.9350545Z Entering 'third_party/protobuf'
2024-08-20T22:07:48.9414490Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-08-20T22:07:48.9437549Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:48.9498553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-08-20T22:07:48.9518547Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:48.9579418Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:48.9604426Z Entering 'third_party/psimd'
2024-08-20T22:07:48.9670552Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-08-20T22:07:48.9694253Z Entering 'third_party/pthreadpool'
2024-08-20T22:07:48.9756633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-08-20T22:07:48.9778314Z Entering 'third_party/pybind11'
2024-08-20T22:07:48.9839509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:07:48.9861128Z Entering 'third_party/python-peachpy'
2024-08-20T22:07:48.9923509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-08-20T22:07:48.9945380Z Entering 'third_party/sleef'
2024-08-20T22:07:49.0006886Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-08-20T22:07:49.0033005Z Entering 'third_party/tensorpipe'
2024-08-20T22:07:49.0095003Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-08-20T22:07:49.0114788Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:49.0176495Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-08-20T22:07:49.0197000Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:49.0258370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-08-20T22:07:49.0280042Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:49.0341084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-08-20T22:07:49.0361751Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:49.0423009Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-08-20T22:07:49.0440792Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:49.0503982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-08-20T22:07:49.1398515Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-08-20T22:07:49.1769850Z Entering 'android/libs/fbjni'
2024-08-20T22:07:49.1821337Z Entering 'third_party/FP16'
2024-08-20T22:07:49.1873952Z Entering 'third_party/FXdiv'
2024-08-20T22:07:49.1925547Z Entering 'third_party/NNPACK'
2024-08-20T22:07:49.1977444Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:07:49.2029009Z Entering 'third_party/XNNPACK'
2024-08-20T22:07:49.2100231Z Entering 'third_party/benchmark'
2024-08-20T22:07:49.2152403Z Entering 'third_party/cpp-httplib'
2024-08-20T22:07:49.2207267Z Entering 'third_party/cpuinfo'
2024-08-20T22:07:49.2263503Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:07:49.2316090Z Entering 'third_party/cutlass'
2024-08-20T22:07:49.2379589Z Entering 'third_party/eigen'
2024-08-20T22:07:49.2432644Z Entering 'third_party/fbgemm'
2024-08-20T22:07:49.2483541Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:49.2533828Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:49.2584350Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:49.2641258Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:49.2690543Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:49.2743174Z Entering 'third_party/flatbuffers'
2024-08-20T22:07:49.2798107Z Entering 'third_party/fmt'
2024-08-20T22:07:49.2849147Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:07:49.2901512Z Entering 'third_party/gloo'
2024-08-20T22:07:49.2954700Z Entering 'third_party/googletest'
2024-08-20T22:07:49.3007276Z Entering 'third_party/ideep'
2024-08-20T22:07:49.3056142Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:49.3116345Z Entering 'third_party/ittapi'
2024-08-20T22:07:49.3168980Z Entering 'third_party/kineto'
2024-08-20T22:07:49.3219282Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:49.3269651Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:49.3321772Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:49.3377511Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:49.3428862Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:49.3478649Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:49.3533825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:49.3587073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:49.3638821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:49.3692117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:49.3747562Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:49.3797487Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:49.3849930Z Entering 'third_party/mimalloc'
2024-08-20T22:07:49.3907219Z Entering 'third_party/nccl/nccl'
2024-08-20T22:07:49.3959194Z Entering 'third_party/nlohmann'
2024-08-20T22:07:49.4017649Z Entering 'third_party/onnx'
2024-08-20T22:07:49.4083234Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:49.4135425Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:49.4192283Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:07:49.4244157Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:49.4293683Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:49.4342627Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:49.4391994Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:49.4443259Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:49.4491430Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:49.4539825Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:49.4587847Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:49.4641004Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:49.4694339Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:49.4764510Z Entering 'third_party/pocketfft'
2024-08-20T22:07:49.4820002Z Entering 'third_party/protobuf'
2024-08-20T22:07:49.4874983Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:49.4925707Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:49.4981168Z Entering 'third_party/psimd'
2024-08-20T22:07:49.5034637Z Entering 'third_party/pthreadpool'
2024-08-20T22:07:49.5087893Z Entering 'third_party/pybind11'
2024-08-20T22:07:49.5139054Z Entering 'third_party/python-peachpy'
2024-08-20T22:07:49.5190709Z Entering 'third_party/sleef'
2024-08-20T22:07:49.5242036Z Entering 'third_party/tensorpipe'
2024-08-20T22:07:49.5297035Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:49.5347538Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:49.5397601Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:49.5447552Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:49.5498858Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:49.5576073Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-08-20T22:07:49.5942945Z Entering 'android/libs/fbjni'
2024-08-20T22:07:49.5995473Z Entering 'third_party/FP16'
2024-08-20T22:07:49.6047780Z Entering 'third_party/FXdiv'
2024-08-20T22:07:49.6101083Z Entering 'third_party/NNPACK'
2024-08-20T22:07:49.6153692Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-20T22:07:49.6210968Z Entering 'third_party/XNNPACK'
2024-08-20T22:07:49.6281844Z Entering 'third_party/benchmark'
2024-08-20T22:07:49.6335056Z Entering 'third_party/cpp-httplib'
2024-08-20T22:07:49.6388297Z Entering 'third_party/cpuinfo'
2024-08-20T22:07:49.6441458Z Entering 'third_party/cudnn_frontend'
2024-08-20T22:07:49.6494787Z Entering 'third_party/cutlass'
2024-08-20T22:07:49.6555320Z Entering 'third_party/eigen'
2024-08-20T22:07:49.6610623Z Entering 'third_party/fbgemm'
2024-08-20T22:07:49.6662806Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-20T22:07:49.6713652Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-20T22:07:49.6763689Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-20T22:07:49.6828981Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-20T22:07:49.6879703Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-20T22:07:49.6933433Z Entering 'third_party/flatbuffers'
2024-08-20T22:07:49.6988994Z Entering 'third_party/fmt'
2024-08-20T22:07:49.7041562Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-20T22:07:49.7098063Z Entering 'third_party/gloo'
2024-08-20T22:07:49.7150631Z Entering 'third_party/googletest'
2024-08-20T22:07:49.7204903Z Entering 'third_party/ideep'
2024-08-20T22:07:49.7255084Z Entering 'third_party/ideep/mkl-dnn'
2024-08-20T22:07:49.7316637Z Entering 'third_party/ittapi'
2024-08-20T22:07:49.7369766Z Entering 'third_party/kineto'
2024-08-20T22:07:49.7421145Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-20T22:07:49.7471724Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-20T22:07:49.7524497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-20T22:07:49.7575518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-20T22:07:49.7626191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-20T22:07:49.7675907Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-20T22:07:49.7733233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-20T22:07:49.7784197Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-20T22:07:49.7834929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-20T22:07:49.7886170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-20T22:07:49.7939792Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-20T22:07:49.7990576Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-20T22:07:49.8045203Z Entering 'third_party/mimalloc'
2024-08-20T22:07:49.8100637Z Entering 'third_party/nccl/nccl'
2024-08-20T22:07:49.8152171Z Entering 'third_party/nlohmann'
2024-08-20T22:07:49.8206041Z Entering 'third_party/onnx'
2024-08-20T22:07:49.8271989Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-20T22:07:49.8326009Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-20T22:07:49.8387288Z Entering 'third_party/opentelemetry-cpp'
2024-08-20T22:07:49.8441407Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-20T22:07:49.8493436Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-20T22:07:49.8543761Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-20T22:07:49.8595137Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-20T22:07:49.8645949Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-20T22:07:49.8697162Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-20T22:07:49.8747750Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-20T22:07:49.8798031Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-20T22:07:49.8849641Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-20T22:07:49.8902634Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-20T22:07:49.8976646Z Entering 'third_party/pocketfft'
2024-08-20T22:07:49.9027347Z Entering 'third_party/protobuf'
2024-08-20T22:07:49.9082682Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-20T22:07:49.9133665Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-20T22:07:49.9186814Z Entering 'third_party/psimd'
2024-08-20T22:07:49.9239565Z Entering 'third_party/pthreadpool'
2024-08-20T22:07:49.9291550Z Entering 'third_party/pybind11'
2024-08-20T22:07:49.9343492Z Entering 'third_party/python-peachpy'
2024-08-20T22:07:49.9395984Z Entering 'third_party/sleef'
2024-08-20T22:07:49.9449102Z Entering 'third_party/tensorpipe'
2024-08-20T22:07:49.9500210Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-20T22:07:49.9549883Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-20T22:07:49.9600079Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-20T22:07:49.9650884Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-20T22:07:49.9699639Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-20T22:07:49.9774201Z ##[endgroup]
2024-08-20T22:07:49.9825935Z [command]/usr/bin/git log -1 --format='%H'
2024-08-20T22:07:49.9863531Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3'
2024-08-20T22:07:50.0072868Z Prepare all required actions
2024-08-20T22:07:50.0073355Z Getting action download info
2024-08-20T22:07:50.1864115Z ##[group]Run ./.github/actions/setup-linux
2024-08-20T22:07:50.1864546Z env:
2024-08-20T22:07:50.1864827Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:50.1865181Z ##[endgroup]
2024-08-20T22:07:50.1936210Z ##[group]Run set -euo pipefail
2024-08-20T22:07:50.1936657Z set -euo pipefail
2024-08-20T22:07:50.1937064Z function get_ec2_metadata() {
2024-08-20T22:07:50.1937772Z   # Pulled from instance metadata endpoint for EC2
2024-08-20T22:07:50.1938903Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2024-08-20T22:07:50.1939669Z   category=$1
2024-08-20T22:07:50.1940261Z   # If it is GCP runner (runner name contains gcp), do not run this
2024-08-20T22:07:50.1941015Z   runner_name_str=i-0e9f01dbcd5a47485
2024-08-20T22:07:50.1941522Z   if [[ -f /.inarc ]]; then
2024-08-20T22:07:50.1942013Z     echo "ARC Runner, no info on ec2 metadata"
2024-08-20T22:07:50.1942564Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2024-08-20T22:07:50.1943243Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2024-08-20T22:07:50.1943910Z   else
2024-08-20T22:07:50.1944427Z     curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2024-08-20T22:07:50.1945067Z   fi
2024-08-20T22:07:50.1945382Z }
2024-08-20T22:07:50.1945748Z echo "ami-id: $(get_ec2_metadata ami-id)"
2024-08-20T22:07:50.1946336Z echo "instance-id: $(get_ec2_metadata instance-id)"
2024-08-20T22:07:50.1946982Z echo "instance-type: $(get_ec2_metadata instance-type)"
2024-08-20T22:07:50.1947544Z echo "system info $(uname -a)"
2024-08-20T22:07:50.1960935Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:50.1961435Z env:
2024-08-20T22:07:50.1961710Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:50.1962059Z ##[endgroup]
2024-08-20T22:07:50.2055753Z ami-id: ami-06c68f701d8090592
2024-08-20T22:07:50.2112025Z instance-id: i-0e9f01dbcd5a47485
2024-08-20T22:07:50.2169506Z instance-type: g5.4xlarge
2024-08-20T22:07:50.2183120Z system info Linux ip-10-0-5-191.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-08-20T22:07:50.2213813Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-08-20T22:07:50.2215179Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-08-20T22:07:50.2225402Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:50.2225903Z env:
2024-08-20T22:07:50.2226187Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:50.2226537Z ##[endgroup]
2024-08-20T22:07:50.2311191Z ##[group]Run if systemctl is-active --quiet docker; then
2024-08-20T22:07:50.2311816Z if systemctl is-active --quiet docker; then
2024-08-20T22:07:50.2312365Z     echo "Docker daemon is running...";
2024-08-20T22:07:50.2312979Z else
2024-08-20T22:07:50.2313479Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2024-08-20T22:07:50.2314240Z fi
2024-08-20T22:07:50.2323359Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:50.2323858Z env:
2024-08-20T22:07:50.2324140Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:50.2324489Z ##[endgroup]
2024-08-20T22:07:50.2407212Z Docker daemon is running...
2024-08-20T22:07:50.2455455Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:07:50.2456018Z with:
2024-08-20T22:07:50.2456292Z   shell: bash
2024-08-20T22:07:50.2456591Z   timeout_minutes: 5
2024-08-20T22:07:50.2456920Z   max_attempts: 3
2024-08-20T22:07:50.2457240Z   retry_wait_seconds: 30
2024-08-20T22:07:50.2460399Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2024-08-20T22:07:50.2463376Z   polling_interval_seconds: 1
2024-08-20T22:07:50.2463789Z   warning_on_retry: true
2024-08-20T22:07:50.2464146Z   continue_on_error: false
2024-08-20T22:07:50.2464487Z env:
2024-08-20T22:07:50.2464764Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:50.2465126Z   AWS_RETRY_MODE: standard
2024-08-20T22:07:50.2465477Z   AWS_MAX_ATTEMPTS: 5
2024-08-20T22:07:50.2465824Z   AWS_DEFAULT_REGION: us-east-1
2024-08-20T22:07:50.2466205Z ##[endgroup]
2024-08-20T22:07:51.4070123Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:07:51.4070991Z Configure a credential helper to remove this warning. See
2024-08-20T22:07:51.4072060Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:07:51.4073230Z 
2024-08-20T22:07:51.4073636Z Login Succeeded
2024-08-20T22:07:52.3026654Z Command completed after 1 attempt(s).
2024-08-20T22:07:52.3089111Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:07:52.3089820Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:07:52.3090462Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-08-20T22:07:52.3099627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:52.3100128Z env:
2024-08-20T22:07:52.3100413Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:52.3100769Z ##[endgroup]
2024-08-20T22:07:52.3194413Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T22:07:52.3195184Z # ignore expansion of "docker ps -q" since it could be empty
2024-08-20T22:07:52.3195780Z # shellcheck disable=SC2046
2024-08-20T22:07:52.3196250Z docker stop $(docker ps -q) || true
2024-08-20T22:07:52.3196743Z # Prune all of the docker images
2024-08-20T22:07:52.3197198Z docker system prune -af
2024-08-20T22:07:52.3206111Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:52.3206609Z env:
2024-08-20T22:07:52.3206895Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:52.3207245Z ##[endgroup]
2024-08-20T22:07:52.3519888Z "docker stop" requires at least 1 argument.
2024-08-20T22:07:52.3520823Z See 'docker stop --help'.
2024-08-20T22:07:52.3521161Z 
2024-08-20T22:07:52.3521404Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-08-20T22:07:52.3521776Z 
2024-08-20T22:07:52.3521936Z Stop one or more running containers
2024-08-20T22:07:55.5661905Z Deleted Images:
2024-08-20T22:07:55.5662603Z untagged: pytorch/manylinux-builder:cuda11.8
2024-08-20T22:07:55.5663562Z untagged: pytorch/manylinux-builder@sha256:1da7448d55e742ab5c3dd804f685a4734c07c70e6bfc2bdd15611316384a5913
2024-08-20T22:07:55.5664535Z deleted: sha256:f82203a72d1525126322842d3cbc33b59f576cbacd350ff1d9a9b5082b4f9f9c
2024-08-20T22:07:55.5665664Z deleted: sha256:e4bbe9c719225e81fc3b74e477d4b9ded7aa6965825b8ecb8a3a603870658d93
2024-08-20T22:07:55.5666504Z deleted: sha256:ec3ebb2da3f73845a4a3868a52e1c8e218a06f4818da979ce7c4c1b11bca95d2
2024-08-20T22:07:55.5667695Z deleted: sha256:114f2f15c700d1c0f0eb6dcaffa4773cc852b93c91fd4eb2e80a6cd53464ee7f
2024-08-20T22:07:55.5668778Z deleted: sha256:0a0612c5178b431314553e6647737fd7a79924c55e43e59746a58075a1c83ff5
2024-08-20T22:07:55.5669593Z deleted: sha256:c6d14924e4a0e255491d25f5c889f850f9c563c50fa6569e30650790c1cd4199
2024-08-20T22:07:55.5670440Z deleted: sha256:b9536c56578c96f672f8fa5e5cb674efd55254a61dc3725d1d211e5cf11e0660
2024-08-20T22:07:55.5671778Z deleted: sha256:34c3858bc0792eae36559d9eabb189c942ccab4fe003c930a3df33c928a8316a
2024-08-20T22:07:55.5672699Z deleted: sha256:330e51dd8887b291c8e50fe1193d476a3e57d361664ab6b228851f9ae4f2bd42
2024-08-20T22:07:55.5673515Z deleted: sha256:42b5b52fce2e86827fb97ab811532cc70257bedb60215a9dfe7fbcecd5f93dc1
2024-08-20T22:07:55.5674341Z deleted: sha256:4e45e06030271f30a21b916c63084d0b376f73d3c4532b0122ca4bae019c81b4
2024-08-20T22:07:55.5675154Z deleted: sha256:98c58bf4ff64e2c18951eee3c0d7e24fdfa41630240a5f6b74cb58b3b2e1be1b
2024-08-20T22:07:55.5675983Z deleted: sha256:647d8037cc4f1cffaa9c2ec674a08f8239f5ea769f01e8c1127234747628be3a
2024-08-20T22:07:55.5676796Z deleted: sha256:81d813e14347b1f136cc81a3de54540bb11026c4f947694dd0b32d333ed0397a
2024-08-20T22:07:55.5677604Z deleted: sha256:55f674ee50e75d0d51dc8f6b7276e6f47c8a50747f316fec7a477b0bb2d963e0
2024-08-20T22:07:55.5678415Z deleted: sha256:d3745dba7379be6639fe7c85b3ab202377d647a8054f53f9374fbeb15f36953a
2024-08-20T22:07:55.5679232Z deleted: sha256:814d656426b0a4d3b235b21be74d58846f89044f3dd424445c1ad7a5f2684ab2
2024-08-20T22:07:55.5680059Z deleted: sha256:fa79b0aaa6127b6d3d1b69bd9978a4cd71dccec7cce58276a8f53c5e809518ac
2024-08-20T22:07:55.5680909Z deleted: sha256:aec1db63f5c7a2b760a2f29a69b0d58a5b8b7cabf6aebf510a089ea2f4ef152b
2024-08-20T22:07:55.5681746Z deleted: sha256:a6f4c066d2722bcf36a96bc525f8577c6a441f6afe74852658d6c17f0035f481
2024-08-20T22:07:55.5682561Z deleted: sha256:89a45875b3418aad1a6a8991f45c186b012c8f376fd81ccfb5ce06791b4ca4b2
2024-08-20T22:07:55.5683374Z deleted: sha256:a3c55328dd48e4449b06f45ead530c20eb162f4d53dd8e6daff449a1542da956
2024-08-20T22:07:55.5684200Z deleted: sha256:ac6adb842e955f6962fe612341f7d8cbd229da6d734fc03de7479b97d012d90c
2024-08-20T22:07:55.5685026Z deleted: sha256:193be6826c48afddcf695d652e54306c60d277f6fc92f1441a25b3dc23478429
2024-08-20T22:07:55.5685839Z deleted: sha256:94c60bef5fd27616803e3c43479a4830ccaddd07320dff03124a5f531da1e42e
2024-08-20T22:07:55.5686673Z deleted: sha256:2d9b98533add278cee7c4f0399ad8a3f69f7d8bb10ede76b2ba3ccda9431efbb
2024-08-20T22:07:55.5687506Z deleted: sha256:a10b8283bffc0ad1b27f994823accc572b7c106e25931c8b679b25645a8311d2
2024-08-20T22:07:55.5688338Z deleted: sha256:e9703f96f03adaac000d2bab73c3bfa49e5c5ee53f3c8f00928ca6de6f09f65d
2024-08-20T22:07:55.5689165Z deleted: sha256:2ae3e34adb641136c168eb683e62b7d3477961d8ed29a7a29e438787a5562a7b
2024-08-20T22:07:55.5689986Z deleted: sha256:70b16fd15168e038d2b19f92dcc3f50a548f455616da4cb0ce037e8e5ef2a49d
2024-08-20T22:07:55.5690921Z deleted: sha256:0fcf3d488fbb56795d2f90e82de3332672055689e40e90564a23b12f4e11d53c
2024-08-20T22:07:55.5691730Z deleted: sha256:6a207a6e3935e86f17d05d0f2de447c6d90d4237823f7e7f20b9c5c9c114a6da
2024-08-20T22:07:55.5692534Z deleted: sha256:298bb1394216f7db84920d49c331f337d91fcbcfdf348b508f8ac8f22fe4c651
2024-08-20T22:07:55.5693356Z deleted: sha256:6e0c95e98eba0477b85cea804af8858bf711384bd27ff082656b4d54df7ed00b
2024-08-20T22:07:55.5694179Z deleted: sha256:ba91bda5c04f40a0824b90d7398509c2b6ad7171518c0a420d35fa3f3acba8de
2024-08-20T22:07:55.5695143Z deleted: sha256:2a81665acee7f6cae462715da3304fd7ba9eba42cd6f2b9f6565c724f7358cf1
2024-08-20T22:07:55.5696046Z deleted: sha256:9c2df2aa30b8b4d6f500a75541ea1b52fd38cbec9a773942623e34aee951dad2
2024-08-20T22:07:55.5696867Z deleted: sha256:174f5685490326fc0a1c0f5570b8663732189b327007e47ff13d2ca59673db02
2024-08-20T22:07:55.5697502Z 
2024-08-20T22:07:55.5697648Z Total reclaimed space: 11.69GB
2024-08-20T22:07:55.5742005Z ##[group]Run set +e
2024-08-20T22:07:55.5742376Z set +e
2024-08-20T22:07:55.5742677Z set -x
2024-08-20T22:07:55.5742977Z 
2024-08-20T22:07:55.5743291Z PT_DOMAIN=download.pytorch.org
2024-08-20T22:07:55.5744066Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2024-08-20T22:07:55.5745123Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2024-08-20T22:07:55.5745860Z # one is returned at random
2024-08-20T22:07:55.5746547Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2024-08-20T22:07:55.5747062Z 
2024-08-20T22:07:55.5747379Z if [ -z "${RESOLVED_IP}" ]; then
2024-08-20T22:07:55.5747993Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2024-08-20T22:07:55.5748757Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2024-08-20T22:07:55.5749298Z 
2024-08-20T22:07:55.5749621Z   if [ -z "${RESOLVED_IP}" ]; then
2024-08-20T22:07:55.5750170Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2024-08-20T22:07:55.5750675Z     exit 1
2024-08-20T22:07:55.5750980Z   fi
2024-08-20T22:07:55.5751264Z fi
2024-08-20T22:07:55.5751551Z 
2024-08-20T22:07:55.5751903Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2024-08-20T22:07:55.5752407Z   # Clean up any old records first
2024-08-20T22:07:55.5752918Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2024-08-20T22:07:55.5753369Z fi
2024-08-20T22:07:55.5753638Z 
2024-08-20T22:07:55.5754066Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2024-08-20T22:07:55.5754611Z cat /etc/hosts
2024-08-20T22:07:55.5763671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:55.5764169Z env:
2024-08-20T22:07:55.5764455Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:55.5764804Z ##[endgroup]
2024-08-20T22:07:55.5794731Z + PT_DOMAIN=download.pytorch.org
2024-08-20T22:07:55.5801396Z ++ dig -4 +short download.pytorch.org
2024-08-20T22:07:55.5801870Z ++ tail -n1
2024-08-20T22:07:55.5983614Z + RESOLVED_IP=18.160.10.76
2024-08-20T22:07:55.5984097Z + '[' -z 18.160.10.76 ']'
2024-08-20T22:07:55.5984534Z + grep -r download.pytorch.org /etc/hosts
2024-08-20T22:07:55.5997905Z 18.160.41.23 download.pytorch.org
2024-08-20T22:07:55.6000092Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2024-08-20T22:07:55.7676311Z + echo '18.160.10.76 download.pytorch.org'
2024-08-20T22:07:55.7677339Z + sudo tee -a /etc/hosts
2024-08-20T22:07:55.7997858Z 18.160.10.76 download.pytorch.org
2024-08-20T22:07:55.8018845Z + cat /etc/hosts
2024-08-20T22:07:55.8029345Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-08-20T22:07:55.8040135Z ::1         localhost6 localhost6.localdomain6
2024-08-20T22:07:55.8040861Z 18.160.10.76 download.pytorch.org
2024-08-20T22:07:55.8196003Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2024-08-20T22:07:55.8196608Z with:
2024-08-20T22:07:55.8197715Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8198938Z   docker-build-dir: .ci/docker
2024-08-20T22:07:55.8199333Z   working-directory: .
2024-08-20T22:07:55.8199816Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:55.8200368Z   force-push: false
2024-08-20T22:07:55.8200718Z env:
2024-08-20T22:07:55.8201009Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:55.8201360Z ##[endgroup]
2024-08-20T22:07:55.8222119Z ##[group]Run set -ex
2024-08-20T22:07:55.8222487Z set -ex
2024-08-20T22:07:55.8222788Z 
2024-08-20T22:07:55.8223539Z # If the docker build directory or the build script doesn't exist, the action will
2024-08-20T22:07:55.8224512Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2024-08-20T22:07:55.8225306Z # job could then download the pre-built image as usual
2024-08-20T22:07:55.8226052Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then
2024-08-20T22:07:55.8226728Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8227368Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8227935Z 
2024-08-20T22:07:55.8228447Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2024-08-20T22:07:55.8229072Z   exit 0
2024-08-20T22:07:55.8229382Z else
2024-08-20T22:07:55.8229759Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8230217Z fi
2024-08-20T22:07:55.8230542Z 
2024-08-20T22:07:55.8231040Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2024-08-20T22:07:55.8231885Z   # The docker image name already includes the ECR prefix and tag, so we can just
2024-08-20T22:07:55.8232669Z   # use it as it is, but first let's extract the tag
2024-08-20T22:07:55.8233380Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2024-08-20T22:07:55.8234103Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8235090Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8235755Z else
2024-08-20T22:07:55.8236397Z   DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2024-08-20T22:07:55.8237129Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8238144Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8239047Z fi
2024-08-20T22:07:55.8249302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:55.8249998Z env:
2024-08-20T22:07:55.8250493Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:55.8250958Z   REPO_NAME: pytorch
2024-08-20T22:07:55.8252294Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8253595Z   DOCKER_BUILD_DIR: .ci/docker
2024-08-20T22:07:55.8254277Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:55.8254895Z ##[endgroup]
2024-08-20T22:07:55.8287372Z + [[ ! -d .ci/docker ]]
2024-08-20T22:07:55.8287896Z + [[ ! -f .ci/docker/build.sh ]]
2024-08-20T22:07:55.8288510Z + echo skip=false
2024-08-20T22:07:55.8290743Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-08-20T22:07:55.8298080Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8299679Z ++ awk -F '[:,]' '{print $2}'
2024-08-20T22:07:55.8321430Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8322351Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8324063Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8363248Z ##[group]Run set +e
2024-08-20T22:07:55.8363788Z set +e
2024-08-20T22:07:55.8364211Z set -x
2024-08-20T22:07:55.8364600Z 
2024-08-20T22:07:55.8364997Z login() {
2024-08-20T22:07:55.8365968Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-08-20T22:07:55.8366773Z }
2024-08-20T22:07:55.8367980Z 
2024-08-20T22:07:55.8368385Z retry () {
2024-08-20T22:07:55.8368879Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-08-20T22:07:55.8369530Z }
2024-08-20T22:07:55.8369908Z 
2024-08-20T22:07:55.8370316Z retry login "${DOCKER_REGISTRY}"
2024-08-20T22:07:55.8371002Z 
2024-08-20T22:07:55.8371575Z # Check if image already exists, if it does then skip building it
2024-08-20T22:07:55.8372446Z if docker manifest inspect "${DOCKER_IMAGE}"; then
2024-08-20T22:07:55.8373010Z   exit 0
2024-08-20T22:07:55.8373411Z fi
2024-08-20T22:07:55.8373866Z 
2024-08-20T22:07:55.8374438Z # NB: This part requires a full checkout. Otherwise, the merge base will
2024-08-20T22:07:55.8375364Z # be empty.  The default action would be to continue rebuild the image
2024-08-20T22:07:55.8376288Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2024-08-20T22:07:55.8377060Z   # if we're on the base branch then use the parent commit
2024-08-20T22:07:55.8377769Z   MERGE_BASE=$(git rev-parse HEAD~)
2024-08-20T22:07:55.8378313Z else
2024-08-20T22:07:55.8378874Z   # otherwise we're on a PR, so use the most recent base commit
2024-08-20T22:07:55.8379673Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2024-08-20T22:07:55.8380330Z fi
2024-08-20T22:07:55.8380710Z 
2024-08-20T22:07:55.8381143Z if [[ -z "${MERGE_BASE}" ]]; then
2024-08-20T22:07:55.8381791Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8382367Z 
2024-08-20T22:07:55.8383072Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2024-08-20T22:07:55.8383965Z   exit 0
2024-08-20T22:07:55.8384383Z fi
2024-08-20T22:07:55.8384720Z 
2024-08-20T22:07:55.8385306Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2024-08-20T22:07:55.8386465Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2024-08-20T22:07:55.8387353Z   exit 1
2024-08-20T22:07:55.8387838Z fi
2024-08-20T22:07:55.8388173Z 
2024-08-20T22:07:55.8388750Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2024-08-20T22:07:55.8389863Z # If no image exists but the hash is the same as the previous hash then we should error out here
2024-08-20T22:07:55.8390816Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2024-08-20T22:07:55.8391929Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2024-08-20T22:07:55.8393255Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2024-08-20T22:07:55.8394265Z fi
2024-08-20T22:07:55.8394773Z 
2024-08-20T22:07:55.8395191Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2024-08-20T22:07:55.8404632Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:55.8405231Z env:
2024-08-20T22:07:55.8405601Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:55.8406118Z   DOCKER_BUILD_DIR: .ci/docker
2024-08-20T22:07:55.8406687Z   BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:07:55.8408071Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8409459Z   DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:55.8410153Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:55.8411062Z ##[endgroup]
2024-08-20T22:07:55.8441836Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:55.8442669Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:55.8444592Z + aws ecr get-login-password --region us-east-1
2024-08-20T22:07:55.8446624Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:56.3829952Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:07:56.3830981Z Configure a credential helper to remove this warning. See
2024-08-20T22:07:56.3831993Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:07:56.3832599Z 
2024-08-20T22:07:56.3832823Z Login Succeeded
2024-08-20T22:07:56.3852692Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:56.6045395Z {
2024-08-20T22:07:56.6045921Z 	"schemaVersion": 2,
2024-08-20T22:07:56.6047504Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-08-20T22:07:56.6048181Z 	"config": {
2024-08-20T22:07:56.6048739Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-08-20T22:07:56.6049588Z 		"size": 48711,
2024-08-20T22:07:56.6050682Z 		"digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb"
2024-08-20T22:07:56.6051976Z 	},
2024-08-20T22:07:56.6052361Z 	"layers": [
2024-08-20T22:07:56.6052790Z 		{
2024-08-20T22:07:56.6053929Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6054941Z 			"size": 28580681,
2024-08-20T22:07:56.6055834Z 			"digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a"
2024-08-20T22:07:56.6056798Z 		},
2024-08-20T22:07:56.6057250Z 		{
2024-08-20T22:07:56.6058084Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6058965Z 			"size": 7943451,
2024-08-20T22:07:56.6059849Z 			"digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d"
2024-08-20T22:07:56.6060701Z 		},
2024-08-20T22:07:56.6061078Z 		{
2024-08-20T22:07:56.6061672Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6062333Z 			"size": 55728572,
2024-08-20T22:07:56.6063025Z 			"digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450"
2024-08-20T22:07:56.6063870Z 		},
2024-08-20T22:07:56.6064179Z 		{
2024-08-20T22:07:56.6064716Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6065514Z 			"size": 186,
2024-08-20T22:07:56.6066157Z 			"digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195"
2024-08-20T22:07:56.6066857Z 		},
2024-08-20T22:07:56.6067676Z 		{
2024-08-20T22:07:56.6068217Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6068888Z 			"size": 6886,
2024-08-20T22:07:56.6069548Z 			"digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80"
2024-08-20T22:07:56.6070263Z 		},
2024-08-20T22:07:56.6070907Z 		{
2024-08-20T22:07:56.6071496Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6072180Z 			"size": 1329001756,
2024-08-20T22:07:56.6074104Z 			"digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108"
2024-08-20T22:07:56.6074900Z 		},
2024-08-20T22:07:56.6075243Z 		{
2024-08-20T22:07:56.6075790Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6076546Z 			"size": 62501,
2024-08-20T22:07:56.6077205Z 			"digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198"
2024-08-20T22:07:56.6078513Z 		},
2024-08-20T22:07:56.6078889Z 		{
2024-08-20T22:07:56.6079434Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6080099Z 			"size": 1684,
2024-08-20T22:07:56.6080767Z 			"digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a"
2024-08-20T22:07:56.6081683Z 		},
2024-08-20T22:07:56.6082124Z 		{
2024-08-20T22:07:56.6082626Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6083269Z 			"size": 1523,
2024-08-20T22:07:56.6083988Z 			"digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1"
2024-08-20T22:07:56.6084711Z 		},
2024-08-20T22:07:56.6085022Z 		{
2024-08-20T22:07:56.6085628Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6086278Z 			"size": 2528295403,
2024-08-20T22:07:56.6087016Z 			"digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5"
2024-08-20T22:07:56.6087685Z 		},
2024-08-20T22:07:56.6088032Z 		{
2024-08-20T22:07:56.6088667Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6089273Z 			"size": 86016,
2024-08-20T22:07:56.6089919Z 			"digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b"
2024-08-20T22:07:56.6090875Z 		},
2024-08-20T22:07:56.6091226Z 		{
2024-08-20T22:07:56.6091726Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6092453Z 			"size": 1822,
2024-08-20T22:07:56.6093101Z 			"digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea"
2024-08-20T22:07:56.6093836Z 		},
2024-08-20T22:07:56.6094219Z 		{
2024-08-20T22:07:56.6094756Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6095455Z 			"size": 246761563,
2024-08-20T22:07:56.6096151Z 			"digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d"
2024-08-20T22:07:56.6096871Z 		},
2024-08-20T22:07:56.6097243Z 		{
2024-08-20T22:07:56.6097834Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6098450Z 			"size": 544,
2024-08-20T22:07:56.6099099Z 			"digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50"
2024-08-20T22:07:56.6099855Z 		},
2024-08-20T22:07:56.6100241Z 		{
2024-08-20T22:07:56.6100742Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6101519Z 			"size": 1258,
2024-08-20T22:07:56.6102185Z 			"digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c"
2024-08-20T22:07:56.6102918Z 		},
2024-08-20T22:07:56.6103302Z 		{
2024-08-20T22:07:56.6103861Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6104687Z + exit 0
2024-08-20T22:07:56.6105029Z 			"size": 484,
2024-08-20T22:07:56.6105680Z 			"digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656"
2024-08-20T22:07:56.6106426Z 		},
2024-08-20T22:07:56.6106799Z 		{
2024-08-20T22:07:56.6107298Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6108001Z 			"size": 91711766,
2024-08-20T22:07:56.6108677Z 			"digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af"
2024-08-20T22:07:56.6109412Z 		},
2024-08-20T22:07:56.6109785Z 		{
2024-08-20T22:07:56.6110343Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6111102Z 			"size": 3390,
2024-08-20T22:07:56.6111825Z 			"digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2"
2024-08-20T22:07:56.6112500Z 		},
2024-08-20T22:07:56.6112855Z 		{
2024-08-20T22:07:56.6113466Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6114078Z 			"size": 1909,
2024-08-20T22:07:56.6114710Z 			"digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c"
2024-08-20T22:07:56.6115483Z 		},
2024-08-20T22:07:56.6115863Z 		{
2024-08-20T22:07:56.6116381Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6117089Z 			"size": 701,
2024-08-20T22:07:56.6117736Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:07:56.6118469Z 		},
2024-08-20T22:07:56.6118835Z 		{
2024-08-20T22:07:56.6119511Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6120170Z 			"size": 2864254879,
2024-08-20T22:07:56.6120910Z 			"digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933"
2024-08-20T22:07:56.6121606Z 		},
2024-08-20T22:07:56.6121958Z 		{
2024-08-20T22:07:56.6122575Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6123210Z 			"size": 380,
2024-08-20T22:07:56.6123849Z 			"digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6"
2024-08-20T22:07:56.6124630Z 		},
2024-08-20T22:07:56.6125007Z 		{
2024-08-20T22:07:56.6125510Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6126241Z 			"size": 12900,
2024-08-20T22:07:56.6126902Z 			"digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b"
2024-08-20T22:07:56.6127674Z 		},
2024-08-20T22:07:56.6128015Z 		{
2024-08-20T22:07:56.6128553Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6129263Z 			"size": 803,
2024-08-20T22:07:56.6129914Z 			"digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3"
2024-08-20T22:07:56.6130714Z 		},
2024-08-20T22:07:56.6131179Z 		{
2024-08-20T22:07:56.6131746Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6132363Z 			"size": 106,
2024-08-20T22:07:56.6133065Z 			"digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47"
2024-08-20T22:07:56.6133764Z 		},
2024-08-20T22:07:56.6134117Z 		{
2024-08-20T22:07:56.6134763Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6135418Z 			"size": 503,
2024-08-20T22:07:56.6136073Z 			"digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb"
2024-08-20T22:07:56.6136858Z 		},
2024-08-20T22:07:56.6137180Z 		{
2024-08-20T22:07:56.6137735Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6138460Z 			"size": 121477267,
2024-08-20T22:07:56.6139084Z 			"digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e"
2024-08-20T22:07:56.6139811Z 		},
2024-08-20T22:07:56.6140218Z 		{
2024-08-20T22:07:56.6140787Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6141432Z 			"size": 109,
2024-08-20T22:07:56.6142131Z 			"digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58"
2024-08-20T22:07:56.6142859Z 		},
2024-08-20T22:07:56.6143235Z 		{
2024-08-20T22:07:56.6143794Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6144468Z 			"size": 490,
2024-08-20T22:07:56.6145108Z 			"digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791"
2024-08-20T22:07:56.6145828Z 		},
2024-08-20T22:07:56.6146200Z 		{
2024-08-20T22:07:56.6146728Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6147422Z 			"size": 297,
2024-08-20T22:07:56.6148037Z 			"digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a"
2024-08-20T22:07:56.6148844Z 		},
2024-08-20T22:07:56.6149279Z 		{
2024-08-20T22:07:56.6149836Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6150447Z 			"size": 103,
2024-08-20T22:07:56.6151204Z 			"digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706"
2024-08-20T22:07:56.6151918Z 		},
2024-08-20T22:07:56.6152238Z 		{
2024-08-20T22:07:56.6152843Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6153491Z 			"size": 1473,
2024-08-20T22:07:56.6154204Z 			"digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3"
2024-08-20T22:07:56.6154871Z 		},
2024-08-20T22:07:56.6155213Z 		{
2024-08-20T22:07:56.6155801Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6156441Z 			"size": 594729516,
2024-08-20T22:07:56.6157078Z 			"digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c"
2024-08-20T22:07:56.6157940Z 		},
2024-08-20T22:07:56.6158299Z 		{
2024-08-20T22:07:56.6158857Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6159532Z 			"size": 164,
2024-08-20T22:07:56.6160164Z 			"digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714"
2024-08-20T22:07:56.6160919Z 		},
2024-08-20T22:07:56.6161325Z 		{
2024-08-20T22:07:56.6161862Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6162541Z 			"size": 564,
2024-08-20T22:07:56.6163232Z 			"digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8"
2024-08-20T22:07:56.6163925Z 		},
2024-08-20T22:07:56.6164301Z 		{
2024-08-20T22:07:56.6164891Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6165580Z 			"size": 35874203,
2024-08-20T22:07:56.6166220Z 			"digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05"
2024-08-20T22:07:56.6167207Z 		},
2024-08-20T22:07:56.6167660Z 		{
2024-08-20T22:07:56.6168163Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6168883Z 			"size": 104,
2024-08-20T22:07:56.6169519Z 			"digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66"
2024-08-20T22:07:56.6170224Z 		},
2024-08-20T22:07:56.6170681Z 		{
2024-08-20T22:07:56.6171257Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6171904Z 			"size": 425,
2024-08-20T22:07:56.6172633Z 			"digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1"
2024-08-20T22:07:56.6173320Z 		},
2024-08-20T22:07:56.6173671Z 		{
2024-08-20T22:07:56.6174280Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6174932Z 			"size": 20262183,
2024-08-20T22:07:56.6175547Z 			"digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192"
2024-08-20T22:07:56.6176335Z 		},
2024-08-20T22:07:56.6176687Z 		{
2024-08-20T22:07:56.6177189Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6177942Z 			"size": 438,
2024-08-20T22:07:56.6178595Z 			"digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c"
2024-08-20T22:07:56.6179377Z 		},
2024-08-20T22:07:56.6179695Z 		{
2024-08-20T22:07:56.6180226Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6181005Z 			"size": 701,
2024-08-20T22:07:56.6181652Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:07:56.6182321Z 		},
2024-08-20T22:07:56.6182738Z 		{
2024-08-20T22:07:56.6183273Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6183943Z 			"size": 141,
2024-08-20T22:07:56.6184601Z 			"digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097"
2024-08-20T22:07:56.6185349Z 		},
2024-08-20T22:07:56.6185732Z 		{
2024-08-20T22:07:56.6186267Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6187068Z 			"size": 135,
2024-08-20T22:07:56.6187729Z 			"digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1"
2024-08-20T22:07:56.6188481Z 		},
2024-08-20T22:07:56.6188814Z 		{
2024-08-20T22:07:56.6189344Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6190046Z 			"size": 4838068375,
2024-08-20T22:07:56.6190724Z 			"digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090"
2024-08-20T22:07:56.6191440Z 		},
2024-08-20T22:07:56.6191870Z 		{
2024-08-20T22:07:56.6192429Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6193040Z 			"size": 194,
2024-08-20T22:07:56.6193721Z 			"digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855"
2024-08-20T22:07:56.6194447Z 		},
2024-08-20T22:07:56.6194803Z 		{
2024-08-20T22:07:56.6195515Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6196160Z 			"size": 564,
2024-08-20T22:07:56.6196799Z 			"digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf"
2024-08-20T22:07:56.6197586Z 		},
2024-08-20T22:07:56.6197899Z 		{
2024-08-20T22:07:56.6198482Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6199193Z 			"size": 43162921,
2024-08-20T22:07:56.6199849Z 			"digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300"
2024-08-20T22:07:56.6200532Z 		},
2024-08-20T22:07:56.6200948Z 		{
2024-08-20T22:07:56.6201485Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6202116Z 			"size": 106,
2024-08-20T22:07:56.6202804Z 			"digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d"
2024-08-20T22:07:56.6203506Z 		},
2024-08-20T22:07:56.6203931Z 		{
2024-08-20T22:07:56.6204423Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6205098Z 			"size": 1380,
2024-08-20T22:07:56.6205811Z 			"digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2"
2024-08-20T22:07:56.6206510Z 		},
2024-08-20T22:07:56.6206850Z 		{
2024-08-20T22:07:56.6207431Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6208170Z 			"size": 701,
2024-08-20T22:07:56.6208794Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:07:56.6209515Z 		},
2024-08-20T22:07:56.6209891Z 		{
2024-08-20T22:07:56.6210537Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6211257Z 			"size": 139,
2024-08-20T22:07:56.6211921Z 			"digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3"
2024-08-20T22:07:56.6212654Z 		},
2024-08-20T22:07:56.6213053Z 		{
2024-08-20T22:07:56.6213565Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6214222Z 			"size": 118,
2024-08-20T22:07:56.6214930Z 			"digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b"
2024-08-20T22:07:56.6215638Z 		},
2024-08-20T22:07:56.6215949Z 		{
2024-08-20T22:07:56.6216568Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6217209Z 			"size": 4324672728,
2024-08-20T22:07:56.6217841Z 			"digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60"
2024-08-20T22:07:56.6218633Z 		},
2024-08-20T22:07:56.6219017Z 		{
2024-08-20T22:07:56.6219608Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6220240Z 			"size": 174,
2024-08-20T22:07:56.6220905Z 			"digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86"
2024-08-20T22:07:56.6221683Z 		},
2024-08-20T22:07:56.6222030Z 		{
2024-08-20T22:07:56.6222518Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6223247Z 			"size": 907,
2024-08-20T22:07:56.6223892Z 			"digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720"
2024-08-20T22:07:56.6224735Z 		},
2024-08-20T22:07:56.6225107Z 		{
2024-08-20T22:07:56.6225635Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6226339Z 			"size": 701,
2024-08-20T22:07:56.6226977Z 			"digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2"
2024-08-20T22:07:56.6227674Z 		},
2024-08-20T22:07:56.6228036Z 		{
2024-08-20T22:07:56.6228599Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6229203Z 			"size": 134,
2024-08-20T22:07:56.6229858Z 			"digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d"
2024-08-20T22:07:56.6230612Z 		},
2024-08-20T22:07:56.6230974Z 		{
2024-08-20T22:07:56.6231461Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6232153Z 			"size": 32,
2024-08-20T22:07:56.6232862Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:07:56.6233617Z 		},
2024-08-20T22:07:56.6234023Z 		{
2024-08-20T22:07:56.6234562Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6235195Z 			"size": 158,
2024-08-20T22:07:56.6235848Z 			"digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d"
2024-08-20T22:07:56.6236570Z 		},
2024-08-20T22:07:56.6236907Z 		{
2024-08-20T22:07:56.6237506Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6238103Z 			"size": 1841,
2024-08-20T22:07:56.6238729Z 			"digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f"
2024-08-20T22:07:56.6239588Z 		},
2024-08-20T22:07:56.6239928Z 		{
2024-08-20T22:07:56.6240414Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6241182Z 			"size": 7529776,
2024-08-20T22:07:56.6241822Z 			"digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8"
2024-08-20T22:07:56.6242498Z 		},
2024-08-20T22:07:56.6242911Z 		{
2024-08-20T22:07:56.6243436Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6244138Z 			"size": 163,
2024-08-20T22:07:56.6244725Z 			"digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e"
2024-08-20T22:07:56.6245420Z 		},
2024-08-20T22:07:56.6245835Z 		{
2024-08-20T22:07:56.6246435Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6247036Z 			"size": 7943,
2024-08-20T22:07:56.6247733Z 			"digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e"
2024-08-20T22:07:56.6248432Z 		},
2024-08-20T22:07:56.6248737Z 		{
2024-08-20T22:07:56.6249327Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6249970Z 			"size": 8068,
2024-08-20T22:07:56.6250697Z 			"digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a"
2024-08-20T22:07:56.6251478Z 		},
2024-08-20T22:07:56.6251819Z 		{
2024-08-20T22:07:56.6252369Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6253132Z 			"size": 302,
2024-08-20T22:07:56.6253716Z 			"digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f"
2024-08-20T22:07:56.6254433Z 		},
2024-08-20T22:07:56.6254830Z 		{
2024-08-20T22:07:56.6255313Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6255975Z 			"size": 7629638,
2024-08-20T22:07:56.6256660Z 			"digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef"
2024-08-20T22:07:56.6257368Z 		},
2024-08-20T22:07:56.6257677Z 		{
2024-08-20T22:07:56.6258275Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6258934Z 			"size": 108,
2024-08-20T22:07:56.6259562Z 			"digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d"
2024-08-20T22:07:56.6260348Z 		},
2024-08-20T22:07:56.6260714Z 		{
2024-08-20T22:07:56.6261243Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6262051Z 			"size": 54145661,
2024-08-20T22:07:56.6262674Z 			"digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db"
2024-08-20T22:07:56.6263378Z 		},
2024-08-20T22:07:56.6263794Z 		{
2024-08-20T22:07:56.6264281Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6264919Z 			"size": 473,
2024-08-20T22:07:56.6265623Z 			"digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8"
2024-08-20T22:07:56.6266324Z 		},
2024-08-20T22:07:56.6266718Z 		{
2024-08-20T22:07:56.6267651Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6268303Z 			"size": 1374858766,
2024-08-20T22:07:56.6269044Z 			"digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37"
2024-08-20T22:07:56.6269767Z 		},
2024-08-20T22:07:56.6270118Z 		{
2024-08-20T22:07:56.6270791Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6458665Z 			"size": 106,
2024-08-20T22:07:56.6459397Z 			"digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f"
2024-08-20T22:07:56.6460038Z 		},
2024-08-20T22:07:56.6460296Z 		{
2024-08-20T22:07:56.6460737Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6461354Z 			"size": 559,
2024-08-20T22:07:56.6461892Z 			"digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79"
2024-08-20T22:07:56.6462502Z 		},
2024-08-20T22:07:56.6462747Z 		{
2024-08-20T22:07:56.6463185Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6463746Z 			"size": 46248488,
2024-08-20T22:07:56.6464307Z 			"digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4"
2024-08-20T22:07:56.6464921Z 		},
2024-08-20T22:07:56.6465179Z 		{
2024-08-20T22:07:56.6465614Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6466171Z 			"size": 111,
2024-08-20T22:07:56.6466711Z 			"digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4"
2024-08-20T22:07:56.6467623Z 		},
2024-08-20T22:07:56.6467887Z 		{
2024-08-20T22:07:56.6468315Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6468875Z 			"size": 32,
2024-08-20T22:07:56.6469421Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:07:56.6470043Z 		},
2024-08-20T22:07:56.6470289Z 		{
2024-08-20T22:07:56.6470774Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6471332Z 			"size": 32,
2024-08-20T22:07:56.6471870Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:07:56.6472479Z 		},
2024-08-20T22:07:56.6472730Z 		{
2024-08-20T22:07:56.6473164Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6473718Z 			"size": 32,
2024-08-20T22:07:56.6474266Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:07:56.6474887Z 		},
2024-08-20T22:07:56.6475139Z 		{
2024-08-20T22:07:56.6475562Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-08-20T22:07:56.6476111Z 			"size": 32,
2024-08-20T22:07:56.6476649Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-08-20T22:07:56.6477272Z 		}
2024-08-20T22:07:56.6477521Z 	]
2024-08-20T22:07:56.6477768Z }
2024-08-20T22:07:56.6579154Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-08-20T22:07:56.6579635Z tag=${ECR_DOCKER_IMAGE##*/}
2024-08-20T22:07:56.6580168Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2024-08-20T22:07:56.6589430Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:56.6589927Z env:
2024-08-20T22:07:56.6590206Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:56.6591362Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:56.6592531Z ##[endgroup]
2024-08-20T22:07:56.6628215Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:56.6683118Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2024-08-20T22:07:56.6683709Z with:
2024-08-20T22:07:56.6684778Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:56.6686088Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:56.6686607Z env:
2024-08-20T22:07:56.6686893Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:56.6687244Z ##[endgroup]
2024-08-20T22:07:56.6704851Z ##[group]Run set -x
2024-08-20T22:07:56.6705193Z set -x
2024-08-20T22:07:56.6705699Z set +e
2024-08-20T22:07:56.6706003Z 
2024-08-20T22:07:56.6706283Z login() {
2024-08-20T22:07:56.6706951Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2024-08-20T22:07:56.6707672Z }
2024-08-20T22:07:56.6707960Z 
2024-08-20T22:07:56.6708270Z retry () {
2024-08-20T22:07:56.6708672Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2024-08-20T22:07:56.6709120Z }
2024-08-20T22:07:56.6709407Z 
2024-08-20T22:07:56.6709731Z retry login "${DOCKER_REGISTRY}"
2024-08-20T22:07:56.6710148Z 
2024-08-20T22:07:56.6710429Z set -e
2024-08-20T22:07:56.6710910Z # ignore output since only exit code is used for conditional
2024-08-20T22:07:56.6711618Z # only pull docker image if it's not available locally
2024-08-20T22:07:56.6712388Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2024-08-20T22:07:56.6713104Z   retry docker pull "${DOCKER_IMAGE}"
2024-08-20T22:07:56.6713551Z fi
2024-08-20T22:07:56.6722344Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:07:56.6722845Z env:
2024-08-20T22:07:56.6723130Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:07:56.6724279Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:56.6725585Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:56.6726102Z ##[endgroup]
2024-08-20T22:07:56.6755896Z + set +e
2024-08-20T22:07:56.6756452Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:56.6757117Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:56.6759808Z + aws ecr get-login-password --region us-east-1
2024-08-20T22:07:56.6761731Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-08-20T22:07:57.2231786Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-08-20T22:07:57.2232777Z Configure a credential helper to remove this warning. See
2024-08-20T22:07:57.2233621Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-08-20T22:07:57.2234151Z 
2024-08-20T22:07:57.2234279Z Login Succeeded
2024-08-20T22:07:57.2254720Z + set -e
2024-08-20T22:07:57.2256046Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:57.2416511Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:57.2418639Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:07:57.4742441Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks
2024-08-20T22:07:57.4744563Z 7a2c55901189: Pulling fs layer
2024-08-20T22:07:57.4745111Z 224fe954d725: Pulling fs layer
2024-08-20T22:07:57.4745949Z 75722010b82e: Pulling fs layer
2024-08-20T22:07:57.4746454Z d527cbbb87e3: Pulling fs layer
2024-08-20T22:07:57.4746945Z b57676e46aee: Pulling fs layer
2024-08-20T22:07:57.4747435Z a8c1e85b5e14: Pulling fs layer
2024-08-20T22:07:57.4747911Z a41a8d1c11c8: Pulling fs layer
2024-08-20T22:07:57.4748424Z 0c1227890755: Pulling fs layer
2024-08-20T22:07:57.4748932Z d8d1234baab3: Pulling fs layer
2024-08-20T22:07:57.4749453Z 7ed32bc8e469: Pulling fs layer
2024-08-20T22:07:57.4749955Z ec1e7978c1fe: Pulling fs layer
2024-08-20T22:07:57.4750463Z ac23c5fcc1ef: Pulling fs layer
2024-08-20T22:07:57.4750990Z ca426ffc9d2b: Pulling fs layer
2024-08-20T22:07:57.4751784Z 4373aa743caf: Pulling fs layer
2024-08-20T22:07:57.4752295Z 75b2abb16c68: Pulling fs layer
2024-08-20T22:07:57.4752806Z ef1e3ef1e85f: Pulling fs layer
2024-08-20T22:07:57.4753326Z 0bf61114d37b: Pulling fs layer
2024-08-20T22:07:57.4753813Z 2df849f72ba4: Pulling fs layer
2024-08-20T22:07:57.4754190Z 34c47d63c5de: Pulling fs layer
2024-08-20T22:07:57.4754704Z 1fef4bd19792: Pulling fs layer
2024-08-20T22:07:57.4755222Z 6efb96fc87ea: Pulling fs layer
2024-08-20T22:07:57.4755696Z b119c4fc6c58: Pulling fs layer
2024-08-20T22:07:57.4756166Z a41a8d1c11c8: Waiting
2024-08-20T22:07:57.4756646Z 33b639226eee: Pulling fs layer
2024-08-20T22:07:57.4757196Z d3547bd43146: Pulling fs layer
2024-08-20T22:07:57.4757793Z 5645520256fa: Pulling fs layer
2024-08-20T22:07:57.4758207Z 829ed7d91e38: Pulling fs layer
2024-08-20T22:07:57.4758567Z 0c1227890755: Waiting
2024-08-20T22:07:57.4758898Z 385d0421d67c: Pulling fs layer
2024-08-20T22:07:57.4759257Z ca426ffc9d2b: Waiting
2024-08-20T22:07:57.4759580Z 4373aa743caf: Waiting
2024-08-20T22:07:57.4759921Z 7cbfce999d76: Pulling fs layer
2024-08-20T22:07:57.4760288Z 75b2abb16c68: Waiting
2024-08-20T22:07:57.4760600Z d8d1234baab3: Waiting
2024-08-20T22:07:57.4760935Z 6658e558ebe4: Pulling fs layer
2024-08-20T22:07:57.4761361Z 2df849f72ba4: Waiting
2024-08-20T22:07:57.4761782Z 7ed32bc8e469: Waiting
2024-08-20T22:07:57.4762248Z 33444a5a7e01: Pulling fs layer
2024-08-20T22:07:57.4762744Z 34c47d63c5de: Waiting
2024-08-20T22:07:57.4763218Z 7c47122f4e09: Pulling fs layer
2024-08-20T22:07:57.4763726Z ef1e3ef1e85f: Waiting
2024-08-20T22:07:57.4764164Z 1fef4bd19792: Waiting
2024-08-20T22:07:57.4764615Z ec1e7978c1fe: Waiting
2024-08-20T22:07:57.4765060Z 0bf61114d37b: Waiting
2024-08-20T22:07:57.4765384Z 1811ff4d3556: Pulling fs layer
2024-08-20T22:07:57.4765737Z ac23c5fcc1ef: Waiting
2024-08-20T22:07:57.4766064Z 7754ee068b33: Pulling fs layer
2024-08-20T22:07:57.4766420Z b119c4fc6c58: Waiting
2024-08-20T22:07:57.4766740Z 0479fbac3361: Pulling fs layer
2024-08-20T22:07:57.4767603Z 6efb96fc87ea: Waiting
2024-08-20T22:07:57.4768090Z 33b639226eee: Waiting
2024-08-20T22:07:57.4768533Z d3547bd43146: Waiting
2024-08-20T22:07:57.4768984Z d527cbbb87e3: Waiting
2024-08-20T22:07:57.4769427Z 6658e558ebe4: Waiting
2024-08-20T22:07:57.4769870Z 33444a5a7e01: Waiting
2024-08-20T22:07:57.4770313Z 829ed7d91e38: Waiting
2024-08-20T22:07:57.4770874Z b57676e46aee: Waiting
2024-08-20T22:07:57.4771365Z f1d6874dee9c: Pulling fs layer
2024-08-20T22:07:57.4771904Z a8c1e85b5e14: Waiting
2024-08-20T22:07:57.4772363Z afc429e87b59: Pulling fs layer
2024-08-20T22:07:57.4772882Z 7c47122f4e09: Waiting
2024-08-20T22:07:57.4773333Z 019620471c25: Pulling fs layer
2024-08-20T22:07:57.4773865Z 385d0421d67c: Waiting
2024-08-20T22:07:57.4774355Z 7754ee068b33: Waiting
2024-08-20T22:07:57.4774788Z 7cbfce999d76: Waiting
2024-08-20T22:07:57.4775222Z 0479fbac3361: Waiting
2024-08-20T22:07:57.4775640Z 1811ff4d3556: Waiting
2024-08-20T22:07:57.4776076Z f1d6874dee9c: Waiting
2024-08-20T22:07:57.4776551Z 27aeb1b2cd5e: Pulling fs layer
2024-08-20T22:07:57.4777109Z 829c0e19f4b7: Pulling fs layer
2024-08-20T22:07:57.4777616Z 019620471c25: Waiting
2024-08-20T22:07:57.4778071Z 560cd6122a38: Pulling fs layer
2024-08-20T22:07:57.4778584Z 27aeb1b2cd5e: Waiting
2024-08-20T22:07:57.4779020Z 560cd6122a38: Waiting
2024-08-20T22:07:57.4779445Z 829c0e19f4b7: Waiting
2024-08-20T22:07:57.4780155Z 5a4eb19d7219: Pulling fs layer
2024-08-20T22:07:57.4780678Z 5645520256fa: Waiting
2024-08-20T22:07:57.4781175Z da6a838a66c7: Pulling fs layer
2024-08-20T22:07:57.4781733Z cbb79a705353: Pulling fs layer
2024-08-20T22:07:57.4782274Z 4d4fa6802ab3: Pulling fs layer
2024-08-20T22:07:57.4782801Z 0f085d4fa279: Pulling fs layer
2024-08-20T22:07:57.4783296Z cbb79a705353: Waiting
2024-08-20T22:07:57.4783734Z da6a838a66c7: Waiting
2024-08-20T22:07:57.4784171Z 4d4fa6802ab3: Waiting
2024-08-20T22:07:57.4784622Z 4a0412373305: Pulling fs layer
2024-08-20T22:07:57.4785133Z 0f085d4fa279: Waiting
2024-08-20T22:07:57.4785595Z 302c8c177944: Pulling fs layer
2024-08-20T22:07:57.4786128Z 02970347dff6: Pulling fs layer
2024-08-20T22:07:57.4786784Z 4a0412373305: Waiting
2024-08-20T22:07:57.4787213Z 302c8c177944: Waiting
2024-08-20T22:07:57.4787663Z 916c5e858016: Pulling fs layer
2024-08-20T22:07:57.4788193Z c42b3602a1fb: Pulling fs layer
2024-08-20T22:07:57.4788721Z 11fad972ef6c: Pulling fs layer
2024-08-20T22:07:57.4789259Z f957c1624481: Pulling fs layer
2024-08-20T22:07:57.4789776Z 916c5e858016: Waiting
2024-08-20T22:07:57.4790210Z c42b3602a1fb: Waiting
2024-08-20T22:07:57.4790678Z 768c31dca016: Pulling fs layer
2024-08-20T22:07:57.4791188Z 11fad972ef6c: Waiting
2024-08-20T22:07:57.4791622Z f957c1624481: Waiting
2024-08-20T22:07:57.4792046Z 768c31dca016: Waiting
2024-08-20T22:07:57.4792507Z 8c32d0ecebbe: Pulling fs layer
2024-08-20T22:07:57.4793040Z 4f4fb700ef54: Pulling fs layer
2024-08-20T22:07:57.4793584Z 78ee77eddef2: Pulling fs layer
2024-08-20T22:07:57.4794114Z 37513e9b43d7: Pulling fs layer
2024-08-20T22:07:57.4794647Z a7a5392e2bf6: Pulling fs layer
2024-08-20T22:07:57.4795189Z b10de02255c0: Pulling fs layer
2024-08-20T22:07:57.4795699Z 78ee77eddef2: Waiting
2024-08-20T22:07:57.4796166Z e56eeff2db93: Pulling fs layer
2024-08-20T22:07:57.4796676Z 4f4fb700ef54: Waiting
2024-08-20T22:07:57.4797113Z 37513e9b43d7: Waiting
2024-08-20T22:07:57.4797564Z 70ac3e38d0fd: Pulling fs layer
2024-08-20T22:07:57.4798342Z a7a5392e2bf6: Waiting
2024-08-20T22:07:57.4798790Z e56eeff2db93: Waiting
2024-08-20T22:07:57.4799253Z eb09cbf3e416: Pulling fs layer
2024-08-20T22:07:57.4799764Z b10de02255c0: Waiting
2024-08-20T22:07:57.4800247Z ee1273153675: Pulling fs layer
2024-08-20T22:07:57.4800793Z e22979f9e0ee: Pulling fs layer
2024-08-20T22:07:57.4801354Z 6c1f0aabce7c: Pulling fs layer
2024-08-20T22:07:57.4801879Z eb09cbf3e416: Waiting
2024-08-20T22:07:57.4802322Z ee1273153675: Waiting
2024-08-20T22:07:57.4802766Z 6c1f0aabce7c: Waiting
2024-08-20T22:07:57.4803210Z e22979f9e0ee: Waiting
2024-08-20T22:07:57.4803685Z 7ebe49c3a23e: Pulling fs layer
2024-08-20T22:07:57.4804200Z 5d115b29b5d5: Pulling fs layer
2024-08-20T22:07:57.4804698Z 70ac3e38d0fd: Waiting
2024-08-20T22:07:57.4805094Z 7ebe49c3a23e: Waiting
2024-08-20T22:07:57.4805422Z 7f2d7a04fbe6: Pulling fs layer
2024-08-20T22:07:57.4805788Z d79fd7d52698: Pulling fs layer
2024-08-20T22:07:57.4806151Z 9752d6fbbcab: Pulling fs layer
2024-08-20T22:07:57.4806520Z 83808b2eff4d: Pulling fs layer
2024-08-20T22:07:57.4806876Z 5d115b29b5d5: Waiting
2024-08-20T22:07:57.4807183Z 9752d6fbbcab: Waiting
2024-08-20T22:07:57.4807485Z 83808b2eff4d: Waiting
2024-08-20T22:07:57.4807794Z 7f2d7a04fbe6: Waiting
2024-08-20T22:07:57.6116089Z 224fe954d725: Verifying Checksum
2024-08-20T22:07:57.6116561Z 224fe954d725: Download complete
2024-08-20T22:07:57.6828777Z d527cbbb87e3: Verifying Checksum
2024-08-20T22:07:57.6829246Z d527cbbb87e3: Download complete
2024-08-20T22:07:57.7544904Z b57676e46aee: Download complete
2024-08-20T22:07:57.8095946Z 7a2c55901189: Verifying Checksum
2024-08-20T22:07:57.8096432Z 7a2c55901189: Download complete
2024-08-20T22:07:57.8800833Z a41a8d1c11c8: Verifying Checksum
2024-08-20T22:07:57.8801355Z a41a8d1c11c8: Download complete
2024-08-20T22:07:57.9530963Z 0c1227890755: Verifying Checksum
2024-08-20T22:07:57.9531566Z 0c1227890755: Download complete
2024-08-20T22:07:58.0315744Z d8d1234baab3: Download complete
2024-08-20T22:07:58.0866609Z 75722010b82e: Verifying Checksum
2024-08-20T22:07:58.0867390Z 75722010b82e: Download complete
2024-08-20T22:07:58.1691507Z ec1e7978c1fe: Verifying Checksum
2024-08-20T22:07:58.1692051Z ec1e7978c1fe: Download complete
2024-08-20T22:07:58.2389715Z ac23c5fcc1ef: Verifying Checksum
2024-08-20T22:07:58.2390211Z ac23c5fcc1ef: Download complete
2024-08-20T22:07:58.9830034Z 7a2c55901189: Pull complete
2024-08-20T22:07:59.2864309Z 224fe954d725: Pull complete
2024-08-20T22:08:00.2105081Z 75722010b82e: Pull complete
2024-08-20T22:08:00.2267541Z d527cbbb87e3: Pull complete
2024-08-20T22:08:00.2416386Z b57676e46aee: Pull complete
2024-08-20T22:08:00.7612197Z ca426ffc9d2b: Verifying Checksum
2024-08-20T22:08:00.7612828Z ca426ffc9d2b: Download complete
2024-08-20T22:08:00.8472768Z 4373aa743caf: Verifying Checksum
2024-08-20T22:08:00.8473210Z 4373aa743caf: Download complete
2024-08-20T22:08:00.9589067Z 75b2abb16c68: Verifying Checksum
2024-08-20T22:08:00.9589527Z 75b2abb16c68: Download complete
2024-08-20T22:08:01.0514495Z ef1e3ef1e85f: Verifying Checksum
2024-08-20T22:08:01.0514944Z ef1e3ef1e85f: Download complete
2024-08-20T22:08:02.0422533Z 0bf61114d37b: Verifying Checksum
2024-08-20T22:08:02.0423362Z 0bf61114d37b: Download complete
2024-08-20T22:08:02.1346729Z 2df849f72ba4: Download complete
2024-08-20T22:08:02.2207067Z 34c47d63c5de: Download complete
2024-08-20T22:08:02.2898580Z 1fef4bd19792: Download complete
2024-08-20T22:08:11.0974915Z a8c1e85b5e14: Verifying Checksum
2024-08-20T22:08:11.0975409Z a8c1e85b5e14: Download complete
2024-08-20T22:08:11.1761624Z b119c4fc6c58: Download complete
2024-08-20T22:08:11.2719912Z 33b639226eee: Download complete
2024-08-20T22:08:11.3735947Z d3547bd43146: Verifying Checksum
2024-08-20T22:08:11.3736736Z d3547bd43146: Download complete
2024-08-20T22:08:11.4474229Z 5645520256fa: Verifying Checksum
2024-08-20T22:08:11.4474852Z 5645520256fa: Download complete
2024-08-20T22:08:11.5269392Z 829ed7d91e38: Download complete
2024-08-20T22:08:12.7925016Z 385d0421d67c: Verifying Checksum
2024-08-20T22:08:12.7925681Z 385d0421d67c: Download complete
2024-08-20T22:08:12.8603521Z 7cbfce999d76: Verifying Checksum
2024-08-20T22:08:12.8604120Z 7cbfce999d76: Download complete
2024-08-20T22:08:12.9382834Z 6658e558ebe4: Verifying Checksum
2024-08-20T22:08:12.9383459Z 6658e558ebe4: Download complete
2024-08-20T22:08:13.0093494Z 33444a5a7e01: Download complete
2024-08-20T22:08:13.1023923Z 7c47122f4e09: Verifying Checksum
2024-08-20T22:08:13.1024597Z 7c47122f4e09: Download complete
2024-08-20T22:08:13.1777638Z 1811ff4d3556: Verifying Checksum
2024-08-20T22:08:13.1778273Z 1811ff4d3556: Download complete
2024-08-20T22:08:19.1775350Z 7754ee068b33: Verifying Checksum
2024-08-20T22:08:19.1775896Z 7754ee068b33: Download complete
2024-08-20T22:08:19.2676951Z 0479fbac3361: Download complete
2024-08-20T22:08:19.3440193Z f1d6874dee9c: Verifying Checksum
2024-08-20T22:08:19.3440686Z f1d6874dee9c: Download complete
2024-08-20T22:08:19.7710645Z afc429e87b59: Verifying Checksum
2024-08-20T22:08:19.7711297Z afc429e87b59: Download complete
2024-08-20T22:08:19.8465035Z 019620471c25: Verifying Checksum
2024-08-20T22:08:19.8465531Z 019620471c25: Download complete
2024-08-20T22:08:19.9317612Z 27aeb1b2cd5e: Verifying Checksum
2024-08-20T22:08:19.9318163Z 27aeb1b2cd5e: Download complete
2024-08-20T22:08:20.1947595Z 829c0e19f4b7: Verifying Checksum
2024-08-20T22:08:20.1948174Z 829c0e19f4b7: Download complete
2024-08-20T22:08:20.2797977Z 560cd6122a38: Verifying Checksum
2024-08-20T22:08:20.3592830Z 5a4eb19d7219: Verifying Checksum
2024-08-20T22:08:20.3593469Z 5a4eb19d7219: Download complete
2024-08-20T22:08:20.4279030Z da6a838a66c7: Verifying Checksum
2024-08-20T22:08:20.4279483Z da6a838a66c7: Download complete
2024-08-20T22:08:23.3709902Z 7ed32bc8e469: Verifying Checksum
2024-08-20T22:08:23.3710563Z 7ed32bc8e469: Download complete
2024-08-20T22:08:23.4523180Z 4d4fa6802ab3: Verifying Checksum
2024-08-20T22:08:23.5300294Z 0f085d4fa279: Verifying Checksum
2024-08-20T22:08:23.5300816Z 0f085d4fa279: Download complete
2024-08-20T22:08:24.0274097Z 4a0412373305: Verifying Checksum
2024-08-20T22:08:24.0274579Z 4a0412373305: Download complete
2024-08-20T22:08:24.1129889Z 302c8c177944: Verifying Checksum
2024-08-20T22:08:24.1130523Z 302c8c177944: Download complete
2024-08-20T22:08:24.1626854Z a8c1e85b5e14: Pull complete
2024-08-20T22:08:24.1918917Z 02970347dff6: Verifying Checksum
2024-08-20T22:08:24.1919379Z 02970347dff6: Download complete
2024-08-20T22:08:24.2635694Z 916c5e858016: Verifying Checksum
2024-08-20T22:08:24.2636222Z 916c5e858016: Download complete
2024-08-20T22:08:24.3795113Z a41a8d1c11c8: Pull complete
2024-08-20T22:08:24.3887716Z c42b3602a1fb: Verifying Checksum
2024-08-20T22:08:24.3888223Z c42b3602a1fb: Download complete
2024-08-20T22:08:24.5912594Z 0c1227890755: Pull complete
2024-08-20T22:08:24.8201500Z d8d1234baab3: Pull complete
2024-08-20T22:08:30.9785064Z 6efb96fc87ea: Verifying Checksum
2024-08-20T22:08:30.9785753Z 6efb96fc87ea: Download complete
2024-08-20T22:08:31.0577539Z f957c1624481: Verifying Checksum
2024-08-20T22:08:31.0578188Z f957c1624481: Download complete
2024-08-20T22:08:31.1214485Z 768c31dca016: Verifying Checksum
2024-08-20T22:08:31.1215148Z 768c31dca016: Download complete
2024-08-20T22:08:31.1908222Z 8c32d0ecebbe: Verifying Checksum
2024-08-20T22:08:31.1908705Z 8c32d0ecebbe: Download complete
2024-08-20T22:08:31.1999685Z 4f4fb700ef54: Verifying Checksum
2024-08-20T22:08:31.2000131Z 4f4fb700ef54: Download complete
2024-08-20T22:08:31.2766619Z 78ee77eddef2: Verifying Checksum
2024-08-20T22:08:31.2767389Z 78ee77eddef2: Download complete
2024-08-20T22:08:31.3566202Z 37513e9b43d7: Verifying Checksum
2024-08-20T22:08:31.3566774Z 37513e9b43d7: Download complete
2024-08-20T22:08:31.5039158Z a7a5392e2bf6: Verifying Checksum
2024-08-20T22:08:31.5039728Z a7a5392e2bf6: Download complete
2024-08-20T22:08:31.5800271Z b10de02255c0: Verifying Checksum
2024-08-20T22:08:31.5800874Z b10de02255c0: Download complete
2024-08-20T22:08:31.6462062Z e56eeff2db93: Download complete
2024-08-20T22:08:31.7209282Z 70ac3e38d0fd: Verifying Checksum
2024-08-20T22:08:31.7209720Z 70ac3e38d0fd: Download complete
2024-08-20T22:08:31.8010086Z eb09cbf3e416: Download complete
2024-08-20T22:08:31.9244817Z ee1273153675: Verifying Checksum
2024-08-20T22:08:31.9245308Z ee1273153675: Download complete
2024-08-20T22:08:31.9848510Z e22979f9e0ee: Verifying Checksum
2024-08-20T22:08:31.9849063Z e22979f9e0ee: Download complete
2024-08-20T22:08:32.5833630Z 6c1f0aabce7c: Verifying Checksum
2024-08-20T22:08:32.5834217Z 6c1f0aabce7c: Download complete
2024-08-20T22:08:32.6525226Z 7ebe49c3a23e: Download complete
2024-08-20T22:08:58.7709406Z 5d115b29b5d5: Verifying Checksum
2024-08-20T22:08:58.7710010Z 5d115b29b5d5: Download complete
2024-08-20T22:08:58.8573803Z 7f2d7a04fbe6: Verifying Checksum
2024-08-20T22:08:58.8574842Z 7f2d7a04fbe6: Download complete
2024-08-20T22:08:58.9332992Z d79fd7d52698: Verifying Checksum
2024-08-20T22:08:58.9333558Z d79fd7d52698: Download complete
2024-08-20T22:09:00.2137710Z 9752d6fbbcab: Verifying Checksum
2024-08-20T22:09:00.2138270Z 9752d6fbbcab: Download complete
2024-08-20T22:09:00.2861942Z 83808b2eff4d: Download complete
2024-08-20T22:09:39.6049145Z cbb79a705353: Verifying Checksum
2024-08-20T22:09:39.6049744Z cbb79a705353: Download complete
2024-08-20T22:09:39.7820749Z 11fad972ef6c: Verifying Checksum
2024-08-20T22:09:39.7821320Z 11fad972ef6c: Download complete
2024-08-20T22:09:54.2001266Z 7ed32bc8e469: Pull complete
2024-08-20T22:09:54.4272226Z ec1e7978c1fe: Pull complete
2024-08-20T22:09:54.6563194Z ac23c5fcc1ef: Pull complete
2024-08-20T22:10:03.2784458Z ca426ffc9d2b: Pull complete
2024-08-20T22:10:03.4977904Z 4373aa743caf: Pull complete
2024-08-20T22:10:03.7110893Z 75b2abb16c68: Pull complete
2024-08-20T22:10:03.9341469Z ef1e3ef1e85f: Pull complete
2024-08-20T22:10:06.4740785Z 0bf61114d37b: Pull complete
2024-08-20T22:10:06.5298664Z 2df849f72ba4: Pull complete
2024-08-20T22:10:06.6114894Z 34c47d63c5de: Pull complete
2024-08-20T22:10:06.6688927Z 1fef4bd19792: Pull complete
2024-08-20T22:11:05.2991351Z 6efb96fc87ea: Pull complete
2024-08-20T22:11:05.5281639Z b119c4fc6c58: Pull complete
2024-08-20T22:11:05.7749786Z 33b639226eee: Pull complete
2024-08-20T22:11:05.9994487Z d3547bd43146: Pull complete
2024-08-20T22:11:06.2209291Z 5645520256fa: Pull complete
2024-08-20T22:11:06.4342231Z 829ed7d91e38: Pull complete
2024-08-20T22:11:10.0704878Z 385d0421d67c: Pull complete
2024-08-20T22:11:10.2509620Z 7cbfce999d76: Pull complete
2024-08-20T22:11:10.4047382Z 6658e558ebe4: Pull complete
2024-08-20T22:11:10.5730296Z 33444a5a7e01: Pull complete
2024-08-20T22:11:10.7597256Z 7c47122f4e09: Pull complete
2024-08-20T22:11:10.9776754Z 1811ff4d3556: Pull complete
2024-08-20T22:11:21.1928212Z 7754ee068b33: Pull complete
2024-08-20T22:11:21.3349104Z 0479fbac3361: Pull complete
2024-08-20T22:11:21.4708519Z f1d6874dee9c: Pull complete
2024-08-20T22:11:22.4005930Z afc429e87b59: Pull complete
2024-08-20T22:11:22.5978937Z 019620471c25: Pull complete
2024-08-20T22:11:22.8009664Z 27aeb1b2cd5e: Pull complete
2024-08-20T22:11:23.2884815Z 829c0e19f4b7: Pull complete
2024-08-20T22:11:23.4817367Z 560cd6122a38: Pull complete
2024-08-20T22:11:23.8463867Z 5a4eb19d7219: Pull complete
2024-08-20T22:11:24.0229127Z da6a838a66c7: Pull complete
2024-08-20T22:12:02.8112668Z cbb79a705353: Pull complete
2024-08-20T22:12:03.0384749Z 4d4fa6802ab3: Pull complete
2024-08-20T22:12:03.2053536Z 0f085d4fa279: Pull complete
2024-08-20T22:12:05.5539861Z 4a0412373305: Pull complete
2024-08-20T22:12:05.5701327Z 302c8c177944: Pull complete
2024-08-20T22:12:05.5870567Z 02970347dff6: Pull complete
2024-08-20T22:12:05.6227122Z 916c5e858016: Pull complete
2024-08-20T22:12:05.6402598Z c42b3602a1fb: Pull complete
2024-08-20T22:13:27.7256421Z 11fad972ef6c: Pull complete
2024-08-20T22:13:27.9484789Z f957c1624481: Pull complete
2024-08-20T22:13:28.1646776Z 768c31dca016: Pull complete
2024-08-20T22:13:28.5764026Z 8c32d0ecebbe: Pull complete
2024-08-20T22:13:28.8061549Z 4f4fb700ef54: Pull complete
2024-08-20T22:13:29.0297036Z 78ee77eddef2: Pull complete
2024-08-20T22:13:29.2463300Z 37513e9b43d7: Pull complete
2024-08-20T22:13:29.6117269Z a7a5392e2bf6: Pull complete
2024-08-20T22:13:29.8450851Z b10de02255c0: Pull complete
2024-08-20T22:13:30.0675140Z e56eeff2db93: Pull complete
2024-08-20T22:13:30.2884257Z 70ac3e38d0fd: Pull complete
2024-08-20T22:13:30.5162568Z eb09cbf3e416: Pull complete
2024-08-20T22:13:31.9211531Z ee1273153675: Pull complete
2024-08-20T22:13:32.1443062Z e22979f9e0ee: Pull complete
2024-08-20T22:13:34.3666121Z 6c1f0aabce7c: Pull complete
2024-08-20T22:13:34.5915289Z 7ebe49c3a23e: Pull complete
2024-08-20T22:13:48.0119036Z 5d115b29b5d5: Pull complete
2024-08-20T22:13:48.2158660Z 7f2d7a04fbe6: Pull complete
2024-08-20T22:13:48.4164361Z d79fd7d52698: Pull complete
2024-08-20T22:13:49.3040353Z 9752d6fbbcab: Pull complete
2024-08-20T22:13:49.5174403Z 83808b2eff4d: Pull complete
2024-08-20T22:13:50.4625375Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402
2024-08-20T22:13:50.5062309Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:13:50.5338310Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:13:50.5439154Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-08-20T22:13:50.5440067Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-08-20T22:13:50.5451779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:13:50.5452283Z env:
2024-08-20T22:13:50.5452556Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:13:50.5452908Z ##[endgroup]
2024-08-20T22:13:50.5634613Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2024-08-20T22:13:50.5635158Z with:
2024-08-20T22:13:50.5635441Z   driver-version: 550.54.15
2024-08-20T22:13:50.5635779Z env:
2024-08-20T22:13:50.5636053Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:13:50.5636399Z ##[endgroup]
2024-08-20T22:13:50.5785108Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:13:50.5785666Z with:
2024-08-20T22:13:50.5785946Z   timeout_minutes: 10
2024-08-20T22:13:50.5786270Z   max_attempts: 3
2024-08-20T22:13:50.5818632Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2024-08-20T22:13:50.5852442Z   retry_wait_seconds: 10
2024-08-20T22:13:50.5852813Z   polling_interval_seconds: 1
2024-08-20T22:13:50.5853187Z   warning_on_retry: true
2024-08-20T22:13:50.5853541Z   continue_on_error: false
2024-08-20T22:13:50.5853878Z env:
2024-08-20T22:13:50.5854159Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:13:50.5854510Z   DRIVER_VERSION: 550.54.15
2024-08-20T22:13:50.5854863Z ##[endgroup]
2024-08-20T22:13:50.6664779Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-08-20T22:13:50.6665397Z + pre_install_nvidia_driver_amzn2
2024-08-20T22:13:50.6665926Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-08-20T22:13:50.9953793Z No match for argument: nvidia-driver-latest-dkms
2024-08-20T22:13:50.9954310Z No packages marked for removal.
2024-08-20T22:13:51.0019318Z Dependencies resolved.
2024-08-20T22:13:51.0029146Z Nothing to do.
2024-08-20T22:13:51.0029465Z Complete!
2024-08-20T22:13:51.0935658Z + install_nvidia_driver_common
2024-08-20T22:13:51.0938769Z + echo 'Before installing NVIDIA driver'
2024-08-20T22:13:51.0939245Z Before installing NVIDIA driver
2024-08-20T22:13:51.0940003Z + lspci
2024-08-20T22:13:51.1051497Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-08-20T22:13:51.1054561Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-08-20T22:13:51.1056028Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-08-20T22:13:51.1056844Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-08-20T22:13:51.1057658Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-08-20T22:13:51.1058509Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-08-20T22:13:51.1059259Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-08-20T22:13:51.1060077Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-08-20T22:13:51.1060661Z + lsmod
2024-08-20T22:13:51.1103781Z Module                  Size  Used by
2024-08-20T22:13:51.1104614Z xt_nat                 16384  0
2024-08-20T22:13:51.1105276Z veth                   36864  0
2024-08-20T22:13:51.1105868Z nvidia_modeset       1351680  0
2024-08-20T22:13:51.1106497Z video                  65536  1 nvidia_modeset
2024-08-20T22:13:51.1107092Z wmi                    36864  1 video
2024-08-20T22:13:51.1107690Z nvidia_uvm           4706304  0
2024-08-20T22:13:51.1108503Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-08-20T22:13:51.1109313Z drm                   602112  1 nvidia
2024-08-20T22:13:51.1110069Z drm_panel_orientation_quirks    28672  1 drm
2024-08-20T22:13:51.1110834Z backlight              24576  3 video,drm,nvidia_modeset
2024-08-20T22:13:51.1111570Z i2c_core              106496  2 nvidia,drm
2024-08-20T22:13:51.1112185Z xt_conntrack           16384  1
2024-08-20T22:13:51.1112726Z nft_chain_nat          16384  3
2024-08-20T22:13:51.1113267Z xt_MASQUERADE          20480  1
2024-08-20T22:13:51.1113963Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-08-20T22:13:51.1114893Z nf_conntrack_netlink    57344  0
2024-08-20T22:13:51.1115795Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-08-20T22:13:51.1116906Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-08-20T22:13:51.1117617Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-08-20T22:13:51.1118340Z xfrm_user              57344  1
2024-08-20T22:13:51.1118956Z xfrm_algo              16384  1 xfrm_user
2024-08-20T22:13:51.1119543Z xt_addrtype            16384  2
2024-08-20T22:13:51.1120059Z nft_compat             20480  4
2024-08-20T22:13:51.1120684Z nf_tables             307200  57 nft_compat,nft_chain_nat
2024-08-20T22:13:51.1121530Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-08-20T22:13:51.1122239Z br_netfilter           36864  0
2024-08-20T22:13:51.1122801Z bridge                307200  1 br_netfilter
2024-08-20T22:13:51.1123387Z stp                    16384  1 bridge
2024-08-20T22:13:51.1123990Z llc                    16384  2 bridge,stp
2024-08-20T22:13:51.1124536Z overlay               167936  0
2024-08-20T22:13:51.1124927Z tls                   114688  0
2024-08-20T22:13:51.1125415Z nls_ascii              16384  1
2024-08-20T22:13:51.1125908Z nls_cp437              20480  1
2024-08-20T22:13:51.1126378Z vfat                   24576  1
2024-08-20T22:13:51.1126852Z fat                    86016  1 vfat
2024-08-20T22:13:51.1127302Z ena                   167936  0
2024-08-20T22:13:51.1127679Z ptp                    36864  1 ena
2024-08-20T22:13:51.1128092Z pps_core               24576  1 ptp
2024-08-20T22:13:51.1128557Z ghash_clmulni_intel    16384  0
2024-08-20T22:13:51.1128938Z aesni_intel           393216  0
2024-08-20T22:13:51.1129308Z i8042                  45056  0
2024-08-20T22:13:51.1129676Z serio                  28672  3 i8042
2024-08-20T22:13:51.1130102Z crypto_simd            16384  1 aesni_intel
2024-08-20T22:13:51.1130799Z cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
2024-08-20T22:13:51.1131304Z button                 24576  0
2024-08-20T22:13:51.1131666Z sunrpc                692224  1
2024-08-20T22:13:51.1132045Z sch_fq_codel           20480  17
2024-08-20T22:13:51.1132423Z dm_mod                188416  0
2024-08-20T22:13:51.1132790Z fuse                  163840  1
2024-08-20T22:13:51.1133153Z configfs               57344  1
2024-08-20T22:13:51.1133521Z loop                   36864  0
2024-08-20T22:13:51.1133900Z dax                    45056  1 dm_mod
2024-08-20T22:13:51.1134294Z dmi_sysfs              20480  0
2024-08-20T22:13:51.1134667Z crc32_pclmul           16384  0
2024-08-20T22:13:51.1135042Z crc32c_intel           24576  0
2024-08-20T22:13:51.1135412Z efivarfs               24576  1
2024-08-20T22:13:51.1135767Z + modinfo nvidia
2024-08-20T22:13:51.1136471Z filename:       /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-08-20T22:13:51.1137158Z alias:          char-major-195-*
2024-08-20T22:13:51.1137682Z version:        550.54.15
2024-08-20T22:13:51.1138043Z supported:      external
2024-08-20T22:13:51.1138416Z license:        NVIDIA
2024-08-20T22:13:51.1138791Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-08-20T22:13:51.1139264Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-08-20T22:13:51.1139720Z srcversion:     833721318DA517F0C2FEC97
2024-08-20T22:13:51.1140183Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-08-20T22:13:51.1140678Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-08-20T22:13:51.1141152Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-08-20T22:13:51.1141638Z depends:        i2c-core,drm
2024-08-20T22:13:51.1141997Z retpoline:      Y
2024-08-20T22:13:51.1142303Z name:           nvidia
2024-08-20T22:13:51.1142906Z vermagic:       6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-08-20T22:13:51.1143601Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-08-20T22:13:51.1144251Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-08-20T22:13:51.1144980Z parm:           NVreg_ResmanDebugLevel:int
2024-08-20T22:13:51.1145417Z parm:           NVreg_RmLogonRC:int
2024-08-20T22:13:51.1145848Z parm:           NVreg_ModifyDeviceFiles:int
2024-08-20T22:13:51.1146304Z parm:           NVreg_DeviceFileUID:int
2024-08-20T22:13:51.1146737Z parm:           NVreg_DeviceFileGID:int
2024-08-20T22:13:51.1147166Z parm:           NVreg_DeviceFileMode:int
2024-08-20T22:13:51.1147690Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-08-20T22:13:51.1148252Z parm:           NVreg_UsePageAttributeTable:int
2024-08-20T22:13:51.1148733Z parm:           NVreg_EnablePCIeGen3:int
2024-08-20T22:13:51.1149155Z parm:           NVreg_EnableMSI:int
2024-08-20T22:13:51.1149571Z parm:           NVreg_TCEBypassMode:int
2024-08-20T22:13:51.1150025Z parm:           NVreg_EnableStreamMemOPs:int
2024-08-20T22:13:51.1150543Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-08-20T22:13:51.1151129Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-08-20T22:13:51.1151692Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-08-20T22:13:51.1152299Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-08-20T22:13:51.1152889Z parm:           NVreg_DynamicPowerManagement:int
2024-08-20T22:13:51.1153496Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-08-20T22:13:51.1154083Z parm:           NVreg_EnableGpuFirmware:int
2024-08-20T22:13:51.1154574Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-08-20T22:13:51.1155101Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-08-20T22:13:51.1155646Z parm:           NVreg_EnableUserNUMAManagement:int
2024-08-20T22:13:51.1156136Z parm:           NVreg_MemoryPoolSize:int
2024-08-20T22:13:51.1156603Z parm:           NVreg_KMallocHeapMaxSize:int
2024-08-20T22:13:51.1157078Z parm:           NVreg_VMallocHeapMaxSize:int
2024-08-20T22:13:51.1157544Z parm:           NVreg_IgnoreMMIOCheck:int
2024-08-20T22:13:51.1157992Z parm:           NVreg_NvLinkDisable:int
2024-08-20T22:13:51.1158496Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-08-20T22:13:51.1159011Z parm:           NVreg_RegisterPCIDriver:int
2024-08-20T22:13:51.1159488Z parm:           NVreg_EnableResizableBar:int
2024-08-20T22:13:51.1159972Z parm:           NVreg_EnableDbgBreakpoint:int
2024-08-20T22:13:51.1160468Z parm:           NVreg_EnableNonblockingOpen:int
2024-08-20T22:13:51.1160942Z parm:           NVreg_RegistryDwords:charp
2024-08-20T22:13:51.1161439Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-08-20T22:13:51.1161909Z parm:           NVreg_RmMsg:charp
2024-08-20T22:13:51.1162309Z parm:           NVreg_GpuBlacklist:charp
2024-08-20T22:13:51.1162777Z parm:           NVreg_TemporaryFilePath:charp
2024-08-20T22:13:51.1163246Z parm:           NVreg_ExcludedGpus:charp
2024-08-20T22:13:51.1163704Z parm:           NVreg_DmaRemapPeerMmio:int
2024-08-20T22:13:51.1164175Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-08-20T22:13:51.1164748Z parm:           NVreg_ImexChannelCount:int
2024-08-20T22:13:51.1165201Z parm:           rm_firmware_active:charp
2024-08-20T22:13:51.1165612Z + HAS_NVIDIA_DRIVER=0
2024-08-20T22:13:51.1165993Z ++ command -v nvidia-smi
2024-08-20T22:13:51.1166388Z + '[' -x /usr/bin/nvidia-smi ']'
2024-08-20T22:13:51.1166753Z + set +e
2024-08-20T22:13:51.1167563Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-08-20T22:13:51.1434695Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-08-20T22:13:51.1435269Z + NVIDIA_SMI_STATUS=0
2024-08-20T22:13:51.1435807Z + '[' 0 -ne 0 ']'
2024-08-20T22:13:51.1436308Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-08-20T22:13:51.1436730Z + HAS_NVIDIA_DRIVER=1
2024-08-20T22:13:51.1437725Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-08-20T22:13:51.1438534Z + set -e
2024-08-20T22:13:51.1438849Z + '[' 1 -eq 0 ']'
2024-08-20T22:13:51.1439429Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-08-20T22:13:51.1440415Z + post_install_nvidia_driver_common
2024-08-20T22:13:51.1440947Z + sudo modprobe nvidia
2024-08-20T22:13:51.2661522Z + echo 'After installing NVIDIA driver'
2024-08-20T22:13:51.2662081Z + lspci
2024-08-20T22:13:51.2662478Z After installing NVIDIA driver
2024-08-20T22:13:51.2768363Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-08-20T22:13:51.2769364Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-08-20T22:13:51.2770294Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-08-20T22:13:51.2771370Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-08-20T22:13:51.2772364Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2024-08-20T22:13:51.2773151Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-08-20T22:13:51.2773877Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2024-08-20T22:13:51.2774699Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-08-20T22:13:51.2775290Z + lsmod
2024-08-20T22:13:51.2805243Z Module                  Size  Used by
2024-08-20T22:13:51.2805804Z xt_nat                 16384  0
2024-08-20T22:13:51.2806290Z veth                   36864  0
2024-08-20T22:13:51.2806791Z nvidia_modeset       1351680  0
2024-08-20T22:13:51.2807320Z video                  65536  1 nvidia_modeset
2024-08-20T22:13:51.2807884Z wmi                    36864  1 video
2024-08-20T22:13:51.2808413Z nvidia_uvm           4706304  0
2024-08-20T22:13:51.2808860Z nvidia              54071296  7 nvidia_uvm,nvidia_modeset
2024-08-20T22:13:51.2809338Z drm                   602112  1 nvidia
2024-08-20T22:13:51.2809773Z drm_panel_orientation_quirks    28672  1 drm
2024-08-20T22:13:51.2810293Z backlight              24576  3 video,drm,nvidia_modeset
2024-08-20T22:13:51.2810883Z i2c_core              106496  2 nvidia,drm
2024-08-20T22:13:51.2811330Z xt_conntrack           16384  1
2024-08-20T22:13:51.2811726Z nft_chain_nat          16384  3
2024-08-20T22:13:51.2812096Z xt_MASQUERADE          20480  1
2024-08-20T22:13:51.2812566Z nf_nat                 57344  3 xt_nat,nft_chain_nat,xt_MASQUERADE
2024-08-20T22:13:51.2813077Z nf_conntrack_netlink    57344  0
2024-08-20T22:13:51.2813679Z nf_conntrack          184320  5 xt_conntrack,nf_nat,xt_nat,nf_conntrack_netlink,xt_MASQUERADE
2024-08-20T22:13:51.2814342Z nf_defrag_ipv6         24576  1 nf_conntrack
2024-08-20T22:13:51.2814802Z nf_defrag_ipv4         16384  1 nf_conntrack
2024-08-20T22:13:51.2815233Z xfrm_user              57344  1
2024-08-20T22:13:51.2815619Z xfrm_algo              16384  1 xfrm_user
2024-08-20T22:13:51.2816035Z xt_addrtype            16384  2
2024-08-20T22:13:51.2816412Z nft_compat             20480  4
2024-08-20T22:13:51.2816865Z nf_tables             307200  57 nft_compat,nft_chain_nat
2024-08-20T22:13:51.2817466Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2024-08-20T22:13:51.2818253Z br_netfilter           36864  0
2024-08-20T22:13:51.2818668Z bridge                307200  1 br_netfilter
2024-08-20T22:13:51.2819106Z stp                    16384  1 bridge
2024-08-20T22:13:51.2819523Z llc                    16384  2 bridge,stp
2024-08-20T22:13:51.2819941Z overlay               167936  0
2024-08-20T22:13:51.2820308Z tls                   114688  0
2024-08-20T22:13:51.2820680Z nls_ascii              16384  1
2024-08-20T22:13:51.2821041Z nls_cp437              20480  1
2024-08-20T22:13:51.2821406Z vfat                   24576  1
2024-08-20T22:13:51.2821776Z fat                    86016  1 vfat
2024-08-20T22:13:51.2822158Z ena                   167936  0
2024-08-20T22:13:51.2822526Z ptp                    36864  1 ena
2024-08-20T22:13:51.2822918Z pps_core               24576  1 ptp
2024-08-20T22:13:51.2823369Z ghash_clmulni_intel    16384  0
2024-08-20T22:13:51.2823800Z aesni_intel           393216  0
2024-08-20T22:13:51.2824166Z i8042                  45056  0
2024-08-20T22:13:51.2824540Z serio                  28672  3 i8042
2024-08-20T22:13:51.2825170Z crypto_simd            16384  1 aesni_intel
2024-08-20T22:13:51.2825764Z cryptd                 28672  2 crypto_simd,ghash_clmulni_intel
2024-08-20T22:13:51.2826404Z button                 24576  0
2024-08-20T22:13:51.2826880Z sunrpc                692224  1
2024-08-20T22:13:51.2827354Z sch_fq_codel           20480  17
2024-08-20T22:13:51.2827758Z dm_mod                188416  0
2024-08-20T22:13:51.2828130Z fuse                  163840  1
2024-08-20T22:13:51.2828497Z configfs               57344  1
2024-08-20T22:13:51.2828856Z loop                   36864  0
2024-08-20T22:13:51.2829229Z dax                    45056  1 dm_mod
2024-08-20T22:13:51.2829628Z dmi_sysfs              20480  0
2024-08-20T22:13:51.2830002Z crc32_pclmul           16384  0
2024-08-20T22:13:51.2830362Z crc32c_intel           24576  0
2024-08-20T22:13:51.2830729Z efivarfs               24576  1
2024-08-20T22:13:51.2831088Z + modinfo nvidia
2024-08-20T22:13:51.2831794Z filename:       /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2024-08-20T22:13:51.2832488Z alias:          char-major-195-*
2024-08-20T22:13:51.2832863Z version:        550.54.15
2024-08-20T22:13:51.2833202Z supported:      external
2024-08-20T22:13:51.2833534Z license:        NVIDIA
2024-08-20T22:13:51.2833896Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-08-20T22:13:51.2834368Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-08-20T22:13:51.2834817Z srcversion:     833721318DA517F0C2FEC97
2024-08-20T22:13:51.2835274Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-08-20T22:13:51.2835759Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-08-20T22:13:51.2836243Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-08-20T22:13:51.2836727Z depends:        i2c-core,drm
2024-08-20T22:13:51.2837072Z retpoline:      Y
2024-08-20T22:13:51.2837375Z name:           nvidia
2024-08-20T22:13:51.2837980Z vermagic:       6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 
2024-08-20T22:13:51.2838669Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-08-20T22:13:51.2839296Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-08-20T22:13:51.2839881Z parm:           NVreg_ResmanDebugLevel:int
2024-08-20T22:13:51.2840315Z parm:           NVreg_RmLogonRC:int
2024-08-20T22:13:51.2840739Z parm:           NVreg_ModifyDeviceFiles:int
2024-08-20T22:13:51.2841179Z parm:           NVreg_DeviceFileUID:int
2024-08-20T22:13:51.2841608Z parm:           NVreg_DeviceFileGID:int
2024-08-20T22:13:51.2842037Z parm:           NVreg_DeviceFileMode:int
2024-08-20T22:13:51.2842554Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-08-20T22:13:51.2843098Z parm:           NVreg_UsePageAttributeTable:int
2024-08-20T22:13:51.2843568Z parm:           NVreg_EnablePCIeGen3:int
2024-08-20T22:13:51.2843993Z parm:           NVreg_EnableMSI:int
2024-08-20T22:13:51.2844402Z parm:           NVreg_TCEBypassMode:int
2024-08-20T22:13:51.2844965Z parm:           NVreg_EnableStreamMemOPs:int
2024-08-20T22:13:51.2845494Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-08-20T22:13:51.2846066Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-08-20T22:13:51.2846620Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-08-20T22:13:51.2847214Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-08-20T22:13:51.2847800Z parm:           NVreg_DynamicPowerManagement:int
2024-08-20T22:13:51.2848407Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-08-20T22:13:51.2848991Z parm:           NVreg_EnableGpuFirmware:int
2024-08-20T22:13:51.2849476Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-08-20T22:13:51.2849997Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-08-20T22:13:51.2850531Z parm:           NVreg_EnableUserNUMAManagement:int
2024-08-20T22:13:51.2851086Z parm:           NVreg_MemoryPoolSize:int
2024-08-20T22:13:51.2851543Z parm:           NVreg_KMallocHeapMaxSize:int
2024-08-20T22:13:51.2852112Z parm:           NVreg_VMallocHeapMaxSize:int
2024-08-20T22:13:51.2852572Z parm:           NVreg_IgnoreMMIOCheck:int
2024-08-20T22:13:51.2853009Z parm:           NVreg_NvLinkDisable:int
2024-08-20T22:13:51.2853497Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-08-20T22:13:51.2854015Z parm:           NVreg_RegisterPCIDriver:int
2024-08-20T22:13:51.2854482Z parm:           NVreg_EnableResizableBar:int
2024-08-20T22:13:51.2854954Z parm:           NVreg_EnableDbgBreakpoint:int
2024-08-20T22:13:51.2855444Z parm:           NVreg_EnableNonblockingOpen:int
2024-08-20T22:13:51.2855920Z parm:           NVreg_RegistryDwords:charp
2024-08-20T22:13:51.2856408Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-08-20T22:13:51.2856871Z parm:           NVreg_RmMsg:charp
2024-08-20T22:13:51.2857275Z parm:           NVreg_GpuBlacklist:charp
2024-08-20T22:13:51.2857735Z parm:           NVreg_TemporaryFilePath:charp
2024-08-20T22:13:51.2858246Z parm:           NVreg_ExcludedGpus:charp
2024-08-20T22:13:51.2858701Z parm:           NVreg_DmaRemapPeerMmio:int
2024-08-20T22:13:51.2859173Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-08-20T22:13:51.2859640Z parm:           NVreg_ImexChannelCount:int
2024-08-20T22:13:51.2860077Z parm:           rm_firmware_active:charp
2024-08-20T22:13:51.2860475Z + set +e
2024-08-20T22:13:51.2860782Z + nvidia-smi
2024-08-20T22:13:51.3038330Z Tue Aug 20 22:13:51 2024       
2024-08-20T22:13:51.3039054Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:13:51.3040141Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-08-20T22:13:51.3040943Z |-----------------------------------------+------------------------+----------------------+
2024-08-20T22:13:51.3041725Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-08-20T22:13:51.3042617Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-08-20T22:13:51.3043303Z |                                         |                        |               MIG M. |
2024-08-20T22:13:51.3043843Z |=========================================+========================+======================|
2024-08-20T22:13:51.3295767Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-08-20T22:13:51.3297297Z |  0%   26C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-08-20T22:13:51.3298305Z |                                         |                        |                  N/A |
2024-08-20T22:13:51.3299024Z +-----------------------------------------+------------------------+----------------------+
2024-08-20T22:13:51.3299849Z                                                                                          
2024-08-20T22:13:51.3300737Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:13:51.3301635Z | Processes:                                                                              |
2024-08-20T22:13:51.3302362Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-08-20T22:13:51.3303062Z |        ID   ID                                                               Usage      |
2024-08-20T22:13:51.3303643Z |=========================================================================================|
2024-08-20T22:13:51.3305785Z |  No running processes found                                                             |
2024-08-20T22:13:51.3306805Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:13:51.6439873Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-08-20T22:13:51.6624256Z NVIDIA A10G
2024-08-20T22:13:51.6688616Z + NVIDIA_SMI_STATUS=0
2024-08-20T22:13:51.6689071Z + '[' 0 -eq 0 ']'
2024-08-20T22:13:51.6689475Z + echo 'INFO: Ignoring allowed status 0'
2024-08-20T22:13:51.6689905Z + set -e
2024-08-20T22:13:51.6690459Z INFO: Ignoring allowed status 0
2024-08-20T22:13:51.6699286Z == Installing nvidia container toolkit for amzn2023 ==
2024-08-20T22:13:51.6702323Z + sudo yum install -y yum-utils
2024-08-20T22:13:52.1123225Z Last metadata expiration check: 0:44:38 ago on Tue Aug 20 21:29:14 2024.
2024-08-20T22:13:52.1346231Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed.
2024-08-20T22:13:52.1649188Z Dependencies resolved.
2024-08-20T22:13:52.1773889Z Nothing to do.
2024-08-20T22:13:52.1774228Z Complete!
2024-08-20T22:13:52.2750354Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2024-08-20T22:13:52.2751478Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:13:52.2752774Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:13:52.5472814Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2024-08-20T22:13:52.6244006Z + sudo yum install -y nvidia-docker2
2024-08-20T22:13:53.1470211Z nvidia-container-toolkit                         10 kB/s | 833  B     00:00    
2024-08-20T22:13:53.1692091Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2024-08-20T22:13:53.1993122Z Dependencies resolved.
2024-08-20T22:13:53.2117279Z Nothing to do.
2024-08-20T22:13:53.2117615Z Complete!
2024-08-20T22:13:53.3115614Z + sudo systemctl restart docker
2024-08-20T22:14:51.6744385Z nvidia-persistenced failed to initialize. Check syslog for more details.
2024-08-20T22:14:51.6981124Z Tue Aug 20 22:14:51 2024       
2024-08-20T22:14:51.6981959Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:51.6982909Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-08-20T22:14:51.6983904Z |-----------------------------------------+------------------------+----------------------+
2024-08-20T22:14:51.6984968Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-08-20T22:14:51.6986110Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-08-20T22:14:51.6986919Z |                                         |                        |               MIG M. |
2024-08-20T22:14:51.6987642Z |=========================================+========================+======================|
2024-08-20T22:14:51.7189581Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2024-08-20T22:14:51.7190529Z |  0%   26C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-08-20T22:14:51.7191175Z |                                         |                        |                  N/A |
2024-08-20T22:14:51.7191926Z +-----------------------------------------+------------------------+----------------------+
2024-08-20T22:14:51.7193100Z                                                                                          
2024-08-20T22:14:51.7193939Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:51.7194565Z | Processes:                                                                              |
2024-08-20T22:14:51.7195268Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-08-20T22:14:51.7195954Z |        ID   ID                                                               Usage      |
2024-08-20T22:14:51.7196616Z |=========================================================================================|
2024-08-20T22:14:51.7197468Z |  No running processes found                                                             |
2024-08-20T22:14:51.7198271Z +-----------------------------------------------------------------------------------------+
2024-08-20T22:14:52.7049483Z Command completed after 1 attempt(s).
2024-08-20T22:14:52.7129976Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-08-20T22:14:52.7131060Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-08-20T22:14:52.7131755Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2024-08-20T22:14:52.7132410Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2024-08-20T22:14:52.7146273Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:14:52.7146780Z env:
2024-08-20T22:14:52.7147065Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:52.7147524Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:14:52.7148008Z ##[endgroup]
2024-08-20T22:14:52.9574842Z Defaulting to user installation because normal site-packages is not writeable
2024-08-20T22:14:52.9742905Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1)
2024-08-20T22:14:52.9748047Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84)
2024-08-20T22:14:53.1033696Z Prepare all required actions
2024-08-20T22:14:53.1034210Z Getting action download info
2024-08-20T22:14:53.2137269Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-08-20T22:14:53.3966337Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2024-08-20T22:14:53.5296430Z ##[group]Run ./.github/actions/download-build-artifacts
2024-08-20T22:14:53.5296909Z with:
2024-08-20T22:14:53.5297236Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:14:53.5297689Z   s3-bucket: gha-artifacts
2024-08-20T22:14:53.5298020Z env:
2024-08-20T22:14:53.5298289Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:53.5298722Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:14:53.5299199Z ##[endgroup]
2024-08-20T22:14:53.5333416Z ##[group]Run seemethere/download-artifact-s3@v4
2024-08-20T22:14:53.5333859Z with:
2024-08-20T22:14:53.5334226Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:14:53.5334679Z   s3-bucket: gha-artifacts
2024-08-20T22:14:53.5335031Z   region: us-east-1
2024-08-20T22:14:53.5335328Z env:
2024-08-20T22:14:53.5335601Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:14:53.5336044Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:14:53.5336531Z ##[endgroup]
2024-08-20T22:14:54.0075191Z (node:139425) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-08-20T22:14:54.0075902Z 
2024-08-20T22:14:54.0076165Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-08-20T22:14:54.0076905Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-08-20T22:14:54.0077786Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-08-20T22:14:54.0774435Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/
2024-08-20T22:14:54.0775527Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-08-20T22:15:15.9518425Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-08-20T22:15:15.9525147Z Artifact download has finished successfully
2024-08-20T22:15:15.9875709Z ##[group]Run unzip -o artifacts.zip
2024-08-20T22:15:15.9876160Z unzip -o artifacts.zip
2024-08-20T22:15:15.9885528Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:15.9886042Z env:
2024-08-20T22:15:15.9886323Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:15.9886788Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:15.9887285Z ##[endgroup]
2024-08-20T22:15:15.9934087Z Archive:  artifacts.zip
2024-08-20T22:15:15.9935181Z    creating: dist/
2024-08-20T22:15:18.1205823Z   inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl  
2024-08-20T22:15:18.1206497Z    creating: build/custom_test_artifacts/
2024-08-20T22:15:18.1207090Z    creating: build/custom_test_artifacts/custom-op-build/
2024-08-20T22:15:18.1207830Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-08-20T22:15:18.1209102Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:18.1214950Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:18.1215903Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-08-20T22:15:18.1216859Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:18.1217864Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:18.1218844Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:18.1220722Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:18.1222419Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:18.1223473Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:18.1224488Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:18.1225659Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:18.1226987Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:18.1228843Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:18.1230002Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:18.1231567Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:18.1233110Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:18.1234171Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:18.1235189Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:18.1275327Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:18.1315037Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:18.1316504Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:18.1361889Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:18.1365013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:18.1368453Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:18.1371515Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:18.1373131Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:18.1374541Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:18.1375943Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:18.1377340Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:18.1378707Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:18.1380182Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:18.1381430Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:18.1382662Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:18.1383895Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:18.1385099Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:18.1386319Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:18.1437648Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:18.1498026Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:18.1499215Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:18.1500201Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:18.1501086Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:18.1502033Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:18.1502991Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-08-20T22:15:18.1504027Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2024-08-20T22:15:18.1505206Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2024-08-20T22:15:18.1506333Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2024-08-20T22:15:18.1507381Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2024-08-20T22:15:18.1508468Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2024-08-20T22:15:18.1509559Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2024-08-20T22:15:18.1510649Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2024-08-20T22:15:18.1511737Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2024-08-20T22:15:18.1512818Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2024-08-20T22:15:18.1530030Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2024-08-20T22:15:18.1674532Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2024-08-20T22:15:18.1675546Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-08-20T22:15:18.1676639Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2024-08-20T22:15:18.1677847Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2024-08-20T22:15:18.1679013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2024-08-20T22:15:18.1680098Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2024-08-20T22:15:18.1681224Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2024-08-20T22:15:18.1682425Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2024-08-20T22:15:18.1683755Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2024-08-20T22:15:18.1684904Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2024-08-20T22:15:18.1686028Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2024-08-20T22:15:18.1703994Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2024-08-20T22:15:18.1787850Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2024-08-20T22:15:18.1789050Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:18.1790127Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:18.1791276Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2024-08-20T22:15:18.1792204Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2024-08-20T22:15:18.1793097Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:18.1794000Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2024-08-20T22:15:18.1796293Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2024-08-20T22:15:18.1797254Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2024-08-20T22:15:18.1798489Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2024-08-20T22:15:18.1918205Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2024-08-20T22:15:18.1981714Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2024-08-20T22:15:18.1982515Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-08-20T22:15:18.1983247Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-08-20T22:15:18.1984085Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:18.1990654Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:18.1991596Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-08-20T22:15:18.1992560Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:18.1993588Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:18.1994561Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:18.1995674Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:18.1996922Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:18.1997935Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:18.1998931Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:18.2000460Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:18.2002162Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:18.2004141Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:18.2005287Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:18.2006845Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:18.2008224Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:18.2009274Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:18.2010642Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:18.2050045Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:18.2090925Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:18.2092407Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:18.2137690Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:18.2139106Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:18.2140751Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:18.2142258Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:18.2143673Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:18.2145131Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:18.2146731Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:18.2148342Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:18.2149894Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:18.2151356Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:18.2152771Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:18.2154157Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:18.2155566Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:18.2156922Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:18.2158300Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:18.2212566Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:18.2273452Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:18.2274642Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:18.2275627Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:18.2276486Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:18.2277383Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:18.2278327Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-08-20T22:15:18.2279391Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2024-08-20T22:15:18.2280567Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2024-08-20T22:15:18.2281723Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2024-08-20T22:15:18.2282972Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2024-08-20T22:15:18.2284089Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2024-08-20T22:15:18.2285195Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2024-08-20T22:15:18.2286304Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2024-08-20T22:15:18.2287415Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2024-08-20T22:15:18.2288510Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2024-08-20T22:15:18.2305587Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2024-08-20T22:15:18.2371262Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2024-08-20T22:15:18.2372710Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:18.2373787Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:18.2374760Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2024-08-20T22:15:18.2375757Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2024-08-20T22:15:18.2376857Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:18.2377746Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2024-08-20T22:15:18.2378598Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2024-08-20T22:15:18.2379727Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2024-08-20T22:15:18.2380826Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2024-08-20T22:15:18.2430938Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2024-08-20T22:15:18.2431792Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-08-20T22:15:18.2432689Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-08-20T22:15:18.2433684Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-08-20T22:15:18.2439576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2024-08-20T22:15:18.2440579Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-08-20T22:15:18.2441565Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake  
2024-08-20T22:15:18.2442670Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-08-20T22:15:18.2443700Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-08-20T22:15:18.2444866Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c  
2024-08-20T22:15:18.2446045Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out  
2024-08-20T22:15:18.2447109Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-08-20T22:15:18.2448145Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-08-20T22:15:18.2449782Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2024-08-20T22:15:18.2451665Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out  
2024-08-20T22:15:18.2454393Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin  
2024-08-20T22:15:18.2455801Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake  
2024-08-20T22:15:18.2457241Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin  
2024-08-20T22:15:18.2458490Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake  
2024-08-20T22:15:18.2459595Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-08-20T22:15:18.2460659Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-08-20T22:15:18.2500712Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2024-08-20T22:15:18.2540487Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2024-08-20T22:15:18.2542215Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2024-08-20T22:15:18.2588136Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2024-08-20T22:15:18.2589619Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2024-08-20T22:15:18.2591132Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2024-08-20T22:15:18.2592714Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2024-08-20T22:15:18.2594192Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2024-08-20T22:15:18.2595648Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2024-08-20T22:15:18.2597109Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2024-08-20T22:15:18.2598611Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2024-08-20T22:15:18.2600221Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2024-08-20T22:15:18.2601573Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2024-08-20T22:15:18.2602878Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c  
2024-08-20T22:15:18.2604153Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin  
2024-08-20T22:15:18.2605461Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2024-08-20T22:15:18.2606710Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o  
2024-08-20T22:15:18.2607998Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2024-08-20T22:15:18.2663341Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out  
2024-08-20T22:15:18.2723799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin  
2024-08-20T22:15:18.2725052Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake  
2024-08-20T22:15:18.2726111Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-08-20T22:15:18.2727044Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-08-20T22:15:18.2728024Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2024-08-20T22:15:18.2729231Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-08-20T22:15:18.2730370Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2024-08-20T22:15:18.2731742Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2024-08-20T22:15:18.2733027Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2024-08-20T22:15:18.2734169Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2024-08-20T22:15:18.2735352Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2024-08-20T22:15:18.2736666Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2024-08-20T22:15:18.2737862Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2024-08-20T22:15:18.2739060Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2024-08-20T22:15:18.2740245Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2024-08-20T22:15:18.2741494Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2024-08-20T22:15:18.2866792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2024-08-20T22:15:18.2869697Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-08-20T22:15:18.2871996Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2024-08-20T22:15:18.2873316Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2024-08-20T22:15:18.2874596Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2024-08-20T22:15:18.2875819Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2024-08-20T22:15:18.2877044Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2024-08-20T22:15:18.2878285Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2024-08-20T22:15:18.2879535Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2024-08-20T22:15:18.2880771Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2024-08-20T22:15:18.2881993Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2024-08-20T22:15:18.2896003Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2024-08-20T22:15:18.2951278Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2024-08-20T22:15:18.2952620Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2024-08-20T22:15:18.2953797Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2024-08-20T22:15:18.2954838Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2024-08-20T22:15:18.2955806Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2024-08-20T22:15:18.2956780Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2024-08-20T22:15:18.2957937Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2024-08-20T22:15:18.2959622Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2024-08-20T22:15:18.2961841Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2024-08-20T22:15:18.2962849Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2024-08-20T22:15:18.3066354Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2024-08-20T22:15:18.3109898Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2024-08-20T22:15:18.3110535Z    creating: build/lib/
2024-08-20T22:15:18.3200304Z   inflating: build/lib/libprotobuf-lite.a  
2024-08-20T22:15:18.3654920Z   inflating: build/lib/libprotobuf.a  
2024-08-20T22:15:18.3664781Z   inflating: build/lib/libpthreadpool.a  
2024-08-20T22:15:18.3673355Z   inflating: build/lib/libcpuinfo.a  
2024-08-20T22:15:18.3681491Z   inflating: build/lib/libcpuinfo_internals.a  
2024-08-20T22:15:18.3682140Z   inflating: build/lib/libclog.a     
2024-08-20T22:15:18.3700621Z   inflating: build/lib/libnnpack.a   
2024-08-20T22:15:18.3703224Z   inflating: build/lib/libnnpack_reference_layers.a  
2024-08-20T22:15:18.3768887Z   inflating: build/lib/libgtest.a    
2024-08-20T22:15:18.3843120Z   inflating: build/lib/libbenchmark.a  
2024-08-20T22:15:18.3907281Z   inflating: build/lib/libasmjit.a   
2024-08-20T22:15:18.3914925Z   inflating: build/lib/libittnotify.a  
2024-08-20T22:15:18.3942922Z   inflating: build/lib/libtensorpipe_uv.a  
2024-08-20T22:15:18.4070280Z   inflating: build/lib/libgloo.a     
2024-08-20T22:15:18.4091080Z   inflating: build/lib/libfmt.a      
2024-08-20T22:15:18.4185987Z   inflating: build/lib/libc10.so     
2024-08-20T22:15:18.4187849Z   inflating: build/lib/libcaffe2_nvrtc.so  
2024-08-20T22:15:18.4189035Z   inflating: build/lib/libtorch_global_deps.so  
2024-08-20T22:15:18.4692895Z   inflating: build/lib/libprotoc.a   
2024-08-20T22:15:18.4712452Z   inflating: build/lib/libpytorch_qnnpack.a  
2024-08-20T22:15:18.4730661Z   inflating: build/lib/libgmock.a    
2024-08-20T22:15:18.4731526Z   inflating: build/lib/libgtest_main.a  
2024-08-20T22:15:18.4732296Z   inflating: build/lib/libbenchmark_main.a  
2024-08-20T22:15:18.5122263Z   inflating: build/lib/libgloo_cuda.a  
2024-08-20T22:15:19.5306449Z   inflating: build/lib/libdnnl.a     
2024-08-20T22:15:19.5881386Z   inflating: build/lib/libtensorpipe.a  
2024-08-20T22:15:19.5938818Z   inflating: build/lib/libc10_cuda.so  
2024-08-20T22:15:19.5939832Z   inflating: build/lib/libgmock_main.a  
2024-08-20T22:15:19.7199924Z   inflating: build/lib/libfbgemm.a   
2024-08-20T22:15:19.7455789Z   inflating: build/lib/libtensorpipe_cuda.a  
2024-08-20T22:15:19.7966078Z   inflating: build/lib/libkineto.a   
2024-08-20T22:15:19.8149594Z   inflating: build/lib/libXNNPACK.a  
2024-08-20T22:15:19.8192324Z   inflating: build/lib/libonnx_proto.a  
2024-08-20T22:15:19.8886132Z   inflating: build/lib/libonnx.a     
2024-08-20T22:15:22.3368875Z   inflating: build/lib/libtorch_cpu.so  
2024-08-20T22:15:22.3373468Z   inflating: build/lib/libunbox_lib.a  
2024-08-20T22:15:22.3377752Z   inflating: build/lib/libshm.so     
2024-08-20T22:15:24.4071067Z   inflating: build/lib/libtorch_cuda.so  
2024-08-20T22:15:24.4072320Z   inflating: build/lib/libtorch.so   
2024-08-20T22:15:24.4075258Z   inflating: build/lib/libc10d_cuda_test.so  
2024-08-20T22:15:25.2274421Z   inflating: build/lib/libtorch_cuda_linalg.so  
2024-08-20T22:15:25.4269182Z   inflating: build/lib/libtorch_python.so  
2024-08-20T22:15:25.4341688Z   inflating: build/lib/libtorchbind_test.so  
2024-08-20T22:15:25.4362463Z   inflating: build/lib/libjitbackend_test.so  
2024-08-20T22:15:25.4388353Z   inflating: build/lib/libbackend_with_compiler.so  
2024-08-20T22:15:25.4414580Z   inflating: build/lib/libaoti_custom_ops.so  
2024-08-20T22:15:25.4449685Z   inflating: build/lib/libnnapi_backend.so  
2024-08-20T22:15:25.4450202Z    creating: build/bin/
2024-08-20T22:15:25.4501254Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2024-08-20T22:15:25.4553875Z   inflating: build/bin/c10_DeviceGuard_test  
2024-08-20T22:15:25.4605527Z   inflating: build/bin/c10_Device_test  
2024-08-20T22:15:25.4665007Z   inflating: build/bin/c10_DispatchKeySet_test  
2024-08-20T22:15:25.4719550Z   inflating: build/bin/c10_Scalar_test  
2024-08-20T22:15:25.4768913Z   inflating: build/bin/c10_StreamGuard_test  
2024-08-20T22:15:25.4820752Z   inflating: build/bin/c10_SymInt_test  
2024-08-20T22:15:25.4875609Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2024-08-20T22:15:25.4931698Z   inflating: build/bin/c10_InlineStreamGuard_test  
2024-08-20T22:15:25.4988477Z   inflating: build/bin/c10_SizesAndStrides_test  
2024-08-20T22:15:25.5059668Z   inflating: build/bin/c10_cow_test  
2024-08-20T22:15:25.5113777Z   inflating: build/bin/c10_Bitset_test  
2024-08-20T22:15:25.5163008Z   inflating: build/bin/c10_ConstexprCrc_test  
2024-08-20T22:15:25.5213325Z   inflating: build/bin/c10_DeadlockDetection_test  
2024-08-20T22:15:25.5264640Z   inflating: build/bin/c10_Half_test  
2024-08-20T22:15:25.5321072Z   inflating: build/bin/c10_LeftRight_test  
2024-08-20T22:15:25.5376729Z   inflating: build/bin/c10_Metaprogramming_test  
2024-08-20T22:15:25.5426846Z   inflating: build/bin/c10_Synchronized_test  
2024-08-20T22:15:25.5482919Z   inflating: build/bin/c10_ThreadLocal_test  
2024-08-20T22:15:25.5534814Z   inflating: build/bin/c10_TypeIndex_test  
2024-08-20T22:15:25.5586124Z   inflating: build/bin/c10_TypeList_test  
2024-08-20T22:15:25.5635129Z   inflating: build/bin/c10_TypeTraits_test  
2024-08-20T22:15:25.5687405Z   inflating: build/bin/c10_accumulate_test  
2024-08-20T22:15:25.5742913Z   inflating: build/bin/c10_bfloat16_test  
2024-08-20T22:15:25.5794486Z   inflating: build/bin/c10_bit_cast_test  
2024-08-20T22:15:25.5850445Z   inflating: build/bin/c10_complex_math_test  
2024-08-20T22:15:25.5906420Z   inflating: build/bin/c10_complex_test  
2024-08-20T22:15:25.5959943Z   inflating: build/bin/c10_exception_test  
2024-08-20T22:15:25.6010812Z   inflating: build/bin/c10_flags_test  
2024-08-20T22:15:25.6060657Z   inflating: build/bin/c10_generic_math_test  
2024-08-20T22:15:25.6226309Z   inflating: build/bin/c10_intrusive_ptr_test  
2024-08-20T22:15:25.6277436Z   inflating: build/bin/c10_irange_test  
2024-08-20T22:15:25.6331327Z   inflating: build/bin/c10_lazy_test  
2024-08-20T22:15:25.6388945Z   inflating: build/bin/c10_logging_test  
2024-08-20T22:15:25.6464378Z   inflating: build/bin/c10_optional_test  
2024-08-20T22:15:25.6527786Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2024-08-20T22:15:25.6581576Z   inflating: build/bin/c10_registry_test  
2024-08-20T22:15:25.6733352Z   inflating: build/bin/c10_small_vector_test  
2024-08-20T22:15:25.6785308Z   inflating: build/bin/c10_ssize_test  
2024-08-20T22:15:25.6837415Z   inflating: build/bin/c10_string_util_test  
2024-08-20T22:15:25.6887999Z   inflating: build/bin/c10_tempfile_test  
2024-08-20T22:15:25.6947189Z   inflating: build/bin/c10_string_view_test  
2024-08-20T22:15:25.6996452Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2024-08-20T22:15:25.7052397Z   inflating: build/bin/c10_typeid_test  
2024-08-20T22:15:25.7501084Z   inflating: build/bin/protoc-3.13.0.0  
2024-08-20T22:15:25.7949109Z   inflating: build/bin/protoc        
2024-08-20T22:15:25.8002505Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2024-08-20T22:15:25.8055578Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2024-08-20T22:15:25.8109181Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2024-08-20T22:15:25.8161242Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2024-08-20T22:15:25.8214584Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2024-08-20T22:15:25.8267837Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2024-08-20T22:15:25.8317484Z   inflating: build/bin/c10_cuda_CUDATest  
2024-08-20T22:15:25.8371673Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2024-08-20T22:15:25.8708334Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2024-08-20T22:15:25.9054430Z   inflating: build/bin/vec_test_all_types_AVX512  
2024-08-20T22:15:25.9415279Z   inflating: build/bin/vec_test_all_types_AVX2  
2024-08-20T22:15:25.9467756Z   inflating: build/bin/BackoffTest   
2024-08-20T22:15:25.9521961Z   inflating: build/bin/FileStoreTest  
2024-08-20T22:15:25.9577966Z   inflating: build/bin/TCPStoreTest  
2024-08-20T22:15:25.9631385Z   inflating: build/bin/HashStoreTest  
2024-08-20T22:15:25.9645095Z   inflating: build/bin/ProcessGroupMPITest  
2024-08-20T22:15:25.9699519Z   inflating: build/bin/test_edge_op_registration  
2024-08-20T22:15:25.9703928Z   inflating: build/bin/torch_shm_manager  
2024-08-20T22:15:25.9707040Z   inflating: build/bin/example_allreduce  
2024-08-20T22:15:25.9762564Z   inflating: build/bin/test_dist_autograd  
2024-08-20T22:15:25.9831744Z   inflating: build/bin/test_cpp_rpc  
2024-08-20T22:15:25.9834559Z   inflating: build/bin/parallel_benchmark  
2024-08-20T22:15:25.9902406Z   inflating: build/bin/test_mobile_nnc  
2024-08-20T22:15:25.9911374Z   inflating: build/bin/aot_model_compiler_test  
2024-08-20T22:15:26.0260507Z   inflating: build/bin/test_lazy     
2024-08-20T22:15:26.1450251Z   inflating: build/bin/test_api      
2024-08-20T22:15:26.1524949Z   inflating: build/bin/Dict_test     
2024-08-20T22:15:26.1577551Z   inflating: build/bin/Dimname_test  
2024-08-20T22:15:26.1642749Z   inflating: build/bin/MaybeOwned_test  
2024-08-20T22:15:26.1700304Z   inflating: build/bin/NamedTensor_test  
2024-08-20T22:15:26.1759736Z   inflating: build/bin/apply_utils_test  
2024-08-20T22:15:26.1819347Z   inflating: build/bin/atest         
2024-08-20T22:15:26.1882990Z   inflating: build/bin/basic         
2024-08-20T22:15:26.1938399Z   inflating: build/bin/broadcast_test  
2024-08-20T22:15:26.1989430Z   inflating: build/bin/cpu_allocator_test  
2024-08-20T22:15:26.2048311Z   inflating: build/bin/cpu_generator_test  
2024-08-20T22:15:26.2102420Z   inflating: build/bin/cpu_profiling_allocator_test  
2024-08-20T22:15:26.2196433Z   inflating: build/bin/cpu_rng_test  
2024-08-20T22:15:26.2246621Z   inflating: build/bin/dispatch_key_set_test  
2024-08-20T22:15:26.2297692Z   inflating: build/bin/dlconvertor_test  
2024-08-20T22:15:26.2356600Z   inflating: build/bin/extension_backend_test  
2024-08-20T22:15:26.2412320Z   inflating: build/bin/half_test     
2024-08-20T22:15:26.2508965Z   inflating: build/bin/ivalue_test   
2024-08-20T22:15:26.2559433Z   inflating: build/bin/lazy_tensor_test  
2024-08-20T22:15:26.2614267Z   inflating: build/bin/math_kernel_test  
2024-08-20T22:15:26.2669407Z   inflating: build/bin/memory_format_test  
2024-08-20T22:15:26.2722927Z   inflating: build/bin/memory_overlapping_test  
2024-08-20T22:15:26.2776722Z   inflating: build/bin/mobile_memory_cleanup  
2024-08-20T22:15:26.2833040Z   inflating: build/bin/native_test   
2024-08-20T22:15:26.2884087Z   inflating: build/bin/operator_name_test  
2024-08-20T22:15:26.2935979Z   inflating: build/bin/operators_test  
2024-08-20T22:15:26.2988186Z   inflating: build/bin/packedtensoraccessor_test  
2024-08-20T22:15:26.3055927Z   inflating: build/bin/pow_test      
2024-08-20T22:15:26.3114127Z   inflating: build/bin/quantized_test  
2024-08-20T22:15:26.3164670Z   inflating: build/bin/reduce_ops_test  
2024-08-20T22:15:26.3216223Z   inflating: build/bin/reportMemoryUsage_test  
2024-08-20T22:15:26.3273609Z   inflating: build/bin/scalar_tensor_test  
2024-08-20T22:15:26.3334874Z   inflating: build/bin/scalar_test   
2024-08-20T22:15:26.3387393Z   inflating: build/bin/StorageUtils_test  
2024-08-20T22:15:26.3441145Z   inflating: build/bin/stride_properties_test  
2024-08-20T22:15:26.3520074Z   inflating: build/bin/tensor_iterator_test  
2024-08-20T22:15:26.3575103Z   inflating: build/bin/test_parallel  
2024-08-20T22:15:26.3578238Z   inflating: build/bin/thread_init_test  
2024-08-20T22:15:26.3634283Z   inflating: build/bin/type_ptr_test  
2024-08-20T22:15:26.3694901Z   inflating: build/bin/type_test     
2024-08-20T22:15:26.3747491Z   inflating: build/bin/undefined_tensor_test  
2024-08-20T22:15:26.3749111Z   inflating: build/bin/verify_api_visibility  
2024-08-20T22:15:26.3819138Z   inflating: build/bin/legacy_vmap_test  
2024-08-20T22:15:26.3871304Z   inflating: build/bin/weakref_test  
2024-08-20T22:15:26.3923263Z   inflating: build/bin/wrapdim_test  
2024-08-20T22:15:26.3975098Z   inflating: build/bin/xla_tensor_test  
2024-08-20T22:15:26.4036164Z   inflating: build/bin/IListRef_test  
2024-08-20T22:15:26.4143519Z   inflating: build/bin/List_test     
2024-08-20T22:15:26.4210542Z   inflating: build/bin/KernelFunction_test  
2024-08-20T22:15:26.4333412Z   inflating: build/bin/kernel_function_legacy_test  
2024-08-20T22:15:26.4430810Z   inflating: build/bin/kernel_function_test  
2024-08-20T22:15:26.4560160Z   inflating: build/bin/kernel_lambda_legacy_test  
2024-08-20T22:15:26.4664701Z   inflating: build/bin/kernel_lambda_test  
2024-08-20T22:15:26.4726919Z   inflating: build/bin/kernel_stackbased_test  
2024-08-20T22:15:26.4823535Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2024-08-20T22:15:26.4875896Z   inflating: build/bin/CppSignature_test  
2024-08-20T22:15:26.4925662Z   inflating: build/bin/op_allowlist_test  
2024-08-20T22:15:26.4981486Z   inflating: build/bin/backend_fallback_test  
2024-08-20T22:15:26.5286359Z   inflating: build/bin/op_registration_test  
2024-08-20T22:15:26.5350422Z   inflating: build/bin/inline_container_test  
2024-08-20T22:15:26.5404361Z   inflating: build/bin/cuda_apply_test  
2024-08-20T22:15:26.5456180Z   inflating: build/bin/cuda_allocator_test  
2024-08-20T22:15:26.5516055Z   inflating: build/bin/cuda_atomic_ops_test  
2024-08-20T22:15:26.5570697Z   inflating: build/bin/cuda_caching_host_allocator_test  
2024-08-20T22:15:26.5641085Z   inflating: build/bin/cuda_complex_math_test  
2024-08-20T22:15:26.5701338Z   inflating: build/bin/cuda_complex_test  
2024-08-20T22:15:26.5750755Z   inflating: build/bin/cuda_device_test  
2024-08-20T22:15:26.5809283Z   inflating: build/bin/cuda_cub_test  
2024-08-20T22:15:26.5859911Z   inflating: build/bin/cuda_dlconvertor_test  
2024-08-20T22:15:26.5925040Z   inflating: build/bin/cuda_distributions_test  
2024-08-20T22:15:26.5982332Z   inflating: build/bin/cuda_generator_test  
2024-08-20T22:15:26.6032112Z   inflating: build/bin/cuda_half_test  
2024-08-20T22:15:26.6084058Z   inflating: build/bin/cuda_integer_divider_test  
2024-08-20T22:15:26.6134253Z   inflating: build/bin/cuda_optional_test  
2024-08-20T22:15:26.6185912Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2024-08-20T22:15:26.6239010Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2024-08-20T22:15:26.6289185Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2024-08-20T22:15:26.6350311Z   inflating: build/bin/cuda_stream_test  
2024-08-20T22:15:26.6400644Z   inflating: build/bin/cuda_cudnn_test  
2024-08-20T22:15:26.6452988Z   inflating: build/bin/cuda_vectorized_test  
2024-08-20T22:15:26.6467016Z   inflating: build/bin/tutorial_tensorexpr  
2024-08-20T22:15:26.6533765Z   inflating: build/bin/ProcessGroupGlooTest  
2024-08-20T22:15:26.6591651Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2024-08-20T22:15:26.6655867Z   inflating: build/bin/ProcessGroupNCCLTest  
2024-08-20T22:15:26.6719033Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2024-08-20T22:15:26.7549761Z   inflating: build/bin/test_tensorexpr  
2024-08-20T22:15:26.8133155Z   inflating: build/bin/test_jit      
2024-08-20T22:15:26.8133736Z    creating: .additional_ci_files/
2024-08-20T22:15:26.8195092Z   inflating: .additional_ci_files/test-times.json  
2024-08-20T22:15:26.8435708Z   inflating: .additional_ci_files/test-class-times.json  
2024-08-20T22:15:26.8471887Z ##[group]Run rm artifacts.zip
2024-08-20T22:15:26.8472291Z rm artifacts.zip
2024-08-20T22:15:26.8481340Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:26.8482089Z env:
2024-08-20T22:15:26.8482391Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:26.8482873Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:26.8483371Z ##[endgroup]
2024-08-20T22:15:26.9752598Z ##[group]Run df -H
2024-08-20T22:15:26.9752913Z df -H
2024-08-20T22:15:26.9761693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:26.9762206Z env:
2024-08-20T22:15:26.9762504Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:26.9762976Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:26.9763478Z ##[endgroup]
2024-08-20T22:15:26.9836149Z Filesystem        Size  Used Avail Use% Mounted on
2024-08-20T22:15:26.9836699Z devtmpfs          4.2M     0  4.2M   0% /dev
2024-08-20T22:15:26.9837175Z tmpfs              34G     0   34G   0% /dev/shm
2024-08-20T22:15:26.9837654Z tmpfs              14G  553k   14G   1% /run
2024-08-20T22:15:26.9841139Z /dev/nvme0n1p1    161G   57G  105G  36% /
2024-08-20T22:15:26.9842052Z tmpfs              34G  8.2k   34G   1% /tmp
2024-08-20T22:15:26.9842569Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2024-08-20T22:15:26.9843088Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2024-08-20T22:15:26.9882396Z Prepare all required actions
2024-08-20T22:15:26.9882849Z Getting action download info
2024-08-20T22:15:27.1239758Z ##[group]Run ./.github/actions/download-td-artifacts
2024-08-20T22:15:27.1240235Z with:
2024-08-20T22:15:27.1240488Z env:
2024-08-20T22:15:27.1240764Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.1241220Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.1241707Z ##[endgroup]
2024-08-20T22:15:27.1275861Z ##[group]Run seemethere/download-artifact-s3@v4
2024-08-20T22:15:27.1276314Z with:
2024-08-20T22:15:27.1276594Z   name: td_results
2024-08-20T22:15:27.1276916Z   s3-bucket: gha-artifacts
2024-08-20T22:15:27.1277281Z   region: us-east-1
2024-08-20T22:15:27.1277590Z env:
2024-08-20T22:15:27.1277866Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.1278334Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.1278825Z ##[endgroup]
2024-08-20T22:15:27.6011163Z (node:139453) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-08-20T22:15:27.6011861Z 
2024-08-20T22:15:27.6012120Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-08-20T22:15:27.6012851Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-08-20T22:15:27.6013706Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-08-20T22:15:27.6631094Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/
2024-08-20T22:15:27.6637012Z Artifact download has finished successfully
2024-08-20T22:15:27.6957407Z ##[group]Run mkdir -p .additional_ci_files
2024-08-20T22:15:27.6957889Z mkdir -p .additional_ci_files
2024-08-20T22:15:27.6958461Z mv td_results.json .additional_ci_files/td_results.json
2024-08-20T22:15:27.6967909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:27.6968430Z env:
2024-08-20T22:15:27.6968712Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.6969184Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.6969676Z ##[endgroup]
2024-08-20T22:15:27.7051776Z mv: cannot stat 'td_results.json': No such file or directory
2024-08-20T22:15:27.7064885Z ##[error]Process completed with exit code 1.
2024-08-20T22:15:27.7100552Z ##[group]Run .github/scripts/parse_ref.py
2024-08-20T22:15:27.7101040Z .github/scripts/parse_ref.py
2024-08-20T22:15:27.7110462Z shell: /usr/bin/bash -e {0}
2024-08-20T22:15:27.7110830Z env:
2024-08-20T22:15:27.7111115Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.7111589Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.7112091Z ##[endgroup]
2024-08-20T22:15:27.7400188Z Prepare all required actions
2024-08-20T22:15:27.7438519Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-08-20T22:15:27.7438979Z with:
2024-08-20T22:15:27.7439708Z   github-token: ***
2024-08-20T22:15:27.7440023Z env:
2024-08-20T22:15:27.7440299Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.7440761Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.7441260Z ##[endgroup]
2024-08-20T22:15:27.7458093Z ##[group]Run set -eux
2024-08-20T22:15:27.7458465Z set -eux
2024-08-20T22:15:27.7459151Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2024-08-20T22:15:27.7468584Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:27.7469093Z env:
2024-08-20T22:15:27.7469370Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:27.7469826Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:27.7470469Z   GITHUB_TOKEN: ***
2024-08-20T22:15:27.7470786Z ##[endgroup]
2024-08-20T22:15:27.7498388Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-0e9f01dbcd5a47485
2024-08-20T22:15:28.4130396Z setting job-id=29026299882
2024-08-20T22:15:28.4131558Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:28.4350637Z Prepare all required actions
2024-08-20T22:15:28.4351092Z Getting action download info
2024-08-20T22:15:28.5452106Z ##[group]Run ./.github/actions/filter-test-configs
2024-08-20T22:15:28.5452571Z with:
2024-08-20T22:15:28.5453021Z   github-token: ***
2024-08-20T22:15:28.5459517Z   test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-08-20T22:15:28.5466920Z   job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:28.5468099Z env:
2024-08-20T22:15:28.5468381Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:28.5468833Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:28.5469315Z ##[endgroup]
2024-08-20T22:15:28.5512046Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-08-20T22:15:28.5512615Z with:
2024-08-20T22:15:28.5512887Z   shell: bash
2024-08-20T22:15:28.5513216Z   timeout_minutes: 10
2024-08-20T22:15:28.5513569Z   max_attempts: 5
2024-08-20T22:15:28.5513907Z   retry_wait_seconds: 30
2024-08-20T22:15:28.5515017Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-08-20T22:15:28.5516172Z   polling_interval_seconds: 1
2024-08-20T22:15:28.5516740Z   warning_on_retry: true
2024-08-20T22:15:28.5517100Z   continue_on_error: false
2024-08-20T22:15:28.5517450Z env:
2024-08-20T22:15:28.5517733Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:28.5518190Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:28.5518839Z   GITHUB_TOKEN: ***
2024-08-20T22:15:28.5519153Z ##[endgroup]
2024-08-20T22:15:28.6281114Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-08-20T22:15:28.8671790Z Defaulting to user installation because normal site-packages is not writeable
2024-08-20T22:15:28.8852486Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1)
2024-08-20T22:15:28.8855160Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1)
2024-08-20T22:15:28.8970215Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2024-08-20T22:15:28.8974369Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4)
2024-08-20T22:15:28.8983557Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2024-08-20T22:15:28.8993626Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12)
2024-08-20T22:15:29.6066681Z Command completed after 1 attempt(s).
2024-08-20T22:15:29.6128454Z ##[group]Run set -x
2024-08-20T22:15:29.6128797Z set -x
2024-08-20T22:15:29.6129095Z 
2024-08-20T22:15:29.6129653Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-08-20T22:15:29.6130337Z # in runner workspace
2024-08-20T22:15:29.6130973Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2024-08-20T22:15:29.6139868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:29.6140389Z env:
2024-08-20T22:15:29.6140677Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:29.6141161Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:29.6141651Z ##[endgroup]
2024-08-20T22:15:29.6172714Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-08-20T22:15:29.6411645Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-08-20T22:15:29.6412191Z echo "Workflow: ${GITHUB_WORKFLOW}"
2024-08-20T22:15:29.6412661Z echo "Job name: ${JOB_NAME}"
2024-08-20T22:15:29.6413063Z 
2024-08-20T22:15:29.6413601Z # Use relative path here as this could be checked out anywhere, not necessarily
2024-08-20T22:15:29.6414279Z # in runner workspace
2024-08-20T22:15:29.6414859Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2024-08-20T22:15:29.6415503Z   --workflow "${GITHUB_WORKFLOW}" \
2024-08-20T22:15:29.6415981Z   --job-name "${JOB_NAME}" \
2024-08-20T22:15:29.6422580Z   --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2024-08-20T22:15:29.6429703Z   --selected-test-configs "" \
2024-08-20T22:15:29.6430171Z   --pr-number "${PR_NUMBER}" \
2024-08-20T22:15:29.6430598Z   --tag "${TAG}" \
2024-08-20T22:15:29.6430995Z   --event-name "${EVENT_NAME}" \
2024-08-20T22:15:29.6431445Z   --schedule "${SCHEDULE}" \
2024-08-20T22:15:29.6431877Z   --branch "${HEAD_BRANCH}"
2024-08-20T22:15:29.6440904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:29.6441424Z env:
2024-08-20T22:15:29.6441705Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:29.6442170Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:29.6442855Z   GITHUB_TOKEN: ***
2024-08-20T22:15:29.6443716Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:29.6444633Z   PR_NUMBER: 
2024-08-20T22:15:29.6444954Z   TAG: ciflow/inductor/133712
2024-08-20T22:15:29.6445328Z   EVENT_NAME: push
2024-08-20T22:15:29.6445648Z   SCHEDULE: 
2024-08-20T22:15:29.6445937Z   HEAD_BRANCH: 
2024-08-20T22:15:29.6446244Z ##[endgroup]
2024-08-20T22:15:29.6476786Z Workflow: inductor-periodic
2024-08-20T22:15:29.6478385Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:29.8663349Z INFO:root:Found no test-config label on the PR, so all test configs are included
2024-08-20T22:15:30.0161181Z ##[group]Run echo "Filtered matrix:"
2024-08-20T22:15:30.0161637Z echo "Filtered matrix:"
2024-08-20T22:15:30.0168336Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2024-08-20T22:15:30.0175205Z 
2024-08-20T22:15:30.0175679Z echo
2024-08-20T22:15:30.0176065Z echo "Is the current job unstable? False"
2024-08-20T22:15:30.0176528Z 
2024-08-20T22:15:30.0176808Z echo
2024-08-20T22:15:30.0177173Z echo "Is keep-going label set? False"
2024-08-20T22:15:30.0177618Z 
2024-08-20T22:15:30.0177889Z echo
2024-08-20T22:15:30.0178210Z echo "Renabled issues? "
2024-08-20T22:15:30.0187094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:30.0187615Z env:
2024-08-20T22:15:30.0187903Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:30.0188374Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:30.0188871Z ##[endgroup]
2024-08-20T22:15:30.0218292Z Filtered matrix:
2024-08-20T22:15:30.0226543Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-08-20T22:15:30.0233129Z 
2024-08-20T22:15:30.0233283Z Is the current job unstable? False
2024-08-20T22:15:30.0233571Z 
2024-08-20T22:15:30.0233921Z Is keep-going label set? False
2024-08-20T22:15:30.0234177Z 
2024-08-20T22:15:30.0234310Z Renabled issues? 
2024-08-20T22:15:30.0277737Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-08-20T22:15:30.0278438Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-08-20T22:15:30.0286644Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-20T22:15:30.0287152Z env:
2024-08-20T22:15:30.0287436Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:30.0287908Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:30.0288405Z   JOB_TIMEOUT: 240
2024-08-20T22:15:30.0288729Z ##[endgroup]
2024-08-20T22:15:30.0371133Z ##[group]Run set -x
2024-08-20T22:15:30.0371550Z set -x
2024-08-20T22:15:30.0371867Z 
2024-08-20T22:15:30.0372231Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2024-08-20T22:15:30.0372788Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2024-08-20T22:15:30.0373355Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2024-08-20T22:15:30.0373874Z   TEST_COMMAND=.ci/onnx/test.sh
2024-08-20T22:15:30.0374298Z else
2024-08-20T22:15:30.0374655Z   TEST_COMMAND=.ci/pytorch/test.sh
2024-08-20T22:15:30.0375084Z fi
2024-08-20T22:15:30.0375371Z 
2024-08-20T22:15:30.0375849Z # detached container should get cleaned up by teardown_ec2_linux
2024-08-20T22:15:30.0376610Z # TODO: Stop building test binaries as part of the build phase
2024-08-20T22:15:30.0377288Z # Used for GPU_FLAG since that doesn't play nice
2024-08-20T22:15:30.0377888Z # shellcheck disable=SC2086,SC2090
2024-08-20T22:15:30.0378558Z container_name=$(docker run \
2024-08-20T22:15:30.0378983Z   ${GPU_FLAG:-} \
2024-08-20T22:15:30.0379369Z   -e BUILD_ENVIRONMENT \
2024-08-20T22:15:30.0379780Z   -e PR_NUMBER \
2024-08-20T22:15:30.0380149Z   -e GITHUB_ACTIONS \
2024-08-20T22:15:30.0380555Z   -e GITHUB_REPOSITORY \
2024-08-20T22:15:30.0380976Z   -e GITHUB_WORKFLOW \
2024-08-20T22:15:30.0381373Z   -e GITHUB_JOB \
2024-08-20T22:15:30.0381741Z   -e GITHUB_RUN_ID \
2024-08-20T22:15:30.0382139Z   -e GITHUB_RUN_NUMBER \
2024-08-20T22:15:30.0382557Z   -e GITHUB_RUN_ATTEMPT \
2024-08-20T22:15:30.0382964Z   -e JOB_ID \
2024-08-20T22:15:30.0383307Z   -e JOB_NAME \
2024-08-20T22:15:30.0383665Z   -e BASE_SHA \
2024-08-20T22:15:30.0384025Z   -e BRANCH \
2024-08-20T22:15:30.0384370Z   -e SHA1 \
2024-08-20T22:15:30.0384722Z   -e AWS_DEFAULT_REGION \
2024-08-20T22:15:30.0385173Z   -e IN_WHEEL_TEST \
2024-08-20T22:15:30.0385581Z   -e SHARD_NUMBER \
2024-08-20T22:15:30.0385955Z   -e TEST_CONFIG \
2024-08-20T22:15:30.0386340Z   -e NUM_TEST_SHARDS \
2024-08-20T22:15:30.0386747Z   -e REENABLED_ISSUES \
2024-08-20T22:15:30.0387170Z   -e CONTINUE_THROUGH_ERROR \
2024-08-20T22:15:30.0387603Z   -e VERBOSE_TEST_LOGS \
2024-08-20T22:15:30.0388014Z   -e TEST_SHOWLOCALS \
2024-08-20T22:15:30.0388417Z   -e NO_TEST_TIMEOUT \
2024-08-20T22:15:30.0388803Z   -e NO_TD \
2024-08-20T22:15:30.0389152Z   -e TD_DISTRIBUTED \
2024-08-20T22:15:30.0389547Z   -e PR_LABELS \
2024-08-20T22:15:30.0389969Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2024-08-20T22:15:30.0390438Z   -e SCCACHE_BUCKET \
2024-08-20T22:15:30.0390841Z   -e SCCACHE_S3_KEY_PREFIX \
2024-08-20T22:15:30.0391261Z   -e XLA_CUDA \
2024-08-20T22:15:30.0391685Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2024-08-20T22:15:30.0392206Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2024-08-20T22:15:30.0392735Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2024-08-20T22:15:30.0393259Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2024-08-20T22:15:30.0393741Z   -e HUGGING_FACE_HUB_TOKEN \
2024-08-20T22:15:30.0394215Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2024-08-20T22:15:30.0394659Z   -e DASHBOARD_TAG \
2024-08-20T22:15:30.0395147Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2024-08-20T22:15:30.0395714Z   --security-opt seccomp=unconfined \
2024-08-20T22:15:30.0396188Z   --cap-add=SYS_PTRACE \
2024-08-20T22:15:30.0396587Z   --ipc=host \
2024-08-20T22:15:30.0396963Z   --shm-size="${SHM_SIZE}" \
2024-08-20T22:15:30.0397374Z   --tty \
2024-08-20T22:15:30.0397701Z   --detach \
2024-08-20T22:15:30.0398192Z   --name="${container_name}" \
2024-08-20T22:15:30.0398631Z   --user jenkins \
2024-08-20T22:15:30.0399132Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2024-08-20T22:15:30.0399699Z   -w /var/lib/jenkins/workspace \
2024-08-20T22:15:30.0400147Z   "${DOCKER_IMAGE}"
2024-08-20T22:15:30.0400514Z )
2024-08-20T22:15:30.0400931Z # Propagate download.pytorch.org IP to container
2024-08-20T22:15:30.0401848Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2024-08-20T22:15:30.0402813Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2024-08-20T22:15:30.0403729Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2024-08-20T22:15:30.0412848Z shell: /usr/bin/bash -e {0}
2024-08-20T22:15:30.0413200Z env:
2024-08-20T22:15:30.0413476Z   GIT_DEFAULT_BRANCH: main
2024-08-20T22:15:30.0413940Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:30.0414636Z   BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:30.0415162Z   PR_NUMBER: 
2024-08-20T22:15:30.0415504Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-08-20T22:15:30.0415939Z   GITHUB_WORKFLOW: inductor-periodic
2024-08-20T22:15:30.0416346Z   GITHUB_JOB: test
2024-08-20T22:15:30.0416659Z   GITHUB_RUN_ID: 10479310721
2024-08-20T22:15:30.0417024Z   GITHUB_RUN_NUMBER: 54565
2024-08-20T22:15:30.0417379Z   GITHUB_RUN_ATTEMPT: 1
2024-08-20T22:15:30.0417716Z   JOB_ID: 29026299882
2024-08-20T22:15:30.0418567Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:30.0419479Z   BRANCH: 
2024-08-20T22:15:30.0419822Z   SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:30.0420345Z   BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:30.0420847Z   TEST_CONFIG: dynamic_aot_eager_torchbench
2024-08-20T22:15:30.0421283Z   SHARD_NUMBER: 1
2024-08-20T22:15:30.0421595Z   NUM_TEST_SHARDS: 2
2024-08-20T22:15:30.0421922Z   REENABLED_ISSUES: 
2024-08-20T22:15:30.0422255Z   CONTINUE_THROUGH_ERROR: False
2024-08-20T22:15:30.0422641Z   VERBOSE_TEST_LOGS: False
2024-08-20T22:15:30.0423002Z   TEST_SHOWLOCALS: False
2024-08-20T22:15:30.0423349Z   NO_TEST_TIMEOUT: False
2024-08-20T22:15:30.0423676Z   NO_TD: False
2024-08-20T22:15:30.0423977Z   TD_DISTRIBUTED: False
2024-08-20T22:15:30.0424395Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-08-20T22:15:30.0424906Z   SCCACHE_S3_KEY_PREFIX: inductor-periodic
2024-08-20T22:15:30.0425343Z   SHM_SIZE: 2g
2024-08-20T22:15:30.0426449Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:15:30.0427602Z   XLA_CUDA: 
2024-08-20T22:15:30.0428083Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:15:30.0428695Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-08-20T22:15:30.0429129Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-08-20T22:15:30.0429537Z   DASHBOARD_TAG: 
2024-08-20T22:15:30.0430051Z   HUGGING_FACE_HUB_TOKEN: ***
2024-08-20T22:15:30.0430431Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: 
2024-08-20T22:15:30.0430805Z ##[endgroup]
2024-08-20T22:15:30.0461353Z + [[ dynamic_aot_eager_torchbench == \m\u\l\t\i\g\p\u ]]
2024-08-20T22:15:30.0462412Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]]
2024-08-20T22:15:30.0463083Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-08-20T22:15:30.0471675Z +++ nproc --ignore=2
2024-08-20T22:15:30.0491158Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-20T22:15:41.3493535Z + container_name=55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-20T22:15:41.3496485Z + grep download.pytorch.org /etc/hosts
2024-08-20T22:15:41.3498180Z + docker exec -i 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271 sudo bash -c '/bin/cat >> /etc/hosts'
2024-08-20T22:15:41.4857017Z + echo DOCKER_CONTAINER_ID=55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-20T22:15:41.4861661Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl
2024-08-20T22:15:41.4864782Z + docker exec -t 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-08-20T22:15:41.8831592Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6)
2024-08-20T22:15:42.2099748Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1)
2024-08-20T22:15:42.2101882Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2)
2024-08-20T22:15:42.2104290Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8)
2024-08-20T22:15:42.2107904Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4)
2024-08-20T22:15:42.2111049Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1)
2024-08-20T22:15:42.2116268Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1)
2024-08-20T22:15:42.2132975Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0)
2024-08-20T22:15:42.2146534Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0)
2024-08-20T22:15:42.2163949Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4)
2024-08-20T22:15:42.2551937Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5)
2024-08-20T22:15:42.8326829Z Installing collected packages: torch
2024-08-20T22:15:52.5822425Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-08-20T22:15:52.5824123Z timm 1.0.9.dev0 requires torchvision, which is not installed.
2024-08-20T22:15:52.5825447Z Successfully installed torch-2.5.0a0+git40ec5f6
2024-08-20T22:15:52.6647616Z + export TERM=vt100
2024-08-20T22:15:52.6647961Z + TERM=vt100
2024-08-20T22:15:52.6652048Z ++ dirname .ci/pytorch/test.sh
2024-08-20T22:15:52.6662728Z + source .ci/pytorch/common.sh
2024-08-20T22:15:52.6666450Z +++ dirname .ci/pytorch/common.sh
2024-08-20T22:15:52.6679177Z ++ source .ci/pytorch/common_utils.sh
2024-08-20T22:15:52.6680968Z +++ declare -f -t trap_add
2024-08-20T22:15:52.6686798Z ++ set -ex
2024-08-20T22:15:52.6687459Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-08-20T22:15:52.6688114Z ++ BUILD_TEST_LIBTORCH=0
2024-08-20T22:15:52.6688628Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]]
2024-08-20T22:15:52.6692042Z ++ stat -c %u /var/lib/jenkins/workspace
2024-08-20T22:15:52.6712159Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-08-20T22:15:52.6712688Z + trap_add cleanup_workspace EXIT
2024-08-20T22:15:52.6713116Z + trap_add_cmd=cleanup_workspace
2024-08-20T22:15:52.6714507Z + shift
2024-08-20T22:15:52.6714800Z + for trap_add_name in "$@"
2024-08-20T22:15:52.6720810Z +++ trap -p EXIT
2024-08-20T22:15:52.6724186Z ++ eval 'extract_trap_cmd '
2024-08-20T22:15:52.6724563Z +++ extract_trap_cmd
2024-08-20T22:15:52.6724929Z +++ printf '%s\n' ''
2024-08-20T22:15:52.6725328Z ++ printf '%s\n' cleanup_workspace
2024-08-20T22:15:52.6727654Z + trap -- '
2024-08-20T22:15:52.6727996Z cleanup_workspace' EXIT
2024-08-20T22:15:52.6728469Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-08-20T22:15:53.3760505Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-08-20T22:15:53.3783872Z + echo 'Environment variables:'
2024-08-20T22:15:53.3784409Z Environment variables:
2024-08-20T22:15:53.3784835Z + env
2024-08-20T22:15:53.3794044Z INSTALLED_DB=yes
2024-08-20T22:15:53.3794600Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-08-20T22:15:53.3795128Z NVIDIA_VISIBLE_DEVICES=all
2024-08-20T22:15:53.3795688Z NV_NVML_DEV_VERSION=12.1.105-1
2024-08-20T22:15:53.3796550Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:15:53.3797246Z CONTINUE_THROUGH_ERROR=False
2024-08-20T22:15:53.3797862Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-08-20T22:15:53.3798564Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:15:53.3799290Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:53.3799816Z HOSTNAME=55aa4f098e1d
2024-08-20T22:15:53.3800855Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.3801687Z GITHUB_ACTION=__self
2024-08-20T22:15:53.3802201Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-08-20T22:15:53.3806541Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-08-20T22:15:53.3810635Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-08-20T22:15:53.3811300Z NV_NVTX_VERSION=12.1.105-1
2024-08-20T22:15:53.3811659Z GITHUB_RUN_NUMBER=54565
2024-08-20T22:15:53.3812029Z TEST_CONFIG=dynamic_aot_eager_torchbench
2024-08-20T22:15:53.3812476Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-08-20T22:15:53.3813083Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-08-20T22:15:53.3813553Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-08-20T22:15:53.3814259Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-08-20T22:15:53.3814684Z SCRIBE_GRAPHQL_ACCESS_TOKEN=
2024-08-20T22:15:53.3815088Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-08-20T22:15:53.3815534Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-08-20T22:15:53.3816035Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-08-20T22:15:53.3816505Z GITHUB_REF_TYPE=tag
2024-08-20T22:15:53.3816834Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-08-20T22:15:53.3817213Z NCCL_VERSION=2.17.1-1
2024-08-20T22:15:53.3817614Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.3818065Z XLA_CUDA=
2024-08-20T22:15:53.3818574Z HUGGING_FACE_HUB_TOKEN=***
2024-08-20T22:15:53.3820813Z ***
2024-08-20T22:15:53.3821124Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-08-20T22:15:53.3821568Z GITHUB_REPOSITORY_ID=65600975
2024-08-20T22:15:53.3821971Z GITHUB_ACTIONS=true
2024-08-20T22:15:53.3822304Z NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:53.3822834Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-08-20T22:15:53.3823377Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-08-20T22:15:53.3824056Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.3824571Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-08-20T22:15:53.3825063Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.3826000Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712
2024-08-20T22:15:53.3826770Z UCC_HOME=/usr
2024-08-20T22:15:53.3827109Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-08-20T22:15:53.3827513Z VERBOSE_TEST_LOGS=False
2024-08-20T22:15:53.3827863Z NVIDIA_PRODUCT_NAME=CUDA
2024-08-20T22:15:53.3828329Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-08-20T22:15:53.3828833Z GITHUB_REF=refs/tags/ciflow/inductor/133712
2024-08-20T22:15:53.3829304Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-08-20T22:15:53.3829690Z SHARD_NUMBER=1
2024-08-20T22:15:53.3830002Z GITHUB_REF_PROTECTED=false
2024-08-20T22:15:53.3830361Z HOME=/var/lib/jenkins
2024-08-20T22:15:53.3830734Z GITHUB_API_URL=https://api.github.com
2024-08-20T22:15:53.3831188Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-08-20T22:15:53.3831716Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-08-20T22:15:53.3832275Z SCCACHE_S3_KEY_PREFIX=inductor-periodic
2024-08-20T22:15:53.3832698Z CUDA_VERSION=12.1.1
2024-08-20T22:15:53.3833133Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-08-20T22:15:53.3833577Z NUM_TEST_SHARDS=2
2024-08-20T22:15:53.3833888Z UCX_HOME=/usr
2024-08-20T22:15:53.3834418Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-08-20T22:15:53.3835504Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.3836969Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:53.3838385Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.3839498Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-08-20T22:15:53.3840156Z GITHUB_EVENT_NAME=push
2024-08-20T22:15:53.3840485Z DASHBOARD_TAG=
2024-08-20T22:15:53.3840798Z GITHUB_RUN_ID=10479310721
2024-08-20T22:15:53.3841276Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-08-20T22:15:53.3841819Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-08-20T22:15:53.3842871Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.3843783Z GITHUB_ACTOR=pytorch-bot[bot]
2024-08-20T22:15:53.3844200Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-08-20T22:15:53.3844574Z PR_NUMBER=
2024-08-20T22:15:53.3844865Z GITHUB_RUN_ATTEMPT=1
2024-08-20T22:15:53.3845211Z ANACONDA_PYTHON_VERSION=3.10
2024-08-20T22:15:53.3845659Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-08-20T22:15:53.3846115Z TERM=vt100
2024-08-20T22:15:53.3846461Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-08-20T22:15:53.3846989Z INSTALLED_VISION=yes
2024-08-20T22:15:53.3847318Z BRANCH=
2024-08-20T22:15:53.3847606Z OPENSSL_ROOT_DIR=/opt/openssl
2024-08-20T22:15:53.3848015Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-08-20T22:15:53.3848450Z CUDA_PATH=/usr/local/cuda
2024-08-20T22:15:53.3849206Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-08-20T22:15:53.3849963Z GITHUB_SERVER_URL=https://github.com
2024-08-20T22:15:53.3850456Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-08-20T22:15:53.3851043Z REENABLED_ISSUES=
2024-08-20T22:15:53.3851347Z SHLVL=1
2024-08-20T22:15:53.3851619Z MAX_JOBS=14
2024-08-20T22:15:53.3851952Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-08-20T22:15:53.3852314Z NVARCH=x86_64
2024-08-20T22:15:53.3852622Z GITHUB_ACTOR_ID=54816060
2024-08-20T22:15:53.3853087Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.3853636Z GITHUB_REF_NAME=ciflow/inductor/133712
2024-08-20T22:15:53.3854135Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-08-20T22:15:53.3855056Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:15:53.3855642Z GITHUB_JOB=test
2024-08-20T22:15:53.3856068Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-08-20T22:15:53.3856622Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-08-20T22:15:53.3857139Z NO_TEST_TIMEOUT=False
2024-08-20T22:15:53.3857476Z TD_DISTRIBUTED=False
2024-08-20T22:15:53.3857863Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-08-20T22:15:53.3858299Z GITHUB_REPOSITORY=pytorch/pytorch
2024-08-20T22:15:53.3858729Z NV_NVPROF_VERSION=12.1.105-1
2024-08-20T22:15:53.3859102Z GITHUB_RETENTION_DAYS=90
2024-08-20T22:15:53.3859459Z OPENSSL_DIR=/opt/openssl
2024-08-20T22:15:53.3859809Z GITHUB_ACTION_REPOSITORY=
2024-08-20T22:15:53.3860845Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:15:53.3861964Z GITHUB_BASE_REF=
2024-08-20T22:15:53.3862308Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-08-20T22:15:53.3862690Z CI=true
2024-08-20T22:15:53.3863016Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:15:53.3863439Z GITHUB_REPOSITORY_OWNER=pytorch
2024-08-20T22:15:53.3863817Z JOB_ID=29026299882
2024-08-20T22:15:53.3864144Z INSTALLED_PROTOBUF=yes
2024-08-20T22:15:53.3864484Z GITHUB_HEAD_REF=
2024-08-20T22:15:53.3864799Z GITHUB_ACTION_REF=
2024-08-20T22:15:53.3865241Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-08-20T22:15:53.3865716Z TEST_SHOWLOCALS=False
2024-08-20T22:15:53.3866123Z GITHUB_WORKFLOW=inductor-periodic
2024-08-20T22:15:53.3866533Z DEBIAN_FRONTEND=noninteractive
2024-08-20T22:15:53.3867747Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.3868577Z NO_TD=False
2024-08-20T22:15:53.3868890Z SKIP_SCCACHE_INITIALIZATION=1
2024-08-20T22:15:53.3869261Z _=/usr/bin/env
2024-08-20T22:15:53.3869773Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-08-20T22:15:53.4026156Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-08-20T22:15:53.4027311Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-08-20T22:15:53.4028452Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-08-20T22:15:53.4029350Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-08-20T22:15:53.4029977Z + BUILD_DIR=build
2024-08-20T22:15:53.4030320Z + BUILD_RENAMED_DIR=build_renamed
2024-08-20T22:15:53.4030726Z + BUILD_BIN_DIR=build/bin
2024-08-20T22:15:53.4031074Z + SHARD_NUMBER=1
2024-08-20T22:15:53.4031389Z + NUM_TEST_SHARDS=2
2024-08-20T22:15:53.4031720Z + export VALGRIND=ON
2024-08-20T22:15:53.4032076Z + VALGRIND=ON
2024-08-20T22:15:53.4032581Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]]
2024-08-20T22:15:53.4033089Z + [[ 0 == \1 ]]
2024-08-20T22:15:53.4033405Z + [[ False == \1 ]]
2024-08-20T22:15:53.4034087Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]]
2024-08-20T22:15:53.4034679Z ++ realpath build/custom_test_artifacts
2024-08-20T22:15:53.4043882Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-08-20T22:15:53.4044806Z + [[ -n '' ]]
2024-08-20T22:15:53.4045167Z + echo 'Environment variables'
2024-08-20T22:15:53.4045555Z Environment variables
2024-08-20T22:15:53.4045880Z + env
2024-08-20T22:15:53.4053166Z INSTALLED_DB=yes
2024-08-20T22:15:53.4053923Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-08-20T22:15:53.4054460Z NVIDIA_VISIBLE_DEVICES=all
2024-08-20T22:15:53.4055072Z NV_NVML_DEV_VERSION=12.1.105-1
2024-08-20T22:15:53.4055887Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-20T22:15:53.4056620Z CONTINUE_THROUGH_ERROR=False
2024-08-20T22:15:53.4057273Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-08-20T22:15:53.4057927Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:15:53.4058574Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-08-20T22:15:53.4059336Z HOSTNAME=55aa4f098e1d
2024-08-20T22:15:53.4060289Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.4061165Z GITHUB_ACTION=__self
2024-08-20T22:15:53.4061515Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-08-20T22:15:53.4065522Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-08-20T22:15:53.4070042Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-08-20T22:15:53.4070566Z NV_NVTX_VERSION=12.1.105-1
2024-08-20T22:15:53.4070916Z GITHUB_RUN_NUMBER=54565
2024-08-20T22:15:53.4071286Z TEST_CONFIG=dynamic_aot_eager_torchbench
2024-08-20T22:15:53.4071751Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-08-20T22:15:53.4072332Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-08-20T22:15:53.4072792Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-08-20T22:15:53.4073236Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-08-20T22:15:53.4073638Z SCRIBE_GRAPHQL_ACCESS_TOKEN=
2024-08-20T22:15:53.4074028Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-08-20T22:15:53.4074467Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2024-08-20T22:15:53.4074958Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-08-20T22:15:53.4075422Z GITHUB_REF_TYPE=tag
2024-08-20T22:15:53.4075750Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-08-20T22:15:53.4076217Z NCCL_VERSION=2.17.1-1
2024-08-20T22:15:53.4076616Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.4077057Z XLA_CUDA=
2024-08-20T22:15:53.4077508Z HUGGING_FACE_HUB_TOKEN=***
2024-08-20T22:15:53.4077901Z ***
2024-08-20T22:15:53.4078192Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-08-20T22:15:53.4078595Z GITHUB_REPOSITORY_ID=65600975
2024-08-20T22:15:53.4078970Z GITHUB_ACTIONS=true
2024-08-20T22:15:53.4079299Z NVIDIA_DRIVER_CAPABILITIES=all
2024-08-20T22:15:53.4079799Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-08-20T22:15:53.4080341Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-08-20T22:15:53.4080809Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.4081338Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-08-20T22:15:53.4081828Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.4082755Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712
2024-08-20T22:15:53.4083862Z UCC_HOME=/usr
2024-08-20T22:15:53.4084238Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-08-20T22:15:53.4084645Z VERBOSE_TEST_LOGS=False
2024-08-20T22:15:53.4084992Z NVIDIA_PRODUCT_NAME=CUDA
2024-08-20T22:15:53.4085452Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-08-20T22:15:53.4085946Z GITHUB_REF=refs/tags/ciflow/inductor/133712
2024-08-20T22:15:53.4086412Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-08-20T22:15:53.4086787Z SHARD_NUMBER=1
2024-08-20T22:15:53.4087089Z GITHUB_REF_PROTECTED=false
2024-08-20T22:15:53.4087437Z HOME=/var/lib/jenkins
2024-08-20T22:15:53.4087800Z GITHUB_API_URL=https://api.github.com
2024-08-20T22:15:53.4088226Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-08-20T22:15:53.4088698Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-08-20T22:15:53.4089250Z SCCACHE_S3_KEY_PREFIX=inductor-periodic
2024-08-20T22:15:53.4089661Z CUDA_VERSION=12.1.1
2024-08-20T22:15:53.4090078Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-08-20T22:15:53.4090515Z NUM_TEST_SHARDS=2
2024-08-20T22:15:53.4090981Z UCX_HOME=/usr
2024-08-20T22:15:53.4091637Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-08-20T22:15:53.4092713Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.4094152Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (dynamic_aot_eager_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu)
2024-08-20T22:15:53.4095555Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.4096737Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-08-20T22:15:53.4097383Z GITHUB_EVENT_NAME=push
2024-08-20T22:15:53.4097711Z DASHBOARD_TAG=
2024-08-20T22:15:53.4098015Z GITHUB_RUN_ID=10479310721
2024-08-20T22:15:53.4098489Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-08-20T22:15:53.4099019Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-08-20T22:15:53.4100009Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.4100911Z GITHUB_ACTOR=pytorch-bot[bot]
2024-08-20T22:15:53.4101318Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-08-20T22:15:53.4101684Z PR_NUMBER=
2024-08-20T22:15:53.4101969Z GITHUB_RUN_ATTEMPT=1
2024-08-20T22:15:53.4102287Z VALGRIND=ON
2024-08-20T22:15:53.4102578Z ANACONDA_PYTHON_VERSION=3.10
2024-08-20T22:15:53.4103019Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-08-20T22:15:53.4103484Z TERM=vt100
2024-08-20T22:15:53.4103821Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-08-20T22:15:53.4104224Z INSTALLED_VISION=yes
2024-08-20T22:15:53.4104540Z BRANCH=
2024-08-20T22:15:53.4104820Z OPENSSL_ROOT_DIR=/opt/openssl
2024-08-20T22:15:53.4105222Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-08-20T22:15:53.4105635Z CUDA_PATH=/usr/local/cuda
2024-08-20T22:15:53.4106391Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-08-20T22:15:53.4107139Z GITHUB_SERVER_URL=https://github.com
2024-08-20T22:15:53.4107619Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-08-20T22:15:53.4108079Z REENABLED_ISSUES=
2024-08-20T22:15:53.4108377Z SHLVL=1
2024-08-20T22:15:53.4108639Z MAX_JOBS=14
2024-08-20T22:15:53.4108945Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-08-20T22:15:53.4109298Z NVARCH=x86_64
2024-08-20T22:15:53.4109594Z GITHUB_ACTOR_ID=54816060
2024-08-20T22:15:53.4110047Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3
2024-08-20T22:15:53.4110573Z GITHUB_REF_NAME=ciflow/inductor/133712
2024-08-20T22:15:53.4111061Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-08-20T22:15:53.4111769Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-08-20T22:15:53.4112342Z GITHUB_JOB=test
2024-08-20T22:15:53.4112736Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-08-20T22:15:53.4113285Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-08-20T22:15:53.4113910Z NO_TEST_TIMEOUT=False
2024-08-20T22:15:53.4114239Z TD_DISTRIBUTED=False
2024-08-20T22:15:53.4114620Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-08-20T22:15:53.4115058Z GITHUB_REPOSITORY=pytorch/pytorch
2024-08-20T22:15:53.4115476Z NV_NVPROF_VERSION=12.1.105-1
2024-08-20T22:15:53.4115833Z GITHUB_RETENTION_DAYS=90
2024-08-20T22:15:53.4116179Z OPENSSL_DIR=/opt/openssl
2024-08-20T22:15:53.4116526Z GITHUB_ACTION_REPOSITORY=
2024-08-20T22:15:53.4117543Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:15:53.4118585Z GITHUB_BASE_REF=
2024-08-20T22:15:53.4118910Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-08-20T22:15:53.4119277Z CI=true
2024-08-20T22:15:53.4119593Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-08-20T22:15:53.4120000Z GITHUB_REPOSITORY_OWNER=pytorch
2024-08-20T22:15:53.4120381Z JOB_ID=29026299882
2024-08-20T22:15:53.4120691Z INSTALLED_PROTOBUF=yes
2024-08-20T22:15:53.4121015Z GITHUB_HEAD_REF=
2024-08-20T22:15:53.4121417Z GITHUB_ACTION_REF=
2024-08-20T22:15:53.4121861Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-08-20T22:15:53.4122320Z TEST_SHOWLOCALS=False
2024-08-20T22:15:53.4122704Z GITHUB_WORKFLOW=inductor-periodic
2024-08-20T22:15:53.4123190Z DEBIAN_FRONTEND=noninteractive
2024-08-20T22:15:53.4124111Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c48a7dae-e205-4c00-996d-072ea4cc35fe
2024-08-20T22:15:53.4124923Z NO_TD=False
2024-08-20T22:15:53.4125225Z SKIP_SCCACHE_INITIALIZATION=1
2024-08-20T22:15:53.4125594Z _=/usr/bin/env
2024-08-20T22:15:53.4125943Z + echo 'Testing pytorch'
2024-08-20T22:15:53.4126285Z Testing pytorch
2024-08-20T22:15:53.4126618Z + export LANG=C.UTF-8
2024-08-20T22:15:53.4126964Z + LANG=C.UTF-8
2024-08-20T22:15:53.4127263Z + PR_NUMBER=
2024-08-20T22:15:53.4127639Z + [[ dynamic_aot_eager_torchbench == \d\e\f\a\u\l\t ]]
2024-08-20T22:15:53.4128226Z + [[ dynamic_aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-08-20T22:15:53.4128802Z + [[ dynamic_aot_eager_torchbench == \s\l\o\w ]]
2024-08-20T22:15:53.4129471Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-08-20T22:15:53.4130161Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]]
2024-08-20T22:15:53.4130710Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-08-20T22:15:53.4131313Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-08-20T22:15:53.4131794Z + [[ dynamic_aot_eager_torchbench == *crossref* ]]
2024-08-20T22:15:53.4132452Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-08-20T22:15:53.4133095Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]]
2024-08-20T22:15:53.4133749Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-08-20T22:15:53.4134311Z + pip_install --user ninja==1.10.2
2024-08-20T22:15:53.4134854Z + pip install --progress-bar off --user ninja==1.10.2
2024-08-20T22:15:54.7542477Z Collecting ninja==1.10.2
2024-08-20T22:15:54.7740995Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-08-20T22:15:54.8064774Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-08-20T22:15:55.4284543Z Installing collected packages: ninja
2024-08-20T22:15:55.4372053Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-08-20T22:15:55.4373459Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2024-08-20T22:15:55.4641089Z Successfully installed ninja-1.10.2
2024-08-20T22:15:55.5443899Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:15:55.5446031Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:15:55.5448068Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-08-20T22:15:55.5448588Z + install_tlparse
2024-08-20T22:15:55.5448999Z + pip_install --user tlparse==0.3.25
2024-08-20T22:15:55.5449574Z + pip install --progress-bar off --user tlparse==0.3.25
2024-08-20T22:15:55.9534123Z Collecting tlparse==0.3.25
2024-08-20T22:15:55.9705323Z   Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
2024-08-20T22:15:56.0020644Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-08-20T22:15:56.6369644Z Installing collected packages: tlparse
2024-08-20T22:15:56.6958803Z Successfully installed tlparse-0.3.25
2024-08-20T22:15:56.7739159Z ++ python -m site --user-base
2024-08-20T22:15:56.7985330Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-08-20T22:15:56.7987627Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]]
2024-08-20T22:15:56.7988381Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]]
2024-08-20T22:15:56.7989132Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-08-20T22:15:56.7990174Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-08-20T22:15:56.7991468Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass
2024-08-20T22:15:56.7992211Z + cd test
2024-08-20T22:15:56.7992792Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-08-20T22:15:58.4347042Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-08-20T22:15:58.4347773Z + [[ dynamic_aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-08-20T22:15:58.4350298Z + DYNAMO_BENCHMARK_FLAGS=()
2024-08-20T22:15:58.4350930Z + [[ dynamic_aot_eager_torchbench == *pr_time_benchmarks* ]]
2024-08-20T22:15:58.4351751Z + [[ dynamic_aot_eager_torchbench == *dynamo_eager* ]]
2024-08-20T22:15:58.4352422Z + [[ dynamic_aot_eager_torchbench == *aot_eager* ]]
2024-08-20T22:15:58.4353407Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager)
2024-08-20T22:15:58.4353913Z + [[ dynamic_aot_eager_torchbench == *dynamic* ]]
2024-08-20T22:15:58.4354581Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only)
2024-08-20T22:15:58.4355177Z + [[ dynamic_aot_eager_torchbench == *cpu* ]]
2024-08-20T22:15:58.4355711Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-08-20T22:15:58.4384412Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-08-20T22:15:58.4385331Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-08-20T22:15:58.4387855Z + cd test
2024-08-20T22:15:58.4388465Z + python -c 'import torch; print(torch.__config__.show())'
2024-08-20T22:15:59.9078962Z PyTorch built with:
2024-08-20T22:15:59.9079890Z   - GCC 9.4
2024-08-20T22:15:59.9080327Z   - C++ Version: 201703
2024-08-20T22:15:59.9081266Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-08-20T22:15:59.9082360Z   - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-08-20T22:15:59.9083033Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-08-20T22:15:59.9083553Z   - LAPACK is enabled (usually provided by MKL)
2024-08-20T22:15:59.9084074Z   - NNPACK is enabled
2024-08-20T22:15:59.9084487Z   - CPU capability usage: AVX2
2024-08-20T22:15:59.9084904Z   - CUDA Runtime 12.1
2024-08-20T22:15:59.9085438Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-08-20T22:15:59.9086041Z   - CuDNN 90.1  (built against CUDA 12.4)
2024-08-20T22:15:59.9086493Z   - Magma 2.6.1
2024-08-20T22:15:59.9093711Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2024-08-20T22:15:59.9099924Z 
2024-08-20T22:16:00.2237365Z + cd test
2024-08-20T22:16:00.2238147Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-08-20T22:16:01.5614126Z ATen/Parallel:
2024-08-20T22:16:01.5614541Z 	at::get_num_threads() : 8
2024-08-20T22:16:01.5614944Z 	at::get_num_interop_threads() : 16
2024-08-20T22:16:01.5615360Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-08-20T22:16:01.5615756Z 	omp_get_max_threads() : 8
2024-08-20T22:16:01.5616812Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-08-20T22:16:01.5617605Z 	mkl_get_max_threads() : 8
2024-08-20T22:16:01.5618222Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-08-20T22:16:01.5618850Z std::thread::hardware_concurrency() : 16
2024-08-20T22:16:01.5619274Z Environment variables:
2024-08-20T22:16:01.5619620Z 	OMP_NUM_THREADS : [not set]
2024-08-20T22:16:01.5619986Z 	MKL_NUM_THREADS : [not set]
2024-08-20T22:16:01.5620381Z ATen parallel backend: OpenMP
2024-08-20T22:16:01.5620646Z 
2024-08-20T22:16:01.8387906Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-08-20T22:16:01.8388503Z + [[ dynamic_aot_eager_torchbench == *backward* ]]
2024-08-20T22:16:01.8389007Z + [[ dynamic_aot_eager_torchbench == *xla* ]]
2024-08-20T22:16:01.8389520Z + [[ dynamic_aot_eager_torchbench == *executorch* ]]
2024-08-20T22:16:01.8390087Z + [[ dynamic_aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]]
2024-08-20T22:16:01.8390763Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-08-20T22:16:01.8391315Z + [[ dynamic_aot_eager_torchbench == distributed ]]
2024-08-20T22:16:01.8391880Z + [[ dynamic_aot_eager_torchbench == *inductor_distributed* ]]
2024-08-20T22:16:01.8392658Z + [[ dynamic_aot_eager_torchbench == *inductor-halide* ]]
2024-08-20T22:16:01.8393541Z + [[ dynamic_aot_eager_torchbench == *inductor-micro-benchmark* ]]
2024-08-20T22:16:01.8394217Z + [[ dynamic_aot_eager_torchbench == *huggingface* ]]
2024-08-20T22:16:01.8394737Z + [[ dynamic_aot_eager_torchbench == *timm* ]]
2024-08-20T22:16:01.8395244Z + [[ dynamic_aot_eager_torchbench == *torchbench* ]]
2024-08-20T22:16:01.8395748Z + [[ dynamic_aot_eager_torchbench == *cpu* ]]
2024-08-20T22:16:01.8396179Z + install_torchaudio cuda
2024-08-20T22:16:01.8396516Z + local commit
2024-08-20T22:16:01.8396825Z ++ get_pinned_commit audio
2024-08-20T22:16:01.8397200Z ++ cat .github/ci_commit_pins/audio.txt
2024-08-20T22:16:01.8414055Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:01.8414537Z + [[ cuda == \c\u\d\a ]]
2024-08-20T22:16:01.8414977Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-08-20T22:16:01.8415886Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:01.8417306Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:02.1775149Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:02.1779046Z   Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-84u48x59
2024-08-20T22:16:02.1807166Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-84u48x59
2024-08-20T22:16:02.9012156Z   Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862'
2024-08-20T22:16:02.9039945Z   Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:03.3147249Z   Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:16:03.3148377Z   Running command git submodule update --init --recursive -q
2024-08-20T22:16:05.7408267Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T22:16:05.7441623Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6)
2024-08-20T22:16:05.7469591Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1)
2024-08-20T22:16:05.7473990Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2)
2024-08-20T22:16:05.7476603Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8)
2024-08-20T22:16:05.7480165Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4)
2024-08-20T22:16:05.7482813Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1)
2024-08-20T22:16:05.7488328Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1)
2024-08-20T22:16:05.7502772Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0)
2024-08-20T22:16:05.7969462Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5)
2024-08-20T22:16:05.8031073Z Building wheels for collected packages: torchaudio
2024-08-20T22:18:39.3848510Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-08-20T22:18:39.3882729Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310400 sha256=6aeffa330f8bb0d23b6f983e8b96637871ebb551de74162bb9b5ce467d42aec6
2024-08-20T22:18:39.3884373Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95
2024-08-20T22:18:39.3913053Z Successfully built torchaudio
2024-08-20T22:18:39.8789507Z Installing collected packages: torchaudio
2024-08-20T22:18:40.0912093Z Successfully installed torchaudio-2.4.0a0+b3f6f51
2024-08-20T22:18:40.2703248Z + install_torchtext
2024-08-20T22:18:40.2703742Z + local data_commit
2024-08-20T22:18:40.2706703Z + local text_commit
2024-08-20T22:18:40.2707323Z ++ get_pinned_commit data
2024-08-20T22:18:40.2707748Z ++ cat .github/ci_commit_pins/data.txt
2024-08-20T22:18:40.2730161Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:40.2733194Z ++ get_pinned_commit text
2024-08-20T22:18:40.2734043Z ++ cat .github/ci_commit_pins/text.txt
2024-08-20T22:18:40.2749752Z + text_commit=b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:18:40.2751066Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:40.2752512Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:40.6118118Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:40.6122601Z   Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-z5dsau42
2024-08-20T22:18:40.6165124Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-z5dsau42
2024-08-20T22:18:41.0735246Z   Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f'
2024-08-20T22:18:41.0764831Z   Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:41.3339281Z   Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:41.4924227Z   Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:18:41.4925304Z   Running command git submodule update --init --recursive -q
2024-08-20T22:20:01.2777952Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T22:20:01.2814459Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19)
2024-08-20T22:20:01.2816532Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3)
2024-08-20T22:20:01.2821146Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6)
2024-08-20T22:20:01.2850345Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1)
2024-08-20T22:20:01.2854148Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2)
2024-08-20T22:20:01.2857475Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8)
2024-08-20T22:20:01.2860416Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4)
2024-08-20T22:20:01.2863250Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1)
2024-08-20T22:20:01.2868235Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1)
2024-08-20T22:20:01.2883048Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0)
2024-08-20T22:20:01.2941541Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2)
2024-08-20T22:20:01.2945072Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7)
2024-08-20T22:20:01.2949576Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4)
2024-08-20T22:20:01.3441062Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5)
2024-08-20T22:20:01.3505079Z Building wheels for collected packages: torchdata
2024-08-20T22:20:03.3197736Z   Building wheel for torchdata (setup.py) ... [?25l- \ | done
2024-08-20T22:20:03.3205521Z [?25h  Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=bc14565b2ba253f700ac35e1b20a19861a1329eca69ef56271f27c272dbad969
2024-08-20T22:20:03.3207098Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce
2024-08-20T22:20:03.3235422Z Successfully built torchdata
2024-08-20T22:20:03.8074146Z Installing collected packages: torchdata
2024-08-20T22:20:03.9034795Z Successfully installed torchdata-0.7.0a0+11bb5b8
2024-08-20T22:20:08.9291106Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:08.9292682Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:09.2642511Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:09.2646903Z   Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-ltj51dw0
2024-08-20T22:20:09.2675328Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-ltj51dw0
2024-08-20T22:20:10.2718489Z   Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db'
2024-08-20T22:20:10.2747030Z   Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:10.7656435Z   Running command git checkout -q b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:10.9385938Z   Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:20:10.9387195Z   Running command git submodule update --init --recursive -q
2024-08-20T22:20:18.5302948Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T22:20:18.5337866Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5)
2024-08-20T22:20:18.5341044Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3)
2024-08-20T22:20:18.5344637Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6)
2024-08-20T22:20:18.5347846Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4)
2024-08-20T22:20:18.5351465Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8)
2024-08-20T22:20:18.5378699Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2)
2024-08-20T22:20:18.5382511Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7)
2024-08-20T22:20:18.5386853Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19)
2024-08-20T22:20:18.5390799Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4)
2024-08-20T22:20:18.5418293Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1)
2024-08-20T22:20:18.5422636Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2)
2024-08-20T22:20:18.5425105Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8)
2024-08-20T22:20:18.5429245Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4)
2024-08-20T22:20:18.5432506Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1)
2024-08-20T22:20:18.5438024Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1)
2024-08-20T22:20:18.5452784Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0)
2024-08-20T22:20:18.6028581Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5)
2024-08-20T22:20:18.6094549Z Building wheels for collected packages: torchtext
2024-08-20T22:20:56.6699697Z   Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done
2024-08-20T22:20:56.6734415Z [?25h  Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045464 sha256=399268bad97cd3fd947b4cf84653a7670e25bb3aa091d8754f65a9efbfbd828a
2024-08-20T22:20:56.6736474Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf
2024-08-20T22:20:56.6757952Z Successfully built torchtext
2024-08-20T22:20:57.1580002Z Installing collected packages: torchtext
2024-08-20T22:20:57.2700018Z Successfully installed torchtext-0.16.0a0+b0ebddc
2024-08-20T22:20:57.4282680Z + install_torchvision
2024-08-20T22:20:57.4283217Z + local orig_preload
2024-08-20T22:20:57.4283604Z + local commit
2024-08-20T22:20:57.4285988Z ++ get_pinned_commit vision
2024-08-20T22:20:57.4286539Z ++ cat .github/ci_commit_pins/vision.txt
2024-08-20T22:20:57.4305982Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:20:57.4306605Z + orig_preload=
2024-08-20T22:20:57.4307145Z + '[' -n '' ']'
2024-08-20T22:20:57.4308000Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:20:57.4309593Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:20:57.7672424Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:20:57.7676146Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-9y6735qa
2024-08-20T22:20:57.7704661Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-9y6735qa
2024-08-20T22:20:59.2616776Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-08-20T22:20:59.2647958Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:21:00.6059185Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:21:00.9090195Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:21:03.4171277Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-08-20T22:21:03.4204258Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2024-08-20T22:21:03.4208278Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6)
2024-08-20T22:21:03.4212468Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0)
2024-08-20T22:21:03.4284074Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1)
2024-08-20T22:21:03.4288393Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-08-20T22:21:03.4291655Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-08-20T22:21:03.4295260Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-08-20T22:21:03.4298401Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1)
2024-08-20T22:21:03.4303529Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1)
2024-08-20T22:21:03.4317634Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-08-20T22:21:03.4803819Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5)
2024-08-20T22:21:03.4864625Z Building wheels for collected packages: torchvision
2024-08-20T22:22:21.2602943Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
2024-08-20T22:22:21.2635798Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116130 sha256=0a5e61583d43dfbbc3de89677cb1fde7139d4789964ff209a310eb9ea25dd7ed
2024-08-20T22:22:21.2637413Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-08-20T22:22:21.2671917Z Successfully built torchvision
2024-08-20T22:22:21.7541514Z Installing collected packages: torchvision
2024-08-20T22:22:22.1680517Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-08-20T22:22:22.3110402Z + '[' -n '' ']'
2024-08-20T22:22:22.3110972Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2024-08-20T22:22:22.3111480Z + pip_install git+https://github.com/pytorch/ao.git
2024-08-20T22:22:22.3112450Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git
2024-08-20T22:22:22.6480281Z Collecting git+https://github.com/pytorch/ao.git
2024-08-20T22:22:22.6484326Z   Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-uedda9bn
2024-08-20T22:22:22.6515091Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-uedda9bn
2024-08-20T22:22:23.2188681Z   Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e
2024-08-20T22:22:25.6373943Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T22:22:25.6410026Z [?25hBuilding wheels for collected packages: torchao
2024-08-20T22:23:44.0738756Z   Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / done
2024-08-20T22:23:44.0755168Z [?25h  Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928015 sha256=3843c6bbd0bea578dee6c2d04b7c1bb7f594f0a00da0a559f4accd7361eeb88e
2024-08-20T22:23:44.0757541Z   Stored in directory: /tmp/pip-ephem-wheel-cache-lcwnrgps/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e
2024-08-20T22:23:44.0787271Z Successfully built torchao
2024-08-20T22:23:44.6193360Z Installing collected packages: torchao
2024-08-20T22:23:44.8603787Z Successfully installed torchao-0.4.0+git2c8e3f3
2024-08-20T22:23:44.9677921Z + id=0
2024-08-20T22:23:44.9678393Z + pip_install opencv-python==4.8.0.74
2024-08-20T22:23:44.9678999Z + pip install --progress-bar off opencv-python==4.8.0.74
2024-08-20T22:23:45.4311900Z Collecting opencv-python==4.8.0.74
2024-08-20T22:23:45.4502357Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2024-08-20T22:23:45.4596821Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4)
2024-08-20T22:23:45.4643205Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2024-08-20T22:23:46.5502682Z Installing collected packages: opencv-python
2024-08-20T22:23:47.4194929Z Successfully installed opencv-python-4.8.0.74
2024-08-20T22:23:47.5168130Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]]
2024-08-20T22:23:47.5169902Z + [[ dynamic_aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]]
2024-08-20T22:23:47.5171606Z + [[ dynamic_aot_eager_torchbench == *torchbench_gcp_smoketest* ]]
2024-08-20T22:23:47.5172701Z + checkout_install_torchbench
2024-08-20T22:23:47.5173119Z + local commit
2024-08-20T22:23:47.5173460Z ++ get_pinned_commit torchbench
2024-08-20T22:23:47.5173895Z ++ cat .github/ci_commit_pins/torchbench.txt
2024-08-20T22:23:47.5191491Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e
2024-08-20T22:23:47.5192108Z + git clone https://github.com/pytorch/benchmark torchbench
2024-08-20T22:23:47.5209670Z Cloning into 'torchbench'...
2024-08-20T22:23:47.6644161Z remote: Enumerating objects: 31001, done.
2024-08-20T22:23:47.6644885Z remote: Counting objects:   0% (1/1741)
2024-08-20T22:23:47.6646064Z remote: Counting objects:   1% (18/1741)
2024-08-20T22:23:47.6648427Z remote: Counting objects:   2% (35/1741)
2024-08-20T22:23:47.6648991Z remote: Counting objects:   3% (53/1741)
2024-08-20T22:23:47.6649699Z remote: Counting objects:   4% (70/1741)
2024-08-20T22:23:47.6650314Z remote: Counting objects:   5% (88/1741)
2024-08-20T22:23:47.6650924Z remote: Counting objects:   6% (105/1741)
2024-08-20T22:23:47.6651560Z remote: Counting objects:   7% (122/1741)
2024-08-20T22:23:47.6652093Z remote: Counting objects:   8% (140/1741)
2024-08-20T22:23:47.6652612Z remote: Counting objects:   9% (157/1741)
2024-08-20T22:23:47.6653145Z remote: Counting objects:  10% (175/1741)
2024-08-20T22:23:47.6653811Z remote: Counting objects:  11% (192/1741)
2024-08-20T22:23:47.6654453Z remote: Counting objects:  12% (209/1741)
2024-08-20T22:23:47.6655113Z remote: Counting objects:  13% (227/1741)
2024-08-20T22:23:47.6655871Z remote: Counting objects:  14% (244/1741)
2024-08-20T22:23:47.6656565Z remote: Counting objects:  15% (262/1741)
2024-08-20T22:23:47.6657198Z remote: Counting objects:  16% (279/1741)
2024-08-20T22:23:47.6657796Z remote: Counting objects:  17% (296/1741)
2024-08-20T22:23:47.6658376Z remote: Counting objects:  18% (314/1741)
2024-08-20T22:23:47.6658901Z remote: Counting objects:  19% (331/1741)
2024-08-20T22:23:47.6659444Z remote: Counting objects:  20% (349/1741)
2024-08-20T22:23:47.6659973Z remote: Counting objects:  21% (366/1741)
2024-08-20T22:23:47.6660610Z remote: Counting objects:  22% (384/1741)
2024-08-20T22:23:47.6661271Z remote: Counting objects:  23% (401/1741)
2024-08-20T22:23:47.6661795Z remote: Counting objects:  24% (418/1741)
2024-08-20T22:23:47.6662320Z remote: Counting objects:  25% (436/1741)
2024-08-20T22:23:47.6662832Z remote: Counting objects:  26% (453/1741)
2024-08-20T22:23:47.6663339Z remote: Counting objects:  27% (471/1741)
2024-08-20T22:23:47.6663857Z remote: Counting objects:  28% (488/1741)
2024-08-20T22:23:47.6665521Z remote: Counting objects:  29% (505/1741)
2024-08-20T22:23:47.6666078Z remote: Counting objects:  30% (523/1741)
2024-08-20T22:23:47.6666595Z remote: Counting objects:  31% (540/1741)
2024-08-20T22:23:47.6667352Z remote: Counting objects:  32% (558/1741)
2024-08-20T22:23:47.6667871Z remote: Counting objects:  33% (575/1741)
2024-08-20T22:23:47.6668382Z remote: Counting objects:  34% (592/1741)
2024-08-20T22:23:47.6668884Z remote: Counting objects:  35% (610/1741)
2024-08-20T22:23:47.6669433Z remote: Counting objects:  36% (627/1741)
2024-08-20T22:23:47.6669944Z remote: Counting objects:  37% (645/1741)
2024-08-20T22:23:47.6670452Z remote: Counting objects:  38% (662/1741)
2024-08-20T22:23:47.6670952Z remote: Counting objects:  39% (679/1741)
2024-08-20T22:23:47.6671475Z remote: Counting objects:  40% (697/1741)
2024-08-20T22:23:47.6671982Z remote: Counting objects:  41% (714/1741)
2024-08-20T22:23:47.6672495Z remote: Counting objects:  42% (732/1741)
2024-08-20T22:23:47.6673009Z remote: Counting objects:  43% (749/1741)
2024-08-20T22:23:47.6673684Z remote: Counting objects:  44% (767/1741)
2024-08-20T22:23:47.6674191Z remote: Counting objects:  45% (784/1741)
2024-08-20T22:23:47.6674697Z remote: Counting objects:  46% (801/1741)
2024-08-20T22:23:47.6675200Z remote: Counting objects:  47% (819/1741)
2024-08-20T22:23:47.6675709Z remote: Counting objects:  48% (836/1741)
2024-08-20T22:23:47.6676215Z remote: Counting objects:  49% (854/1741)
2024-08-20T22:23:47.6676724Z remote: Counting objects:  50% (871/1741)
2024-08-20T22:23:47.6677227Z remote: Counting objects:  51% (888/1741)
2024-08-20T22:23:47.6677733Z remote: Counting objects:  52% (906/1741)
2024-08-20T22:23:47.6678240Z remote: Counting objects:  53% (923/1741)
2024-08-20T22:23:47.6678741Z remote: Counting objects:  54% (941/1741)
2024-08-20T22:23:47.6679252Z remote: Counting objects:  55% (958/1741)
2024-08-20T22:23:47.6679759Z remote: Counting objects:  56% (975/1741)
2024-08-20T22:23:47.6680270Z remote: Counting objects:  57% (993/1741)
2024-08-20T22:23:47.6680787Z remote: Counting objects:  58% (1010/1741)
2024-08-20T22:23:47.6681317Z remote: Counting objects:  59% (1028/1741)
2024-08-20T22:23:47.6681840Z remote: Counting objects:  60% (1045/1741)
2024-08-20T22:23:47.6682382Z remote: Counting objects:  61% (1063/1741)
2024-08-20T22:23:47.6682921Z remote: Counting objects:  62% (1080/1741)
2024-08-20T22:23:47.6683440Z remote: Counting objects:  63% (1097/1741)
2024-08-20T22:23:47.6683955Z remote: Counting objects:  64% (1115/1741)
2024-08-20T22:23:47.6684469Z remote: Counting objects:  65% (1132/1741)
2024-08-20T22:23:47.6684977Z remote: Counting objects:  66% (1150/1741)
2024-08-20T22:23:47.6685491Z remote: Counting objects:  67% (1167/1741)
2024-08-20T22:23:47.6686006Z remote: Counting objects:  68% (1184/1741)
2024-08-20T22:23:47.6686520Z remote: Counting objects:  69% (1202/1741)
2024-08-20T22:23:47.6687027Z remote: Counting objects:  70% (1219/1741)
2024-08-20T22:23:47.6687550Z remote: Counting objects:  71% (1237/1741)
2024-08-20T22:23:47.6688065Z remote: Counting objects:  72% (1254/1741)
2024-08-20T22:23:47.6688570Z remote: Counting objects:  73% (1271/1741)
2024-08-20T22:23:47.6689089Z remote: Counting objects:  74% (1289/1741)
2024-08-20T22:23:47.6689602Z remote: Counting objects:  75% (1306/1741)
2024-08-20T22:23:47.6690111Z remote: Counting objects:  76% (1324/1741)
2024-08-20T22:23:47.6690686Z remote: Counting objects:  77% (1341/1741)
2024-08-20T22:23:47.6691203Z remote: Counting objects:  78% (1358/1741)
2024-08-20T22:23:47.6691714Z remote: Counting objects:  79% (1376/1741)
2024-08-20T22:23:47.6692226Z remote: Counting objects:  80% (1393/1741)
2024-08-20T22:23:47.6692765Z remote: Counting objects:  81% (1411/1741)
2024-08-20T22:23:47.6693294Z remote: Counting objects:  82% (1428/1741)
2024-08-20T22:23:47.6693808Z remote: Counting objects:  83% (1446/1741)
2024-08-20T22:23:47.6694438Z remote: Counting objects:  84% (1463/1741)
2024-08-20T22:23:47.6694954Z remote: Counting objects:  85% (1480/1741)
2024-08-20T22:23:47.6695468Z remote: Counting objects:  86% (1498/1741)
2024-08-20T22:23:47.6695977Z remote: Counting objects:  87% (1515/1741)
2024-08-20T22:23:47.6696489Z remote: Counting objects:  88% (1533/1741)
2024-08-20T22:23:47.6696997Z remote: Counting objects:  89% (1550/1741)
2024-08-20T22:23:47.6697507Z remote: Counting objects:  90% (1567/1741)
2024-08-20T22:23:47.6698018Z remote: Counting objects:  91% (1585/1741)
2024-08-20T22:23:47.6698527Z remote: Counting objects:  92% (1602/1741)
2024-08-20T22:23:47.6699032Z remote: Counting objects:  93% (1620/1741)
2024-08-20T22:23:47.6699541Z remote: Counting objects:  94% (1637/1741)
2024-08-20T22:23:47.6700065Z remote: Counting objects:  95% (1654/1741)
2024-08-20T22:23:47.6700577Z remote: Counting objects:  96% (1672/1741)
2024-08-20T22:23:47.6701082Z remote: Counting objects:  97% (1689/1741)
2024-08-20T22:23:47.6701598Z remote: Counting objects:  98% (1707/1741)
2024-08-20T22:23:47.6702196Z remote: Counting objects:  99% (1724/1741)
2024-08-20T22:23:47.6702743Z remote: Counting objects: 100% (1741/1741)
2024-08-20T22:23:47.6703317Z remote: Counting objects: 100% (1741/1741), done.
2024-08-20T22:23:47.6703873Z remote: Compressing objects:   0% (1/900)
2024-08-20T22:23:47.6783337Z remote: Compressing objects:   1% (9/900)
2024-08-20T22:23:47.6863799Z remote: Compressing objects:   2% (18/900)
2024-08-20T22:23:47.6948955Z remote: Compressing objects:   3% (27/900)
2024-08-20T22:23:47.6999169Z remote: Compressing objects:   4% (36/900)
2024-08-20T22:23:47.7047622Z remote: Compressing objects:   5% (45/900)
2024-08-20T22:23:47.7081331Z remote: Compressing objects:   6% (54/900)
2024-08-20T22:23:47.7109674Z remote: Compressing objects:   7% (63/900)
2024-08-20T22:23:47.7142070Z remote: Compressing objects:   8% (72/900)
2024-08-20T22:23:47.7179149Z remote: Compressing objects:   9% (81/900)
2024-08-20T22:23:47.7206871Z remote: Compressing objects:  10% (90/900)
2024-08-20T22:23:47.7223632Z remote: Compressing objects:  11% (99/900)
2024-08-20T22:23:47.7234971Z remote: Compressing objects:  12% (108/900)
2024-08-20T22:23:47.7249504Z remote: Compressing objects:  13% (117/900)
2024-08-20T22:23:47.7266405Z remote: Compressing objects:  14% (126/900)
2024-08-20T22:23:47.7276254Z remote: Compressing objects:  15% (135/900)
2024-08-20T22:23:47.7284300Z remote: Compressing objects:  16% (144/900)
2024-08-20T22:23:47.7290778Z remote: Compressing objects:  17% (153/900)
2024-08-20T22:23:47.7294064Z remote: Compressing objects:  18% (162/900)
2024-08-20T22:23:47.7298591Z remote: Compressing objects:  19% (171/900)
2024-08-20T22:23:47.7299227Z remote: Compressing objects:  20% (180/900)
2024-08-20T22:23:47.7300135Z remote: Compressing objects:  21% (189/900)
2024-08-20T22:23:47.7300773Z remote: Compressing objects:  22% (198/900)
2024-08-20T22:23:47.7304083Z remote: Compressing objects:  23% (207/900)
2024-08-20T22:23:47.7304667Z remote: Compressing objects:  24% (216/900)
2024-08-20T22:23:47.7305564Z remote: Compressing objects:  25% (225/900)
2024-08-20T22:23:47.7309674Z remote: Compressing objects:  26% (234/900)
2024-08-20T22:23:47.7316515Z remote: Compressing objects:  27% (243/900)
2024-08-20T22:23:47.7319562Z remote: Compressing objects:  28% (252/900)
2024-08-20T22:23:47.7322853Z remote: Compressing objects:  29% (261/900)
2024-08-20T22:23:47.7329464Z remote: Compressing objects:  30% (270/900)
2024-08-20T22:23:47.7330033Z remote: Compressing objects:  31% (279/900)
2024-08-20T22:23:47.7330693Z remote: Compressing objects:  32% (288/900)
2024-08-20T22:23:47.7334697Z remote: Compressing objects:  33% (297/900)
2024-08-20T22:23:47.7337159Z remote: Compressing objects:  34% (306/900)
2024-08-20T22:23:47.7340623Z remote: Compressing objects:  35% (315/900)
2024-08-20T22:23:47.7342199Z remote: Compressing objects:  36% (324/900)
2024-08-20T22:23:47.7344400Z remote: Compressing objects:  37% (333/900)
2024-08-20T22:23:47.7346550Z remote: Compressing objects:  38% (342/900)
2024-08-20T22:23:47.7347942Z remote: Compressing objects:  39% (351/900)
2024-08-20T22:23:47.7349369Z remote: Compressing objects:  40% (360/900)
2024-08-20T22:23:47.7351704Z remote: Compressing objects:  41% (369/900)
2024-08-20T22:23:47.7352377Z remote: Compressing objects:  42% (378/900)
2024-08-20T22:23:47.7353395Z remote: Compressing objects:  43% (387/900)
2024-08-20T22:23:47.7354131Z remote: Compressing objects:  44% (396/900)
2024-08-20T22:23:47.7354837Z remote: Compressing objects:  45% (405/900)
2024-08-20T22:23:47.7355917Z remote: Compressing objects:  46% (414/900)
2024-08-20T22:23:47.7356666Z remote: Compressing objects:  47% (423/900)
2024-08-20T22:23:47.7357381Z remote: Compressing objects:  48% (432/900)
2024-08-20T22:23:47.7357929Z remote: Compressing objects:  49% (441/900)
2024-08-20T22:23:47.7358478Z remote: Compressing objects:  50% (450/900)
2024-08-20T22:23:47.7359311Z remote: Compressing objects:  51% (459/900)
2024-08-20T22:23:47.7359896Z remote: Compressing objects:  52% (468/900)
2024-08-20T22:23:47.7360604Z remote: Compressing objects:  53% (477/900)
2024-08-20T22:23:47.7361272Z remote: Compressing objects:  54% (486/900)
2024-08-20T22:23:47.7362003Z remote: Compressing objects:  55% (495/900)
2024-08-20T22:23:47.7362696Z remote: Compressing objects:  56% (504/900)
2024-08-20T22:23:47.7363280Z remote: Compressing objects:  57% (513/900)
2024-08-20T22:23:47.7363816Z remote: Compressing objects:  58% (522/900)
2024-08-20T22:23:47.7364351Z remote: Compressing objects:  59% (531/900)
2024-08-20T22:23:47.7364871Z remote: Compressing objects:  60% (540/900)
2024-08-20T22:23:47.7365517Z remote: Compressing objects:  61% (549/900)
2024-08-20T22:23:47.7366054Z remote: Compressing objects:  62% (558/900)
2024-08-20T22:23:47.7366600Z remote: Compressing objects:  63% (567/900)
2024-08-20T22:23:47.7367461Z remote: Compressing objects:  64% (576/900)
2024-08-20T22:23:47.7368035Z remote: Compressing objects:  65% (585/900)
2024-08-20T22:23:47.7368583Z remote: Compressing objects:  66% (594/900)
2024-08-20T22:23:47.7369112Z remote: Compressing objects:  67% (603/900)
2024-08-20T22:23:47.7369653Z remote: Compressing objects:  68% (612/900)
2024-08-20T22:23:47.7370183Z remote: Compressing objects:  69% (621/900)
2024-08-20T22:23:47.7370817Z remote: Compressing objects:  70% (630/900)
2024-08-20T22:23:47.7371343Z remote: Compressing objects:  71% (639/900)
2024-08-20T22:23:47.7371869Z remote: Compressing objects:  72% (648/900)
2024-08-20T22:23:47.7372399Z remote: Compressing objects:  73% (657/900)
2024-08-20T22:23:47.7372932Z remote: Compressing objects:  74% (666/900)
2024-08-20T22:23:47.7373508Z remote: Compressing objects:  75% (675/900)
2024-08-20T22:23:47.7376980Z remote: Compressing objects:  76% (684/900)
2024-08-20T22:23:47.7384170Z remote: Compressing objects:  77% (693/900)
2024-08-20T22:23:47.7389230Z remote: Compressing objects:  78% (702/900)
2024-08-20T22:23:47.7396746Z remote: Compressing objects:  79% (711/900)
2024-08-20T22:23:47.7401704Z remote: Compressing objects:  80% (720/900)
2024-08-20T22:23:47.7410049Z remote: Compressing objects:  81% (729/900)
2024-08-20T22:23:47.7418378Z remote: Compressing objects:  82% (738/900)
2024-08-20T22:23:47.7421305Z remote: Compressing objects:  83% (747/900)
2024-08-20T22:23:47.7424463Z remote: Compressing objects:  84% (756/900)
2024-08-20T22:23:47.7429212Z remote: Compressing objects:  85% (765/900)
2024-08-20T22:23:47.7434057Z remote: Compressing objects:  86% (774/900)
2024-08-20T22:23:47.7437978Z remote: Compressing objects:  87% (783/900)
2024-08-20T22:23:47.7441958Z remote: Compressing objects:  88% (792/900)
2024-08-20T22:23:47.7445535Z remote: Compressing objects:  89% (801/900)
2024-08-20T22:23:47.7449502Z remote: Compressing objects:  90% (810/900)
2024-08-20T22:23:47.7453868Z remote: Compressing objects:  91% (819/900)
2024-08-20T22:23:47.7456006Z remote: Compressing objects:  92% (828/900)
2024-08-20T22:23:47.7461128Z remote: Compressing objects:  93% (837/900)
2024-08-20T22:23:47.7461682Z remote: Compressing objects:  94% (846/900)
2024-08-20T22:23:47.7463722Z remote: Compressing objects:  95% (855/900)
2024-08-20T22:23:47.7467507Z remote: Compressing objects:  96% (864/900)
2024-08-20T22:23:47.7469650Z remote: Compressing objects:  97% (873/900)
2024-08-20T22:23:47.7470720Z remote: Compressing objects:  98% (882/900)
2024-08-20T22:23:47.7471336Z remote: Compressing objects:  99% (891/900)
2024-08-20T22:23:47.7471952Z remote: Compressing objects: 100% (900/900)
2024-08-20T22:23:47.7472790Z remote: Compressing objects: 100% (900/900), done.
2024-08-20T22:23:47.7550945Z Receiving objects:   0% (1/31001)
2024-08-20T22:23:47.7593726Z Receiving objects:   1% (311/31001)
2024-08-20T22:23:47.7633397Z Receiving objects:   2% (621/31001)
2024-08-20T22:23:47.7674054Z Receiving objects:   3% (931/31001)
2024-08-20T22:23:47.7708473Z Receiving objects:   4% (1241/31001)
2024-08-20T22:23:47.7742253Z Receiving objects:   5% (1551/31001)
2024-08-20T22:23:47.7778988Z Receiving objects:   6% (1861/31001)
2024-08-20T22:23:47.7817977Z Receiving objects:   7% (2171/31001)
2024-08-20T22:23:47.7847888Z Receiving objects:   8% (2481/31001)
2024-08-20T22:23:47.7881485Z Receiving objects:   9% (2791/31001)
2024-08-20T22:23:47.7950639Z Receiving objects:  10% (3101/31001)
2024-08-20T22:23:47.8000299Z Receiving objects:  11% (3411/31001)
2024-08-20T22:23:47.8071593Z Receiving objects:  12% (3721/31001)
2024-08-20T22:23:47.8204285Z Receiving objects:  13% (4031/31001)
2024-08-20T22:23:47.8349365Z Receiving objects:  14% (4341/31001)
2024-08-20T22:23:47.8426443Z Receiving objects:  15% (4651/31001)
2024-08-20T22:23:47.8499569Z Receiving objects:  16% (4961/31001)
2024-08-20T22:23:47.8558931Z Receiving objects:  17% (5271/31001)
2024-08-20T22:23:47.8587219Z Receiving objects:  18% (5581/31001)
2024-08-20T22:23:47.8644715Z Receiving objects:  19% (5891/31001)
2024-08-20T22:23:47.8679719Z Receiving objects:  20% (6201/31001)
2024-08-20T22:23:47.8696436Z Receiving objects:  21% (6511/31001)
2024-08-20T22:23:47.8883237Z Receiving objects:  22% (6821/31001)
2024-08-20T22:23:48.7498597Z Receiving objects:  23% (7131/31001)
2024-08-20T22:23:48.8808972Z Receiving objects:  23% (7375/31001), 75.28 MiB | 75.42 MiB/s
2024-08-20T22:23:49.4351843Z Receiving objects:  24% (7441/31001), 75.28 MiB | 75.42 MiB/s
2024-08-20T22:23:49.5074373Z Receiving objects:  25% (7751/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.5803019Z Receiving objects:  26% (8061/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.6520668Z Receiving objects:  27% (8371/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.7234267Z Receiving objects:  28% (8681/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.7497461Z Receiving objects:  29% (8991/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.7961719Z Receiving objects:  29% (9102/31001), 112.24 MiB | 74.92 MiB/s
2024-08-20T22:23:49.8678029Z Receiving objects:  30% (9301/31001), 150.06 MiB | 75.10 MiB/s
2024-08-20T22:23:49.9401679Z Receiving objects:  31% (9611/31001), 150.06 MiB | 75.10 MiB/s
2024-08-20T22:23:50.1742088Z Receiving objects:  32% (9921/31001), 150.06 MiB | 75.10 MiB/s
2024-08-20T22:23:50.2479332Z Receiving objects:  33% (10231/31001), 150.06 MiB | 75.10 MiB/s
2024-08-20T22:23:50.2602101Z Receiving objects:  34% (10541/31001), 150.06 MiB | 75.10 MiB/s
2024-08-20T22:23:50.5201746Z Receiving objects:  35% (10851/31001), 184.36 MiB | 73.80 MiB/s
2024-08-20T22:23:50.7500885Z Receiving objects:  36% (11161/31001), 184.36 MiB | 73.80 MiB/s
2024-08-20T22:23:50.7744927Z Receiving objects:  36% (11303/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7763614Z Receiving objects:  37% (11471/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7786486Z Receiving objects:  38% (11781/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7804945Z Receiving objects:  39% (12091/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7819811Z Receiving objects:  40% (12401/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7830810Z Receiving objects:  41% (12711/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7843738Z Receiving objects:  42% (13021/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7859645Z Receiving objects:  43% (13331/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7876447Z Receiving objects:  44% (13641/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7884381Z Receiving objects:  45% (13951/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7889357Z Receiving objects:  46% (14261/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7903773Z Receiving objects:  47% (14571/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7924911Z Receiving objects:  48% (14881/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.7938110Z Receiving objects:  49% (15191/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8140743Z Receiving objects:  50% (15501/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8146995Z Receiving objects:  51% (15811/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8153702Z Receiving objects:  52% (16121/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8159999Z Receiving objects:  53% (16431/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8165453Z Receiving objects:  54% (16741/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8183027Z Receiving objects:  55% (17051/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8194004Z Receiving objects:  56% (17361/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8200184Z Receiving objects:  57% (17671/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8214254Z Receiving objects:  58% (17981/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8226140Z Receiving objects:  59% (18291/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8236709Z Receiving objects:  60% (18601/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8376324Z Receiving objects:  61% (18911/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8382467Z Receiving objects:  62% (19221/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8388349Z Receiving objects:  63% (19531/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8396470Z Receiving objects:  64% (19841/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8404289Z Receiving objects:  65% (20151/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8413362Z Receiving objects:  66% (20461/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.8674114Z Receiving objects:  67% (20771/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9079394Z Receiving objects:  68% (21081/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9087267Z Receiving objects:  69% (21391/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9101285Z Receiving objects:  70% (21701/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9112454Z Receiving objects:  71% (22011/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9127267Z Receiving objects:  72% (22321/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9136649Z Receiving objects:  73% (22631/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9151822Z Receiving objects:  74% (22941/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9162702Z Receiving objects:  75% (23251/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9170330Z Receiving objects:  76% (23561/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9178727Z Receiving objects:  77% (23871/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9393721Z Receiving objects:  78% (24181/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9427516Z Receiving objects:  79% (24491/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9436636Z Receiving objects:  80% (24801/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9448819Z Receiving objects:  81% (25111/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9462242Z Receiving objects:  82% (25421/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9510846Z Receiving objects:  83% (25731/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9651752Z Receiving objects:  84% (26041/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9657233Z Receiving objects:  85% (26351/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9665254Z Receiving objects:  86% (26661/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9670907Z Receiving objects:  87% (26971/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9678408Z Receiving objects:  88% (27281/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9684395Z Receiving objects:  89% (27591/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9797762Z Receiving objects:  90% (27901/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9810892Z Receiving objects:  91% (28211/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9822575Z Receiving objects:  92% (28521/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9839547Z Receiving objects:  93% (28831/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9873448Z Receiving objects:  94% (29141/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9884857Z Receiving objects:  95% (29451/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9892893Z Receiving objects:  96% (29761/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9897800Z Receiving objects:  97% (30071/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9968457Z Receiving objects:  98% (30381/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9990737Z Receiving objects:  99% (30691/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:50.9991841Z remote: Total 31001 (delta 1012), reused 1473 (delta 822), pack-reused 29260 (from 1)
2024-08-20T22:23:51.0008558Z Receiving objects: 100% (31001/31001), 220.14 MiB | 73.43 MiB/s
2024-08-20T22:23:51.0009402Z Receiving objects: 100% (31001/31001), 232.50 MiB | 71.52 MiB/s, done.
2024-08-20T22:23:51.0039637Z Resolving deltas:   0% (0/16725)
2024-08-20T22:23:51.0055112Z Resolving deltas:   1% (168/16725)
2024-08-20T22:23:51.0065147Z Resolving deltas:   2% (396/16725)
2024-08-20T22:23:51.0089374Z Resolving deltas:   3% (513/16725)
2024-08-20T22:23:51.0105232Z Resolving deltas:   4% (674/16725)
2024-08-20T22:23:51.0115163Z Resolving deltas:   5% (901/16725)
2024-08-20T22:23:51.0124738Z Resolving deltas:   6% (1024/16725)
2024-08-20T22:23:51.0137997Z Resolving deltas:   7% (1172/16725)
2024-08-20T22:23:51.0147095Z Resolving deltas:   8% (1357/16725)
2024-08-20T22:23:51.0159358Z Resolving deltas:   9% (1510/16725)
2024-08-20T22:23:51.0169851Z Resolving deltas:  10% (1677/16725)
2024-08-20T22:23:51.0184326Z Resolving deltas:  11% (1840/16725)
2024-08-20T22:23:51.0198629Z Resolving deltas:  12% (2013/16725)
2024-08-20T22:23:51.0211502Z Resolving deltas:  13% (2211/16725)
2024-08-20T22:23:51.0224810Z Resolving deltas:  14% (2400/16725)
2024-08-20T22:23:51.0240873Z Resolving deltas:  15% (2511/16725)
2024-08-20T22:23:51.0256906Z Resolving deltas:  16% (2681/16725)
2024-08-20T22:23:51.0270371Z Resolving deltas:  17% (2864/16725)
2024-08-20T22:23:51.0321338Z Resolving deltas:  18% (3018/16725)
2024-08-20T22:23:51.0342808Z Resolving deltas:  19% (3198/16725)
2024-08-20T22:23:51.0343398Z Resolving deltas:  20% (3491/16725)
2024-08-20T22:23:51.0354613Z Resolving deltas:  21% (3513/16725)
2024-08-20T22:23:51.0365049Z Resolving deltas:  22% (3688/16725)
2024-08-20T22:23:51.0376709Z Resolving deltas:  23% (3857/16725)
2024-08-20T22:23:51.0386304Z Resolving deltas:  24% (4039/16725)
2024-08-20T22:23:51.0427590Z Resolving deltas:  25% (4184/16725)
2024-08-20T22:23:51.0446037Z Resolving deltas:  26% (4361/16725)
2024-08-20T22:23:51.0460683Z Resolving deltas:  27% (4529/16725)
2024-08-20T22:23:51.0477145Z Resolving deltas:  28% (4683/16725)
2024-08-20T22:23:51.0493909Z Resolving deltas:  29% (4853/16725)
2024-08-20T22:23:51.0506631Z Resolving deltas:  30% (5038/16725)
2024-08-20T22:23:51.0519923Z Resolving deltas:  31% (5187/16725)
2024-08-20T22:23:51.0527693Z Resolving deltas:  32% (5404/16725)
2024-08-20T22:23:51.0543636Z Resolving deltas:  33% (5521/16725)
2024-08-20T22:23:51.0562749Z Resolving deltas:  34% (5698/16725)
2024-08-20T22:23:51.0582470Z Resolving deltas:  35% (5854/16725)
2024-08-20T22:23:51.0596006Z Resolving deltas:  36% (6034/16725)
2024-08-20T22:23:51.0607316Z Resolving deltas:  37% (6228/16725)
2024-08-20T22:23:51.0624925Z Resolving deltas:  38% (6359/16725)
2024-08-20T22:23:51.0634944Z Resolving deltas:  39% (6532/16725)
2024-08-20T22:23:51.0651843Z Resolving deltas:  40% (6692/16725)
2024-08-20T22:23:51.0665151Z Resolving deltas:  41% (6884/16725)
2024-08-20T22:23:51.0674260Z Resolving deltas:  42% (7033/16725)
2024-08-20T22:23:51.0683756Z Resolving deltas:  43% (7222/16725)
2024-08-20T22:23:51.0702219Z Resolving deltas:  44% (7359/16725)
2024-08-20T22:23:51.0723618Z Resolving deltas:  45% (7544/16725)
2024-08-20T22:23:51.0735877Z Resolving deltas:  46% (7695/16725)
2024-08-20T22:23:51.0758822Z Resolving deltas:  47% (7861/16725)
2024-08-20T22:23:51.0766339Z Resolving deltas:  48% (8030/16725)
2024-08-20T22:23:51.0775214Z Resolving deltas:  49% (8196/16725)
2024-08-20T22:23:51.0784304Z Resolving deltas:  50% (8363/16725)
2024-08-20T22:23:51.0812917Z Resolving deltas:  51% (8531/16725)
2024-08-20T22:23:51.0813617Z Resolving deltas:  52% (8863/16725)
2024-08-20T22:23:51.0823159Z Resolving deltas:  53% (8870/16725)
2024-08-20T22:23:51.0831387Z Resolving deltas:  54% (9063/16725)
2024-08-20T22:23:51.0847239Z Resolving deltas:  55% (9201/16725)
2024-08-20T22:23:51.0856403Z Resolving deltas:  56% (9366/16725)
2024-08-20T22:23:51.0866776Z Resolving deltas:  57% (9570/16725)
2024-08-20T22:23:51.0871303Z Resolving deltas:  58% (9831/16725)
2024-08-20T22:23:51.0883701Z Resolving deltas:  59% (9896/16725)
2024-08-20T22:23:51.0904735Z Resolving deltas:  60% (10052/16725)
2024-08-20T22:23:51.0912213Z Resolving deltas:  61% (10337/16725)
2024-08-20T22:23:51.0916387Z Resolving deltas:  62% (10477/16725)
2024-08-20T22:23:51.0939085Z Resolving deltas:  63% (10544/16725)
2024-08-20T22:23:51.0961885Z Resolving deltas:  65% (10902/16725)
2024-08-20T22:23:51.0984970Z Resolving deltas:  67% (11220/16725)
2024-08-20T22:23:51.0992858Z Resolving deltas:  68% (11417/16725)
2024-08-20T22:23:51.1004952Z Resolving deltas:  69% (11586/16725)
2024-08-20T22:23:51.1020012Z Resolving deltas:  70% (11746/16725)
2024-08-20T22:23:51.1032453Z Resolving deltas:  71% (11876/16725)
2024-08-20T22:23:51.1042516Z Resolving deltas:  72% (12044/16725)
2024-08-20T22:23:51.1054413Z Resolving deltas:  73% (12210/16725)
2024-08-20T22:23:51.1063485Z Resolving deltas:  74% (12382/16725)
2024-08-20T22:23:51.1073238Z Resolving deltas:  75% (12603/16725)
2024-08-20T22:23:51.1079761Z Resolving deltas:  76% (12745/16725)
2024-08-20T22:23:51.1087922Z Resolving deltas:  77% (12895/16725)
2024-08-20T22:23:51.1100537Z Resolving deltas:  78% (13046/16725)
2024-08-20T22:23:51.1110147Z Resolving deltas:  79% (13216/16725)
2024-08-20T22:23:51.1119272Z Resolving deltas:  80% (13381/16725)
2024-08-20T22:23:51.1129104Z Resolving deltas:  81% (13566/16725)
2024-08-20T22:23:51.1138911Z Resolving deltas:  82% (13738/16725)
2024-08-20T22:23:51.1148573Z Resolving deltas:  83% (13884/16725)
2024-08-20T22:23:51.1156673Z Resolving deltas:  84% (14078/16725)
2024-08-20T22:23:51.1163401Z Resolving deltas:  85% (14244/16725)
2024-08-20T22:23:51.1173970Z Resolving deltas:  86% (14386/16725)
2024-08-20T22:23:51.1182091Z Resolving deltas:  87% (14552/16725)
2024-08-20T22:23:51.1191854Z Resolving deltas:  88% (14721/16725)
2024-08-20T22:23:51.1198351Z Resolving deltas:  89% (14905/16725)
2024-08-20T22:23:51.1207042Z Resolving deltas:  90% (15064/16725)
2024-08-20T22:23:51.1215978Z Resolving deltas:  91% (15221/16725)
2024-08-20T22:23:51.1223334Z Resolving deltas:  92% (15387/16725)
2024-08-20T22:23:51.1232468Z Resolving deltas:  93% (15555/16725)
2024-08-20T22:23:51.1242095Z Resolving deltas:  94% (15722/16725)
2024-08-20T22:23:51.1251165Z Resolving deltas:  95% (15905/16725)
2024-08-20T22:23:51.1260753Z Resolving deltas:  96% (16111/16725)
2024-08-20T22:23:51.1267812Z Resolving deltas:  97% (16294/16725)
2024-08-20T22:23:51.1313105Z Resolving deltas:  98% (16391/16725)
2024-08-20T22:23:51.1388228Z Resolving deltas:  99% (16560/16725)
2024-08-20T22:23:51.1388665Z Resolving deltas: 100% (16725/16725)
2024-08-20T22:23:51.1389100Z Resolving deltas: 100% (16725/16725), done.
2024-08-20T22:23:52.1410877Z + pushd torchbench
2024-08-20T22:23:52.1411369Z ~/workspace/torchbench ~/workspace
2024-08-20T22:23:52.1411882Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e
2024-08-20T22:23:52.2195524Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'.
2024-08-20T22:23:52.2196075Z 
2024-08-20T22:23:52.2196581Z You are in 'detached HEAD' state. You can look around, make experimental
2024-08-20T22:23:52.2197539Z changes and commit them, and you can discard any commits you make in this
2024-08-20T22:23:52.2198290Z state without impacting any branches by switching back to a branch.
2024-08-20T22:23:52.2198726Z 
2024-08-20T22:23:52.2199033Z If you want to create a new branch to retain commits you create, you may
2024-08-20T22:23:52.2199847Z do so (now or later) by using -c with the switch command. Example:
2024-08-20T22:23:52.2200503Z 
2024-08-20T22:23:52.2200700Z   git switch -c <new-branch-name>
2024-08-20T22:23:52.2200985Z 
2024-08-20T22:23:52.2201126Z Or undo this operation with:
2024-08-20T22:23:52.2201378Z 
2024-08-20T22:23:52.2201499Z   git switch -
2024-08-20T22:23:52.2201679Z 
2024-08-20T22:23:52.2202018Z Turn off this advice by setting config variable advice.detachedHead to false
2024-08-20T22:23:52.2202493Z 
2024-08-20T22:23:52.2202664Z HEAD is now at 23512dbe Pin numpy version
2024-08-20T22:23:52.2207415Z + '[' '' ']'
2024-08-20T22:23:52.2207862Z + python install.py --continue_on_fail
2024-08-20T22:23:54.5277588Z checking packages torch, torchvision, torchaudio are installed...OK
2024-08-20T22:24:12.8046166Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2024-08-20T22:24:15.9640091Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2024-08-20T22:24:17.8565962Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2024-08-20T22:24:19.4839294Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2024-08-20T22:24:19.5070191Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2024-08-20T22:24:27.1004232Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2024-08-20T22:24:29.8136707Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2024-08-20T22:24:32.5321188Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2024-08-20T22:24:35.2261055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2024-08-20T22:24:35.2263191Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2024-08-20T22:24:36.2782573Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2024-08-20T22:24:39.0662447Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2024-08-20T22:24:39.0902234Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2024-08-20T22:25:36.2515940Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2024-08-20T22:25:44.8923992Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2024-08-20T22:25:51.9548607Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2024-08-20T22:25:58.7328223Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2024-08-20T22:26:07.1186975Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2024-08-20T22:26:14.0134251Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2024-08-20T22:26:20.6004864Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2024-08-20T22:26:27.4887257Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2024-08-20T22:26:34.5881963Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2024-08-20T22:26:41.9690060Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2024-08-20T22:26:48.7643279Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2024-08-20T22:26:55.6983645Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2024-08-20T22:26:56.8920811Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2024-08-20T22:27:32.1493463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2024-08-20T22:27:36.7746864Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2024-08-20T22:27:41.9846897Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2024-08-20T22:27:52.9102637Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2024-08-20T22:27:54.0937895Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2024-08-20T22:27:55.5589460Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2024-08-20T22:28:02.0480717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2024-08-20T22:28:09.0939773Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2024-08-20T22:28:14.5734757Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2024-08-20T22:28:22.2296892Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2024-08-20T22:28:28.1348497Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2024-08-20T22:28:33.2639642Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2024-08-20T22:28:39.4344287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2024-08-20T22:28:55.8381263Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2024-08-20T22:29:01.9920536Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2024-08-20T22:29:05.8916975Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2024-08-20T22:29:10.6244410Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2024-08-20T22:29:17.3913442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2024-08-20T22:29:17.3916675Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2024-08-20T22:29:30.3836486Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2024-08-20T22:29:34.2925280Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2024-08-20T22:29:34.2927559Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found
2024-08-20T22:29:41.9304665Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK
2024-08-20T22:29:43.1592733Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2024-08-20T22:29:44.4090461Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2024-08-20T22:30:29.3804190Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2024-08-20T22:31:58.7596082Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK
2024-08-20T22:31:58.7837903Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2024-08-20T22:32:00.3184083Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2024-08-20T22:32:00.3426377Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2024-08-20T22:32:00.3660100Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2024-08-20T22:32:00.3895701Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2024-08-20T22:32:00.4134411Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2024-08-20T22:32:00.4375664Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2024-08-20T22:32:21.5596488Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK
2024-08-20T22:32:21.5599236Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found
2024-08-20T22:32:22.8085467Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2024-08-20T22:32:25.2399013Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2024-08-20T22:32:25.2641195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2024-08-20T22:32:25.2878630Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2024-08-20T22:32:25.3123727Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2024-08-20T22:32:25.3367466Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2024-08-20T22:32:25.3614610Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2024-08-20T22:32:31.9925358Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2024-08-20T22:32:33.4323972Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2024-08-20T22:32:36.9409427Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2024-08-20T22:32:36.9654453Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2024-08-20T22:32:36.9896032Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2024-08-20T22:32:37.0136521Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2024-08-20T22:32:37.0382231Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2024-08-20T22:32:37.0620299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2024-08-20T22:32:53.6551943Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2024-08-20T22:33:13.7630442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK
2024-08-20T22:33:13.7912495Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2024-08-20T22:33:13.7915837Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found
2024-08-20T22:33:13.7919358Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found
2024-08-20T22:33:16.4780857Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2024-08-20T22:33:18.2146879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2024-08-20T22:33:18.2393262Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2024-08-20T22:33:41.9633788Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK
2024-08-20T22:33:48.5455463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK
2024-08-20T22:33:51.8575636Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2024-08-20T22:33:58.3136274Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2024-08-20T22:33:58.3382023Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2024-08-20T22:33:58.3625777Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2024-08-20T22:33:58.3869212Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2024-08-20T22:33:58.4116776Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2024-08-20T22:33:58.4360304Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2024-08-20T22:33:58.4601974Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2024-08-20T22:33:58.4842588Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2024-08-20T22:34:05.7342596Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK
2024-08-20T22:34:17.8627648Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2024-08-20T22:34:17.8874931Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2024-08-20T22:34:19.9820547Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2024-08-20T22:34:21.9540145Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2024-08-20T22:34:22.4471676Z + echo 'Print all dependencies after TorchBench is installed'
2024-08-20T22:34:22.4472966Z Print all dependencies after TorchBench is installed
2024-08-20T22:34:22.4474029Z + python -mpip freeze
2024-08-20T22:34:22.9468017Z absl-py==2.1.0
2024-08-20T22:34:22.9469169Z accelerate==0.33.0
2024-08-20T22:34:22.9469704Z aiohappyeyeballs==2.4.0
2024-08-20T22:34:22.9470070Z aiohttp==3.10.5
2024-08-20T22:34:22.9470434Z aiosignal==1.3.1
2024-08-20T22:34:22.9471063Z annotated-types==0.7.0
2024-08-20T22:34:22.9471583Z antlr4-python3-runtime==4.9.3
2024-08-20T22:34:22.9471965Z anyascii==0.3.2
2024-08-20T22:34:22.9476827Z astroid==3.2.4
2024-08-20T22:34:22.9477267Z astunparse==1.6.3
2024-08-20T22:34:22.9477705Z async-timeout==4.0.3
2024-08-20T22:34:22.9478118Z attrs==23.1.0
2024-08-20T22:34:22.9478456Z audioread==3.0.1
2024-08-20T22:34:22.9478802Z babel==2.16.0
2024-08-20T22:34:22.9479110Z beautifulsoup4==4.12.3
2024-08-20T22:34:22.9480461Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch
2024-08-20T22:34:22.9481731Z bibtexparser==2.0.0b7
2024-08-20T22:34:22.9485275Z black==24.8.0
2024-08-20T22:34:22.9485721Z blinker==1.8.2
2024-08-20T22:34:22.9486119Z blis==0.7.11
2024-08-20T22:34:22.9486535Z blobfile==2.1.1
2024-08-20T22:34:22.9486848Z bokeh==3.5.1
2024-08-20T22:34:22.9487141Z boto3==1.19.12
2024-08-20T22:34:22.9487435Z botocore==1.22.12
2024-08-20T22:34:22.9488080Z bs4==0.0.2
2024-08-20T22:34:22.9488822Z cachetools==5.5.0
2024-08-20T22:34:22.9489144Z cardboardlint==1.3.1
2024-08-20T22:34:22.9489469Z catalogue==2.0.10
2024-08-20T22:34:22.9489778Z certifi==2024.7.4
2024-08-20T22:34:22.9490089Z cffi==1.17.0
2024-08-20T22:34:22.9490544Z charset-normalizer==3.3.2
2024-08-20T22:34:22.9491097Z click==8.1.7
2024-08-20T22:34:22.9491486Z clldutils==3.22.2
2024-08-20T22:34:22.9491879Z cloudpathlib==0.18.1
2024-08-20T22:34:22.9492283Z cloudpickle==3.0.0
2024-08-20T22:34:22.9492674Z colorama==0.4.6
2024-08-20T22:34:22.9493053Z colorlog==6.8.2
2024-08-20T22:34:22.9493510Z confection==0.1.5
2024-08-20T22:34:22.9493899Z contourpy==1.2.1
2024-08-20T22:34:22.9494295Z coremltools==5.0b5
2024-08-20T22:34:22.9494746Z cryptography==43.0.0
2024-08-20T22:34:22.9495155Z csvw==3.3.0
2024-08-20T22:34:22.9495440Z cycler==0.12.1
2024-08-20T22:34:22.9495728Z cymem==2.0.8
2024-08-20T22:34:22.9496070Z DALL-E==0.1
2024-08-20T22:34:22.9496356Z datasets==2.21.0
2024-08-20T22:34:22.9496672Z decorator==5.1.1
2024-08-20T22:34:22.9497181Z defusedxml==0.7.1
2024-08-20T22:34:22.9497490Z Deprecated==1.2.14
2024-08-20T22:34:22.9498214Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e
2024-08-20T22:34:22.9498986Z diffusers==0.20.2
2024-08-20T22:34:22.9499297Z dill==0.3.8
2024-08-20T22:34:22.9499580Z diskcache==5.6.3
2024-08-20T22:34:22.9499881Z distro==1.9.0
2024-08-20T22:34:22.9500166Z dlinfo==1.2.1
2024-08-20T22:34:22.9500518Z docker-pycreds==0.4.0
2024-08-20T22:34:22.9500850Z dominate==2.9.1
2024-08-20T22:34:22.9501649Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881
2024-08-20T22:34:22.9502419Z einops==0.8.0
2024-08-20T22:34:22.9502721Z exceptiongroup==1.2.2
2024-08-20T22:34:22.9503049Z execnet==2.1.1
2024-08-20T22:34:22.9503340Z expecttest==0.1.6
2024-08-20T22:34:22.9503647Z FastNLP==0.6.0
2024-08-20T22:34:22.9503992Z ffmpeg-python==0.2.0
2024-08-20T22:34:22.9504319Z filelock==3.13.1
2024-08-20T22:34:22.9504626Z Flask==3.0.3
2024-08-20T22:34:22.9504918Z flatbuffers==2.0
2024-08-20T22:34:22.9505280Z fonttools==4.53.1
2024-08-20T22:34:22.9505582Z frozenlist==1.4.1
2024-08-20T22:34:22.9505889Z fsspec==2024.6.1
2024-08-20T22:34:22.9506183Z ftfy==6.2.3
2024-08-20T22:34:22.9506787Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74
2024-08-20T22:34:22.9507490Z future==1.0.0
2024-08-20T22:34:22.9507792Z fvcore==0.1.5.post20221221
2024-08-20T22:34:22.9508138Z gdown==5.2.0
2024-08-20T22:34:22.9508426Z geojson==2.5.0
2024-08-20T22:34:22.9508712Z ghstack==0.8.0
2024-08-20T22:34:22.9509016Z gitdb==4.0.11
2024-08-20T22:34:22.9509314Z GitPython==3.1.43
2024-08-20T22:34:22.9509659Z google-auth==2.34.0
2024-08-20T22:34:22.9510034Z google-auth-oauthlib==1.0.0
2024-08-20T22:34:22.9510396Z grpcio==1.65.5
2024-08-20T22:34:22.9510693Z gym==0.26.2
2024-08-20T22:34:22.9511007Z gym-notices==0.0.8
2024-08-20T22:34:22.9511319Z h5py==3.11.0
2024-08-20T22:34:22.9511612Z higher==0.2.1
2024-08-20T22:34:22.9511951Z huggingface-hub==0.24.6
2024-08-20T22:34:22.9512319Z hydra-core==1.3.2
2024-08-20T22:34:22.9512628Z hypothesis==5.35.1
2024-08-20T22:34:22.9512937Z idna==3.7
2024-08-20T22:34:22.9513205Z imageio==2.35.1
2024-08-20T22:34:22.9513516Z importlib_metadata==8.4.0
2024-08-20T22:34:22.9513866Z inflect==7.3.1
2024-08-20T22:34:22.9514155Z iniconfig==2.0.0
2024-08-20T22:34:22.9514456Z iopath==0.1.9
2024-08-20T22:34:22.9514746Z isodate==0.6.1
2024-08-20T22:34:22.9515052Z isort==5.13.2
2024-08-20T22:34:22.9515372Z itsdangerous==2.2.0
2024-08-20T22:34:22.9515687Z Jinja2==3.1.4
2024-08-20T22:34:22.9515985Z jmespath==0.10.0
2024-08-20T22:34:22.9516284Z joblib==1.4.2
2024-08-20T22:34:22.9516574Z jsonpatch==1.33
2024-08-20T22:34:22.9516882Z jsonpointer==3.0.0
2024-08-20T22:34:22.9517195Z jsonschema==4.23.0
2024-08-20T22:34:22.9517588Z jsonschema-specifications==2023.12.1
2024-08-20T22:34:22.9518001Z junitparser==2.1.1
2024-08-20T22:34:22.9518350Z kaldi-io==0.9.8
2024-08-20T22:34:22.9518743Z kiwisolver==1.4.5
2024-08-20T22:34:22.9519062Z kornia==0.7.3
2024-08-20T22:34:22.9519352Z kornia_rs==0.1.5
2024-08-20T22:34:22.9519653Z lameenc==1.7.0
2024-08-20T22:34:22.9519941Z langcodes==3.4.0
2024-08-20T22:34:22.9520245Z langdetect==1.0.9
2024-08-20T22:34:22.9520603Z language-tags==1.2.0
2024-08-20T22:34:22.9520934Z language_data==1.2.0
2024-08-20T22:34:22.9521244Z lark==0.12.0
2024-08-20T22:34:22.9521536Z lazy_loader==0.4
2024-08-20T22:34:22.9521836Z libcst==1.4.0
2024-08-20T22:34:22.9522124Z librosa==0.9.2
2024-08-20T22:34:22.9522422Z lintrunner==0.12.5
2024-08-20T22:34:22.9522726Z llvmlite==0.38.1
2024-08-20T22:34:22.9523022Z lxml==4.9.4
2024-08-20T22:34:22.9523336Z marisa-trie==1.2.0
2024-08-20T22:34:22.9523641Z Markdown==3.7
2024-08-20T22:34:22.9523969Z markdown-it-py==3.0.0
2024-08-20T22:34:22.9524291Z MarkupSafe==2.1.5
2024-08-20T22:34:22.9524593Z matplotlib==3.8.4
2024-08-20T22:34:22.9524891Z mccabe==0.7.0
2024-08-20T22:34:22.9525175Z mdurl==0.1.2
2024-08-20T22:34:22.9525496Z ml-dtypes==0.4.0
2024-08-20T22:34:22.9525896Z MonkeyType==23.3.0
2024-08-20T22:34:22.9526250Z more-itertools==10.4.0
2024-08-20T22:34:22.9526581Z mpmath==1.3.0
2024-08-20T22:34:22.9526872Z msgpack==1.0.8
2024-08-20T22:34:22.9527168Z multidict==6.0.5
2024-08-20T22:34:22.9527484Z multiprocess==0.70.16
2024-08-20T22:34:22.9527805Z murmurhash==1.0.10
2024-08-20T22:34:22.9528117Z musdb==0.4.2
2024-08-20T22:34:22.9528408Z museval==0.4.1
2024-08-20T22:34:22.9528693Z mypy==1.10.0
2024-08-20T22:34:22.9529030Z mypy-extensions==1.0.0
2024-08-20T22:34:22.9529360Z networkx==2.8.8
2024-08-20T22:34:22.9529661Z ninja==1.10.2
2024-08-20T22:34:22.9529942Z nose==1.3.7
2024-08-20T22:34:22.9530235Z numba==0.55.2
2024-08-20T22:34:22.9530556Z numpy==1.21.2
2024-08-20T22:34:22.9530904Z nvidia-cublas-cu12==12.1.3.1
2024-08-20T22:34:22.9531450Z nvidia-cuda-cupti-cu12==12.1.105
2024-08-20T22:34:22.9531896Z nvidia-cuda-nvrtc-cu12==12.1.105
2024-08-20T22:34:22.9532343Z nvidia-cuda-runtime-cu12==12.1.105
2024-08-20T22:34:22.9532781Z nvidia-cudnn-cu12==9.1.0.70
2024-08-20T22:34:22.9533201Z nvidia-cufft-cu12==11.0.2.54
2024-08-20T22:34:22.9533616Z nvidia-curand-cu12==10.3.2.106
2024-08-20T22:34:22.9534046Z nvidia-cusolver-cu12==11.4.5.107
2024-08-20T22:34:22.9534477Z nvidia-cusparse-cu12==12.1.0.106
2024-08-20T22:34:22.9534898Z nvidia-nccl-cu12==2.20.5
2024-08-20T22:34:22.9535304Z nvidia-nvjitlink-cu12==12.6.20
2024-08-20T22:34:22.9535717Z nvidia-nvtx-cu12==12.1.105
2024-08-20T22:34:22.9536059Z oauthlib==3.2.2
2024-08-20T22:34:22.9536401Z omegaconf==2.3.0
2024-08-20T22:34:22.9536694Z onnx==1.16.1
2024-08-20T22:34:22.9536984Z onnxscript==0.1.0.dev20240817
2024-08-20T22:34:22.9537334Z opacus==1.5.2
2024-08-20T22:34:22.9537670Z opencv-python==4.8.0.74
2024-08-20T22:34:22.9538030Z opt-einsum==3.3.0
2024-08-20T22:34:22.9538329Z optree==0.12.1
2024-08-20T22:34:22.9538624Z packaging==24.1
2024-08-20T22:34:22.9538922Z pandas==2.0.3
2024-08-20T22:34:22.9539203Z patch==1.16
2024-08-20T22:34:22.9539482Z pathspec==0.12.1
2024-08-20T22:34:22.9539785Z phonemizer==3.3.0
2024-08-20T22:34:22.9540093Z pillow==10.3.0
2024-08-20T22:34:22.9540393Z platformdirs==4.2.2
2024-08-20T22:34:22.9540701Z pluggy==1.5.0
2024-08-20T22:34:22.9540985Z pooch==1.8.2
2024-08-20T22:34:22.9541271Z portalocker==2.10.1
2024-08-20T22:34:22.9541581Z preshed==3.0.9
2024-08-20T22:34:22.9541881Z prettytable==3.11.0
2024-08-20T22:34:22.9542188Z protobuf==3.20.2
2024-08-20T22:34:22.9542489Z psutil==6.0.0
2024-08-20T22:34:22.9542818Z py-cpuinfo==9.0.0
2024-08-20T22:34:22.9543117Z pyaml==24.7.0
2024-08-20T22:34:22.9543392Z pyarrow==17.0.0
2024-08-20T22:34:22.9543688Z pyasn1==0.6.0
2024-08-20T22:34:22.9543976Z pyasn1_modules==0.4.0
2024-08-20T22:34:22.9544302Z pyclipper==1.3.0.post5
2024-08-20T22:34:22.9544621Z pycocotools==2.0.8
2024-08-20T22:34:22.9544928Z pycparser==2.22
2024-08-20T22:34:22.9545278Z pycryptodomex==3.20.0
2024-08-20T22:34:22.9545590Z pydantic==2.8.2
2024-08-20T22:34:22.9545888Z pydantic_core==2.20.1
2024-08-20T22:34:22.9546202Z pydot==3.0.1
2024-08-20T22:34:22.9546481Z pygame==2.6.0
2024-08-20T22:34:22.9546870Z PyGithub==2.3.0
2024-08-20T22:34:22.9547173Z Pygments==2.15.0
2024-08-20T22:34:22.9547467Z PyJWT==2.9.0
2024-08-20T22:34:22.9547746Z pylatexenc==2.10
2024-08-20T22:34:22.9548041Z pylint==3.2.6
2024-08-20T22:34:22.9548322Z PyNaCl==1.5.0
2024-08-20T22:34:22.9548602Z pynvml==11.5.3
2024-08-20T22:34:22.9548887Z pyparsing==3.1.2
2024-08-20T22:34:22.9549183Z pypdfium2==4.30.0
2024-08-20T22:34:22.9549478Z pysbd==0.3.4
2024-08-20T22:34:22.9549753Z PySocks==1.7.1
2024-08-20T22:34:22.9550037Z pytest==7.3.2
2024-08-20T22:34:22.9550375Z pytest-benchmark==4.0.0
2024-08-20T22:34:22.9550750Z pytest-cpp==2.3.0
2024-08-20T22:34:22.9551101Z pytest-flakefinder==1.1.0
2024-08-20T22:34:22.9551502Z pytest-rerunfailures==14.0
2024-08-20T22:34:22.9551897Z pytest-xdist==3.3.1
2024-08-20T22:34:22.9552260Z python-dateutil==2.9.0.post0
2024-08-20T22:34:22.9553039Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56
2024-08-20T22:34:22.9554493Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4
2024-08-20T22:34:22.9555588Z pytz==2024.1
2024-08-20T22:34:22.9555868Z PyWavelets==1.4.1
2024-08-20T22:34:22.9556237Z PyYAML @ file:///croot/pyyaml_1698096049011/work
2024-08-20T22:34:22.9556681Z rapidfuzz==3.9.6
2024-08-20T22:34:22.9556973Z rdflib==7.0.0
2024-08-20T22:34:22.9557248Z redis==5.0.8
2024-08-20T22:34:22.9557531Z referencing==0.35.1
2024-08-20T22:34:22.9557835Z regex==2024.7.24
2024-08-20T22:34:22.9558127Z requests==2.32.3
2024-08-20T22:34:22.9558480Z requests-oauthlib==2.0.0
2024-08-20T22:34:22.9558814Z resampy==0.4.3
2024-08-20T22:34:22.9559098Z rfc3986==1.5.0
2024-08-20T22:34:22.9559371Z rich==13.7.1
2024-08-20T22:34:22.9559649Z rockset==1.0.3
2024-08-20T22:34:22.9559968Z rpds-py==0.20.0
2024-08-20T22:34:22.9560253Z rsa==4.9
2024-08-20T22:34:22.9560515Z s3transfer==0.5.2
2024-08-20T22:34:22.9560817Z safetensors==0.4.4
2024-08-20T22:34:22.9561156Z scikit-image==0.21.0
2024-08-20T22:34:22.9561520Z scikit-learn==1.5.1
2024-08-20T22:34:22.9561824Z scipy==1.10.1
2024-08-20T22:34:22.9562105Z segments==2.2.1
2024-08-20T22:34:22.9562400Z sentencepiece==0.2.0
2024-08-20T22:34:22.9562746Z sentry-sdk==2.13.0
2024-08-20T22:34:22.9563059Z setproctitle==1.3.3
2024-08-20T22:34:22.9563366Z shapely==2.0.6
2024-08-20T22:34:22.9563691Z shellingham==1.5.4
2024-08-20T22:34:22.9563992Z simplejson==3.19.3
2024-08-20T22:34:22.9564378Z six @ file:///tmp/build/80754af9/six_1644875935023/work
2024-08-20T22:34:22.9564882Z smart-open==7.0.4
2024-08-20T22:34:22.9565231Z smmap==5.0.1
2024-08-20T22:34:22.9565520Z sortedcontainers==2.4.0
2024-08-20T22:34:22.9565848Z soundfile==0.12.1
2024-08-20T22:34:22.9566150Z soupsieve==2.6
2024-08-20T22:34:22.9566431Z soxr==0.4.0
2024-08-20T22:34:22.9566703Z spacy==3.7.6
2024-08-20T22:34:22.9567861Z spacy-legacy==3.0.12
2024-08-20T22:34:22.9568263Z spacy-loggers==1.0.5
2024-08-20T22:34:22.9568569Z srsly==2.4.8
2024-08-20T22:34:22.9568852Z stempeg==0.2.3
2024-08-20T22:34:22.9569157Z submitit==1.5.1
2024-08-20T22:34:22.9569448Z sympy==1.13.1
2024-08-20T22:34:22.9569734Z tabulate==0.9.0
2024-08-20T22:34:22.9570081Z tb-nightly==2.13.0a20230426
2024-08-20T22:34:22.9570428Z tensorboard==2.13.0
2024-08-20T22:34:22.9570794Z tensorboard-data-server==0.7.2
2024-08-20T22:34:22.9571250Z tensorboardX==2.6.2.2
2024-08-20T22:34:22.9571570Z termcolor==2.4.0
2024-08-20T22:34:22.9571865Z thinc==8.2.5
2024-08-20T22:34:22.9572149Z threadpoolctl==3.5.0
2024-08-20T22:34:22.9572471Z tifffile==2024.8.10
2024-08-20T22:34:22.9573274Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514
2024-08-20T22:34:22.9574036Z tlparse==0.3.25
2024-08-20T22:34:22.9574329Z tokenizers==0.15.2
2024-08-20T22:34:22.9574627Z tomli==2.0.1
2024-08-20T22:34:22.9574957Z tomlkit==0.13.2
2024-08-20T22:34:22.9576135Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b70a70c0c1fc7cc3f43c6342d758f4ffd3a540a88b786d266604d87618be5127
2024-08-20T22:34:22.9577901Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527
2024-08-20T22:34:22.9578955Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e
2024-08-20T22:34:22.9579917Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862
2024-08-20T22:34:22.9580892Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f
2024-08-20T22:34:22.9581981Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21
2024-08-20T22:34:22.9583056Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db
2024-08-20T22:34:22.9584033Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-08-20T22:34:22.9584723Z tornado==6.4.1
2024-08-20T22:34:22.9585009Z tqdm==4.66.5
2024-08-20T22:34:22.9585304Z transformers==4.38.1
2024-08-20T22:34:22.9585760Z treetable==0.2.5
2024-08-20T22:34:22.9586555Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python
2024-08-20T22:34:22.9587346Z typeguard==4.3.0
2024-08-20T22:34:22.9587646Z typer==0.12.4
2024-08-20T22:34:22.9587950Z typing_extensions==4.12.2
2024-08-20T22:34:22.9588286Z tzdata==2024.1
2024-08-20T22:34:22.9588579Z Unidecode==1.3.8
2024-08-20T22:34:22.9588961Z unittest-xml-reporting==3.2.0
2024-08-20T22:34:22.9589342Z uritemplate==4.1.1
2024-08-20T22:34:22.9589651Z urllib3==1.26.19
2024-08-20T22:34:22.9589956Z visdom==0.2.4
2024-08-20T22:34:22.9590251Z wandb==0.17.7
2024-08-20T22:34:22.9590535Z wasabi==1.1.3
2024-08-20T22:34:22.9590829Z wcwidth==0.2.13
2024-08-20T22:34:22.9591132Z weasel==0.4.1
2024-08-20T22:34:22.9591480Z websocket-client==1.8.0
2024-08-20T22:34:22.9591814Z Werkzeug==3.0.3
2024-08-20T22:34:22.9592113Z wrapt==1.16.0
2024-08-20T22:34:22.9592409Z xdoctest==1.1.0
2024-08-20T22:34:22.9592707Z xxhash==3.5.0
2024-08-20T22:34:22.9593013Z xyzservices==2024.6.0
2024-08-20T22:34:22.9593337Z yacs==0.1.8
2024-08-20T22:34:22.9593616Z yarl==1.9.4
2024-08-20T22:34:22.9593938Z z3-solver==4.12.2.0
2024-08-20T22:34:22.9594251Z zipp==3.20.0
2024-08-20T22:34:22.9773916Z + popd
2024-08-20T22:34:22.9774451Z ~/workspace
2024-08-20T22:34:22.9774801Z + [[ dynamic_aot_eager_torchbench != *cpu* ]]
2024-08-20T22:34:22.9775261Z + install_torchrec_and_fbgemm
2024-08-20T22:34:22.9775639Z + local torchrec_commit
2024-08-20T22:34:22.9777679Z ++ get_pinned_commit torchrec
2024-08-20T22:34:22.9778091Z ++ cat .github/ci_commit_pins/torchrec.txt
2024-08-20T22:34:22.9839560Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T22:34:22.9840068Z + local fbgemm_commit
2024-08-20T22:34:22.9844213Z ++ get_pinned_commit fbgemm
2024-08-20T22:34:22.9845196Z ++ cat .github/ci_commit_pins/fbgemm.txt
2024-08-20T22:34:22.9859457Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd
2024-08-20T22:34:22.9860277Z + pip_uninstall torchrec-nightly
2024-08-20T22:34:22.9860787Z + pip uninstall -y torchrec-nightly
2024-08-20T22:34:23.3625747Z WARNING: Skipping torchrec-nightly as it is not installed.
2024-08-20T22:34:23.3977123Z + pip_uninstall fbgemm-gpu-nightly
2024-08-20T22:34:23.3977709Z + pip uninstall -y fbgemm-gpu-nightly
2024-08-20T22:34:23.7668367Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2024-08-20T22:34:23.8015667Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2024-08-20T22:34:23.8016812Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2024-08-20T22:34:24.3136207Z Collecting setuptools-git-versioning
2024-08-20T22:34:24.3324196Z   Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB)
2024-08-20T22:34:24.4474208Z Collecting scikit-build
2024-08-20T22:34:24.4511591Z   Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB)
2024-08-20T22:34:24.5744906Z Collecting pyre-extensions
2024-08-20T22:34:24.5783370Z   Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB)
2024-08-20T22:34:24.6318842Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1)
2024-08-20T22:34:24.6322650Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0)
2024-08-20T22:34:24.6895512Z Collecting toml>=0.10.2 (from setuptools-git-versioning)
2024-08-20T22:34:24.6927305Z   Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB)
2024-08-20T22:34:24.7446800Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0)
2024-08-20T22:34:24.7453441Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1)
2024-08-20T22:34:24.7456240Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0)
2024-08-20T22:34:24.7960101Z Collecting typing-inspect (from pyre-extensions)
2024-08-20T22:34:24.7991631Z   Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)
2024-08-20T22:34:24.8564252Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2)
2024-08-20T22:34:24.8643928Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2024-08-20T22:34:24.8693347Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB)
2024-08-20T22:34:24.9182910Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB)
2024-08-20T22:34:24.9779620Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB)
2024-08-20T22:34:25.0308163Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
2024-08-20T22:34:25.0819875Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)
2024-08-20T22:34:25.9720978Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions
2024-08-20T22:34:26.3415959Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0
2024-08-20T22:34:26.4503636Z + CUDA_PATH=/usr/local/cuda-12.1
2024-08-20T22:34:26.4504960Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-08-20T22:34:26.4506895Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2024-08-20T22:34:26.8265374Z Collecting fbgemm-gpu
2024-08-20T22:34:26.8267717Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-vppem7ra/fbgemm-gpu_796c0c62c15e453695c407864730ddbb
2024-08-20T22:34:26.8299367Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-vppem7ra/fbgemm-gpu_796c0c62c15e453695c407864730ddbb
2024-08-20T22:34:27.9793417Z   Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd'
2024-08-20T22:34:27.9821959Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd
2024-08-20T22:34:28.4894680Z   Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd
2024-08-20T22:34:28.9926127Z   Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd
2024-08-20T22:34:28.9927278Z   Running command git submodule update --init --recursive -q
2024-08-20T22:34:38.6728620Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T22:34:38.6748515Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2)
2024-08-20T22:34:38.6755927Z Building wheels for collected packages: fbgemm-gpu
2024-08-20T23:19:31.3609617Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
2024-08-20T23:19:31.5439631Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252619808 sha256=eb53aab6f36ff1afbf9a731a828ee2cdbf6a8473924864711db4e1c49c84712a
2024-08-20T23:19:31.5441280Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39
2024-08-20T23:19:31.5461015Z Successfully built fbgemm-gpu
2024-08-20T23:19:32.3825930Z Installing collected packages: fbgemm-gpu
2024-08-20T23:19:35.8793779Z Successfully installed fbgemm-gpu-0.4.1rc0.post421
2024-08-20T23:19:36.7920453Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:36.7921992Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:37.1600734Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:37.1605048Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-84nkyuz0
2024-08-20T23:19:37.1631395Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-84nkyuz0
2024-08-20T23:19:37.9336681Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2024-08-20T23:19:37.9365430Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:38.2454161Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:38.5940283Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2024-08-20T23:19:38.8664378Z   Preparing metadata (setup.py) ... [?25l- done
2024-08-20T23:19:38.8703993Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2024-08-20T23:19:38.8707153Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3)
2024-08-20T23:19:38.8710458Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2024-08-20T23:19:38.9196112Z Collecting torchmetrics (from torchrec==0.3.2)
2024-08-20T23:19:38.9368561Z   Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB)
2024-08-20T23:19:38.9748049Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5)
2024-08-20T23:19:38.9750115Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421)
2024-08-20T23:19:38.9760037Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2)
2024-08-20T23:19:38.9769969Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1)
2024-08-20T23:19:39.0032320Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0)
2024-08-20T23:19:39.0036659Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1)
2024-08-20T23:19:39.0039266Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1)
2024-08-20T23:19:39.0062009Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1)
2024-08-20T23:19:39.0065929Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6)
2024-08-20T23:19:39.0233324Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2)
2024-08-20T23:19:39.0273280Z   Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB)
2024-08-20T23:19:39.0360622Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0)
2024-08-20T23:19:39.0364107Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2)
2024-08-20T23:19:39.0384678Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0)
2024-08-20T23:19:39.0417863Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1)
2024-08-20T23:19:39.0421336Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8)
2024-08-20T23:19:39.0424874Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4)
2024-08-20T23:19:39.0427449Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1)
2024-08-20T23:19:39.0432924Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1)
2024-08-20T23:19:39.0448066Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0)
2024-08-20T23:19:39.0855344Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5)
2024-08-20T23:19:39.1021500Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB)
2024-08-20T23:19:39.1199657Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB)
2024-08-20T23:19:39.1293651Z Building wheels for collected packages: torchrec
2024-08-20T23:19:39.6455849Z   Building wheel for torchrec (setup.py) ... [?25l- \ | done
2024-08-20T23:19:39.6462468Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=d0f29f2e91496931edfef0043870853c9f176a8a9bf2b940f3c379bb46fa4b7e
2024-08-20T23:19:39.6463969Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2024-08-20T23:19:39.6486006Z Successfully built torchrec
2024-08-20T23:19:40.4276107Z Installing collected packages: lightning-utilities, torchmetrics, torchrec
2024-08-20T23:19:41.0370187Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2
2024-08-20T23:19:41.1781154Z ++ pwd
2024-08-20T23:19:41.1783936Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2024-08-20T23:19:41.1786063Z + test_dynamo_benchmark torchbench 0
2024-08-20T23:19:41.1789292Z ++ pwd
2024-08-20T23:19:41.1792868Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T23:19:41.1793442Z + local suite=torchbench
2024-08-20T23:19:41.1795164Z + shift
2024-08-20T23:19:41.1795524Z + local shard_id=0
2024-08-20T23:19:41.1796297Z + shift
2024-08-20T23:19:41.1796760Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]]
2024-08-20T23:19:41.1797410Z + [[ dynamic_aot_eager_torchbench == *perf* ]]
2024-08-20T23:19:41.1798010Z + [[ dynamic_aot_eager_torchbench == *cpu* ]]
2024-08-20T23:19:41.1800828Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]]
2024-08-20T23:19:41.1802746Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16
2024-08-20T23:19:41.1803342Z ++ pwd
2024-08-20T23:19:41.1803822Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T23:19:41.1804494Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-08-20T23:19:41.1847477Z + local name=inference
2024-08-20T23:19:41.1847907Z + shift
2024-08-20T23:19:41.1848320Z + local suite=torchbench
2024-08-20T23:19:41.1848696Z + shift
2024-08-20T23:19:41.1848996Z + local shard_id=0
2024-08-20T23:19:41.1849335Z + shift
2024-08-20T23:19:41.1849612Z + partition_flags=()
2024-08-20T23:19:41.1849956Z + local partition_flags
2024-08-20T23:19:41.1850354Z + [[ -n 2 ]]
2024-08-20T23:19:41.1850810Z + [[ -n 0 ]]
2024-08-20T23:19:41.1851410Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-08-20T23:19:41.1852134Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]]
2024-08-20T23:19:41.1852668Z + [[ dynamic_aot_eager_torchbench == *perf* ]]
2024-08-20T23:19:41.1853190Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]]
2024-08-20T23:19:41.1853710Z + [[ dynamic_aot_eager_torchbench == *_avx2* ]]
2024-08-20T23:19:41.1854213Z + [[ dynamic_aot_eager_torchbench == *_avx512* ]]
2024-08-20T23:19:41.1856100Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv
2024-08-20T23:19:47.3265309Z 
2024-08-20T23:19:48.0577834Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
2024-08-20T23:19:48.0579812Z   from torch.distributed.optim import _apply_optimizer_in_backward
2024-08-20T23:19:48.4402720Z 
2024-08-20T23:19:48.4403819Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:19:48.4404803Z cuda eval  torchrec_dlrm                      
2024-08-20T23:19:48.4446578Z Traceback (most recent call last):
2024-08-20T23:19:48.4447425Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:19:48.4448424Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:19:48.4449472Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:19:48.4450240Z     return mod(*inputs)
2024-08-20T23:19:48.4451446Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:19:48.4452335Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:19:48.4453331Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:19:48.4454245Z     return forward_call(*args, **kwargs)
2024-08-20T23:19:48.4455143Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward
2024-08-20T23:19:48.4456067Z     logits = self.model(batch.dense_features, batch.sparse_features)
2024-08-20T23:19:48.4457257Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:19:48.4458160Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:19:48.4459116Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:19:48.4460137Z     return forward_call(*args, **kwargs)
2024-08-20T23:19:48.4461006Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward
2024-08-20T23:19:48.4461797Z     concatenated_dense = self.inter_arch(
2024-08-20T23:19:48.4462782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:19:48.4463669Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:19:48.4464596Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:19:48.4465410Z     return forward_call(*args, **kwargs)
2024-08-20T23:19:48.4466282Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward
2024-08-20T23:19:48.4467619Z     return self.crossnet(combined_values.reshape([B, -1]))
2024-08-20T23:19:48.4468732Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:19:48.4469618Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:19:48.4470559Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:19:48.4471403Z     return forward_call(*args, **kwargs)
2024-08-20T23:19:48.4472304Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward
2024-08-20T23:19:48.4473229Z     x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer])
2024-08-20T23:19:48.4474060Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16
2024-08-20T23:19:48.4474620Z 
2024-08-20T23:19:48.4474909Z The above exception was the direct cause of the following exception:
2024-08-20T23:19:48.4475351Z 
2024-08-20T23:19:48.4475499Z Traceback (most recent call last):
2024-08-20T23:19:48.4476147Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:19:48.4476789Z     ) = runner.load_model(
2024-08-20T23:19:48.4477442Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:19:48.4478189Z     self.validate_model(model, example_inputs)
2024-08-20T23:19:48.4478931Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:19:48.4479682Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:19:48.4480150Z RuntimeError: Eager run failed
2024-08-20T23:19:48.4480408Z 
2024-08-20T23:19:48.4480524Z eager_fail_to_run
2024-08-20T23:19:52.0107189Z 
2024-08-20T23:19:54.2879874Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:19:54.2880349Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:19:54.2938124Z cuda eval  BERT_pytorch                       
2024-08-20T23:20:35.0923682Z pass
2024-08-20T23:20:35.0941817Z TIMING: backend_compile:27.54601 entire_frame_compile:39.95003 total_wall_time:39.95003
2024-08-20T23:20:35.0943403Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:4 | FakeTensorMode.__torch_dispatch__:42887 | attempt fast:2244 | fast is_contiguous:2244 | ProxyTorchDispatchMode.__torch_dispatch__:9196
2024-08-20T23:20:35.0944701Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique)
2024-08-20T23:20:40.5516500Z 
2024-08-20T23:20:43.4204557Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:20:43.4205168Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:20:43.4251760Z cuda eval  Background_Matting                 
2024-08-20T23:20:43.4254504Z pass_due_to_skip
2024-08-20T23:20:43.4781673Z TIMING:
2024-08-20T23:20:43.4782085Z STATS: call_* op count: 0
2024-08-20T23:20:43.4782810Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T23:20:47.0797709Z 
2024-08-20T23:20:50.0351427Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:20:50.0351938Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:20:50.0376451Z cuda eval  LearningToPaint                    
2024-08-20T23:20:53.8295368Z pass
2024-08-20T23:20:53.8296138Z TIMING: backend_compile:1.48197 entire_frame_compile:3.2829 total_wall_time:3.2829
2024-08-20T23:20:53.8297291Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:7195 | ProxyTorchDispatchMode.__torch_dispatch__:1711
2024-08-20T23:20:53.8298246Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique)
2024-08-20T23:20:57.7640004Z 
2024-08-20T23:20:59.2215458Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2024-08-20T23:20:59.2392462Z 
2024-08-20T23:20:59.2392762Z 
2024-08-20T23:20:59.3421573Z   0% 0.00/528M [00:00<?, ?B/s]
2024-08-20T23:20:59.3421876Z 
2024-08-20T23:20:59.4485505Z   2% 8.25M/528M [00:00<00:06, 84.2MB/s]
2024-08-20T23:20:59.4485844Z 
2024-08-20T23:20:59.5487440Z   3% 17.5M/528M [00:00<00:06, 88.4MB/s]
2024-08-20T23:20:59.5487810Z 
2024-08-20T23:20:59.6489833Z   9% 47.1M/528M [00:00<00:02, 187MB/s] 
2024-08-20T23:20:59.6490185Z 
2024-08-20T23:20:59.7491772Z  17% 90.5M/528M [00:00<00:01, 291MB/s]
2024-08-20T23:20:59.7492126Z 
2024-08-20T23:20:59.8493639Z  25% 134M/528M [00:00<00:01, 350MB/s] 
2024-08-20T23:20:59.8493989Z 
2024-08-20T23:20:59.9494592Z  34% 178M/528M [00:00<00:00, 387MB/s]
2024-08-20T23:20:59.9494967Z 
2024-08-20T23:21:00.0496857Z  42% 222M/528M [00:00<00:00, 410MB/s]
2024-08-20T23:21:00.0497217Z 
2024-08-20T23:21:00.1725176Z  50% 263M/528M [00:00<00:00, 417MB/s]
2024-08-20T23:21:00.1725533Z 
2024-08-20T23:21:00.2728202Z  57% 303M/528M [00:00<00:00, 390MB/s]
2024-08-20T23:21:00.2728546Z 
2024-08-20T23:21:00.3729134Z  66% 346M/528M [00:01<00:00, 407MB/s]
2024-08-20T23:21:00.3729464Z 
2024-08-20T23:21:00.4905446Z  73% 387M/528M [00:01<00:00, 414MB/s]
2024-08-20T23:21:00.4905780Z 
2024-08-20T23:21:00.6044757Z  81% 427M/528M [00:01<00:00, 394MB/s]
2024-08-20T23:21:00.6045105Z 
2024-08-20T23:21:00.7073160Z  88% 465M/528M [00:01<00:00, 380MB/s]
2024-08-20T23:21:00.7073503Z 
2024-08-20T23:21:00.7714744Z  95% 502M/528M [00:01<00:00, 378MB/s]
2024-08-20T23:21:00.7715805Z 100% 528M/528M [00:01<00:00, 361MB/s]
2024-08-20T23:21:01.3598406Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-08-20T23:21:01.6916886Z 
2024-08-20T23:21:01.6917594Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:21:01.6919101Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-08-20T23:21:01.6919856Z cuda eval  Super_SloMo                        
2024-08-20T23:21:12.9107562Z WARNING:common:Model Super_SloMo does not support bfloat16, running with amp instead
2024-08-20T23:23:15.9792852Z pass
2024-08-20T23:23:16.0078135Z TIMING: backend_compile:27.81907 entire_frame_compile:122.52683 total_wall_time:122.52683
2024-08-20T23:23:16.0079851Z STATS: call_* op count: 752 | FakeTensorMode.__torch_dispatch__:45292 | attempt fast:2220 | fast is_contiguous:2016 | slow no contiguity match:204 | ProxyTorchDispatchMode.__torch_dispatch__:10066
2024-08-20T23:23:16.0081176Z Dynamo produced 1 graphs covering 752 ops with 0 graph breaks (0 unique)
2024-08-20T23:23:22.0611340Z 
2024-08-20T23:23:22.5984768Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/alexnet-owt-7be5be79.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/alexnet-owt-7be5be79.pth
2024-08-20T23:23:22.6180874Z 
2024-08-20T23:23:22.6181013Z 
2024-08-20T23:23:22.7185406Z   0% 0.00/233M [00:00<?, ?B/s]
2024-08-20T23:23:22.7185703Z 
2024-08-20T23:23:22.8278716Z  18% 42.1M/233M [00:00<00:00, 440MB/s]
2024-08-20T23:23:22.8279030Z 
2024-08-20T23:23:22.9279313Z  36% 84.2M/233M [00:00<00:00, 419MB/s]
2024-08-20T23:23:22.9279709Z 
2024-08-20T23:23:23.0328208Z  54% 126M/233M [00:00<00:00, 427MB/s] 
2024-08-20T23:23:23.0328610Z 
2024-08-20T23:23:23.1446869Z  72% 167M/233M [00:00<00:00, 419MB/s]
2024-08-20T23:23:23.1447613Z 
2024-08-20T23:23:23.2161270Z  89% 207M/233M [00:00<00:00, 403MB/s]
2024-08-20T23:23:23.2161829Z 100% 233M/233M [00:00<00:00, 409MB/s]
2024-08-20T23:23:23.9597046Z 
2024-08-20T23:23:23.9597861Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:23:23.9602216Z cuda eval  alexnet                            
2024-08-20T23:23:25.1280948Z pass
2024-08-20T23:23:25.1281731Z TIMING: backend_compile:0.25362 entire_frame_compile:0.63263 total_wall_time:0.63263
2024-08-20T23:23:25.1282812Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:827 | ProxyTorchDispatchMode.__torch_dispatch__:308
2024-08-20T23:23:25.1283767Z Dynamo produced 1 graphs covering 44 ops with 0 graph breaks (0 unique)
2024-08-20T23:23:28.8312502Z 
2024-08-20T23:23:30.8577235Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:23:30.8578127Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:23:30.8579616Z cuda eval  basic_gnn_edgecnn                  
2024-08-20T23:23:31.9784419Z pass
2024-08-20T23:23:31.9818614Z TIMING: backend_compile:0.27092 entire_frame_compile:0.84563 total_wall_time:0.84563
2024-08-20T23:23:31.9819792Z STATS: call_* op count: 110 | FakeTensorMode.__torch_dispatch__:1565 | ProxyTorchDispatchMode.__torch_dispatch__:618
2024-08-20T23:23:31.9820758Z Dynamo produced 1 graphs covering 110 ops with 0 graph breaks (0 unique)
2024-08-20T23:23:35.7095162Z 
2024-08-20T23:23:37.7014220Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:23:37.7014699Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:23:37.7015487Z cuda eval  basic_gnn_gcn                      
2024-08-20T23:23:39.1151694Z pass
2024-08-20T23:23:39.1271343Z TIMING: entire_frame_compile:0.86093 backend_compile:0.2334 total_wall_time:0.86093
2024-08-20T23:23:39.1272659Z STATS: call_* op count: 57 | FakeTensorMode.__torch_dispatch__:1101 | FakeTensor.__torch_dispatch__:18 | ProxyTorchDispatchMode.__torch_dispatch__:339
2024-08-20T23:23:39.1273764Z Dynamo produced 6 graphs covering 57 ops with 6 graph breaks (1 unique)
2024-08-20T23:23:42.8952921Z 
2024-08-20T23:23:44.8142145Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:23:44.8142637Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:23:44.8145840Z cuda eval  basic_gnn_gin                      
2024-08-20T23:23:46.1177147Z pass
2024-08-20T23:23:46.1271586Z TIMING: backend_compile:0.26361 entire_frame_compile:0.8188 total_wall_time:0.8188
2024-08-20T23:23:46.1272624Z STATS: call_* op count: 104 | FakeTensorMode.__torch_dispatch__:1493 | ProxyTorchDispatchMode.__torch_dispatch__:606
2024-08-20T23:23:46.1273591Z Dynamo produced 1 graphs covering 104 ops with 0 graph breaks (0 unique)
2024-08-20T23:23:49.9212443Z 
2024-08-20T23:23:51.8192522Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:23:51.8193002Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:23:51.8196301Z cuda eval  basic_gnn_sage                     
2024-08-20T23:23:53.0315015Z pass
2024-08-20T23:23:53.0409882Z TIMING: backend_compile:0.26536 entire_frame_compile:0.72236 total_wall_time:0.72236
2024-08-20T23:23:53.0411322Z STATS: call_* op count: 116 | FakeTensorMode.__torch_dispatch__:1511 | ProxyTorchDispatchMode.__torch_dispatch__:609
2024-08-20T23:23:53.0412316Z Dynamo produced 1 graphs covering 116 ops with 0 graph breaks (0 unique)
2024-08-20T23:23:56.7746913Z 
2024-08-20T23:24:00.2996950Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:24:00.2997438Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:24:00.3087842Z cuda eval  cm3leon_generate                   
2024-08-20T23:27:14.1555516Z pass
2024-08-20T23:27:14.2014010Z TIMING: entire_frame_compile:160.67911 backend_compile:98.03648 total_wall_time:160.67911
2024-08-20T23:27:14.2015544Z STATS: call_* op count: 4835 | FakeTensorMode.__torch_dispatch__:142655 | FakeTensor.__torch_dispatch__:40 | ProxyTorchDispatchMode.__torch_dispatch__:40181 | attempt fast:4398 | fast is_contiguous:4388 | slow no contiguity match:10
2024-08-20T23:27:14.2020592Z Dynamo produced 10 graphs covering 4835 ops with 4 graph breaks (1 unique)
2024-08-20T23:27:39.5124973Z 
2024-08-20T23:27:40.0124475Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:27:40.0125364Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:27:40.0130889Z cuda eval  dcgan                              
2024-08-20T23:27:40.9673017Z pass
2024-08-20T23:27:40.9673650Z TIMING: backend_compile:0.31166 entire_frame_compile:0.7012 total_wall_time:0.7012
2024-08-20T23:27:40.9674801Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1105 | FakeTensor.__torch_dispatch__:8 | ProxyTorchDispatchMode.__torch_dispatch__:267
2024-08-20T23:27:40.9675891Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique)
2024-08-20T23:27:44.6157697Z 
2024-08-20T23:27:47.4518956Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:27:47.4519405Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:27:47.4537231Z cuda eval  demucs                             
2024-08-20T23:28:27.8511429Z pass
2024-08-20T23:28:27.8642405Z TIMING: entire_frame_compile:2.33058 backend_compile:0.46357 total_wall_time:2.33058
2024-08-20T23:28:27.8643619Z STATS: call_* op count: 62 | FakeTensorMode.__torch_dispatch__:1577 | ProxyTorchDispatchMode.__torch_dispatch__:582 | FakeTensor.__torch_dispatch__:2
2024-08-20T23:28:27.8644730Z Dynamo produced 3 graphs covering 62 ops with 3 graph breaks (1 unique)
2024-08-20T23:28:31.5111531Z 
2024-08-20T23:28:31.7789116Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/densenet121-a639ec97.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/densenet121-a639ec97.pth
2024-08-20T23:28:31.7973964Z 
2024-08-20T23:28:31.7974155Z 
2024-08-20T23:28:31.8723131Z   0% 0.00/30.8M [00:00<?, ?B/s]
2024-08-20T23:28:31.8723560Z 100% 30.8M/30.8M [00:00<00:00, 433MB/s]
2024-08-20T23:28:32.8254520Z 
2024-08-20T23:28:32.8254789Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:28:32.8399260Z cuda eval  densenet121                        
2024-08-20T23:28:56.3482848Z pass
2024-08-20T23:28:56.3493979Z TIMING: backend_compile:9.69577 entire_frame_compile:21.13141 total_wall_time:21.13141
2024-08-20T23:28:56.3495095Z STATS: call_* op count: 862 | FakeTensorMode.__torch_dispatch__:40645 | ProxyTorchDispatchMode.__torch_dispatch__:9536
2024-08-20T23:28:56.3497072Z Dynamo produced 1 graphs covering 862 ops with 0 graph breaks (0 unique)
2024-08-20T23:29:00.4897572Z 
2024-08-20T23:29:04.8121881Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:04.8122435Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:29:04.8122894Z cuda eval  detectron2_fasterrcnn_r_101_c4     
2024-08-20T23:29:04.8166772Z Traceback (most recent call last):
2024-08-20T23:29:04.8167919Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:04.8168764Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:04.8169537Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:04.8170307Z     return mod(*inputs)
2024-08-20T23:29:04.8174782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:04.8176165Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:04.8177412Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:04.8178562Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:04.8179866Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:04.8180755Z     return self.inference(batched_inputs)
2024-08-20T23:29:04.8181756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:04.8182743Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:04.8183838Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:04.8184729Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:04.8185816Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:04.8187179Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:04.8188295Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-08-20T23:29:04.8189204Z     box_features = self._shared_roi_transform(
2024-08-20T23:29:04.8190307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-08-20T23:29:04.8191261Z     x = self.pooler(features, boxes)
2024-08-20T23:29:04.8192223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:04.8193098Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:04.8194029Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:04.8194866Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:04.8195795Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:29:04.8196677Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:29:04.8197712Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:04.8198597Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:04.8199532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:04.8200351Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:04.8201255Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:04.8202044Z     return roi_align(
2024-08-20T23:29:04.8202871Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:04.8203973Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:04.8205173Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:04.8206248Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:04.8207479Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:04.8208353Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:04.8209255Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:04.8210057Z     return input[
2024-08-20T23:29:04.8213420Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26886.45 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.44 GiB is free. Process 175727 has 3.53 GiB memory in use. Of the allocated memory 2.99 GiB is allocated by PyTorch, and 242.98 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:04.8216102Z 
2024-08-20T23:29:04.8216397Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:04.8216845Z 
2024-08-20T23:29:04.8216989Z Traceback (most recent call last):
2024-08-20T23:29:04.8217647Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:04.8218306Z     ) = runner.load_model(
2024-08-20T23:29:04.8218962Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:04.8219718Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:04.8220480Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:04.8221333Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:04.8221796Z RuntimeError: Eager run failed
2024-08-20T23:29:04.8222056Z 
2024-08-20T23:29:04.8222173Z eager_fail_to_run
2024-08-20T23:29:08.5088129Z 
2024-08-20T23:29:15.5120270Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:15.5120896Z loading model: 0it [00:07, ?it/s]
2024-08-20T23:29:15.5121478Z cuda eval  detectron2_fasterrcnn_r_101_dc5    
2024-08-20T23:29:15.5135444Z Traceback (most recent call last):
2024-08-20T23:29:15.5136452Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:15.5137488Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:15.5138501Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:15.5139229Z     return mod(*inputs)
2024-08-20T23:29:15.5140373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:15.5141265Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:15.5142207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:15.5143035Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:15.5144045Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:15.5144910Z     return self.inference(batched_inputs)
2024-08-20T23:29:15.5145886Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:15.5146869Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:15.5147968Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:15.5148859Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:15.5149805Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:15.5150645Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:15.5151643Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:29:15.5152608Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:29:15.5153705Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:29:15.5154807Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:29:15.5155986Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:15.5156872Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:15.5158068Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:15.5158904Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:15.5159837Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:29:15.5160730Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:29:15.5161779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:15.5162663Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:15.5163652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:15.5164745Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:15.5165738Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:15.5166784Z     return roi_align(
2024-08-20T23:29:15.5168019Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:15.5169143Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:15.5170357Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:15.5171567Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:15.5172813Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:15.5173752Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:15.5174677Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:15.5175490Z     return input[
2024-08-20T23:29:15.5178599Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 13324.51 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.63 GiB is free. Process 175856 has 4.34 GiB memory in use. Of the allocated memory 3.82 GiB is allocated by PyTorch, and 195.43 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:15.5181344Z 
2024-08-20T23:29:15.5181642Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:15.5182092Z 
2024-08-20T23:29:15.5182251Z Traceback (most recent call last):
2024-08-20T23:29:15.5182898Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:15.5183556Z     ) = runner.load_model(
2024-08-20T23:29:15.5184223Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:15.5184993Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:15.5185750Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:15.5186514Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:15.5186989Z RuntimeError: Eager run failed
2024-08-20T23:29:15.5187245Z 
2024-08-20T23:29:15.5187369Z eager_fail_to_run
2024-08-20T23:29:19.1924944Z 
2024-08-20T23:29:22.4166005Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:22.4166472Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:29:22.4166929Z cuda eval  detectron2_fasterrcnn_r_101_fpn    
2024-08-20T23:29:22.4181196Z Traceback (most recent call last):
2024-08-20T23:29:22.4181934Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:22.4182885Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:22.4184895Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:22.4185883Z     return mod(*inputs)
2024-08-20T23:29:22.4187062Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:22.4187950Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:22.4188887Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:22.4189711Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:22.4190685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:22.4191555Z     return self.inference(batched_inputs)
2024-08-20T23:29:22.4192535Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:22.4193509Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:22.4194989Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:22.4195913Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:22.4196857Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:22.4197698Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:22.4198689Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:29:22.4199653Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:29:22.4200752Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:29:22.4201846Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:29:22.4203034Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:22.4203912Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:22.4204852Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:22.4205729Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:22.4206644Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-08-20T23:29:22.4207596Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-08-20T23:29:22.4208710Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:22.4209589Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:22.4210513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:22.4211468Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:22.4212381Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:22.4213377Z     return roi_align(
2024-08-20T23:29:22.4214369Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:22.4215488Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:22.4216692Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:22.4217774Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:22.4219008Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:22.4219887Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:22.4220959Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:22.4221776Z     return input[
2024-08-20T23:29:22.4224862Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 19073.68 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.77 GiB is free. Process 175975 has 4.20 GiB memory in use. Of the allocated memory 3.79 GiB is allocated by PyTorch, and 112.62 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:22.4227519Z 
2024-08-20T23:29:22.4227813Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:22.4228250Z 
2024-08-20T23:29:22.4228401Z Traceback (most recent call last):
2024-08-20T23:29:22.4229051Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:22.4229781Z     ) = runner.load_model(
2024-08-20T23:29:22.4230431Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:22.4231181Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:22.4231926Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:22.4232677Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:22.4233142Z RuntimeError: Eager run failed
2024-08-20T23:29:22.4233398Z 
2024-08-20T23:29:22.4233514Z eager_fail_to_run
2024-08-20T23:29:26.1124975Z 
2024-08-20T23:29:29.1159368Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:29.1159942Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:29:29.1160411Z cuda eval  detectron2_fasterrcnn_r_50_c4      
2024-08-20T23:29:29.1173499Z Traceback (most recent call last):
2024-08-20T23:29:29.1174493Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:29.1175514Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:29.1176501Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:29.1177239Z     return mod(*inputs)
2024-08-20T23:29:29.1178341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:29.1179233Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:29.1180171Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:29.1181030Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:29.1181989Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:29.1182843Z     return self.inference(batched_inputs)
2024-08-20T23:29:29.1183830Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:29.1184821Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:29.1185918Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:29.1186806Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:29.1187744Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:29.1188579Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:29.1189571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-08-20T23:29:29.1190483Z     box_features = self._shared_roi_transform(
2024-08-20T23:29:29.1191581Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-08-20T23:29:29.1192835Z     x = self.pooler(features, boxes)
2024-08-20T23:29:29.1193821Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:29.1194703Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:29.1195633Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:29.1196465Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:29.1197390Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:29:29.1198273Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:29:29.1199309Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:29.1200195Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:29.1201135Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:29.1202167Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:29.1203086Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:29.1203882Z     return roi_align(
2024-08-20T23:29:29.1204707Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:29.1205822Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:29.1207022Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:29.1208097Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:29.1209339Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:29.1210217Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:29.1211222Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:29.1212018Z     return input[
2024-08-20T23:29:29.1215091Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 27475.92 GiB. GPU 0 has a total capacity of 21.98 GiB of which 19.58 GiB is free. Process 176093 has 2.38 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 233.53 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:29.1217787Z 
2024-08-20T23:29:29.1218088Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:29.1218541Z 
2024-08-20T23:29:29.1218691Z Traceback (most recent call last):
2024-08-20T23:29:29.1219352Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:29.1220007Z     ) = runner.load_model(
2024-08-20T23:29:29.1220664Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:29.1221427Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:29.1222185Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:29.1222951Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:29.1223423Z RuntimeError: Eager run failed
2024-08-20T23:29:29.1223684Z 
2024-08-20T23:29:29.1223801Z eager_fail_to_run
2024-08-20T23:29:32.7918217Z 
2024-08-20T23:29:37.3945004Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:37.3945621Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:29:37.3946172Z cuda eval  detectron2_fasterrcnn_r_50_dc5     
2024-08-20T23:29:37.3959541Z Traceback (most recent call last):
2024-08-20T23:29:37.3960572Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:37.3961610Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:37.3962638Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:37.3963412Z     return mod(*inputs)
2024-08-20T23:29:37.3964560Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:37.3965711Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:37.3966932Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:37.3968085Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:37.3969066Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:37.3969935Z     return self.inference(batched_inputs)
2024-08-20T23:29:37.3971187Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:37.3972155Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:37.3973227Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:37.3974099Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:37.3975014Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:37.3975832Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:37.3976808Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:29:37.3977760Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:29:37.3978873Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:29:37.3979956Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:29:37.3981108Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:37.3981978Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:37.3982899Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:37.3983714Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:37.3984610Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:29:37.3985491Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:29:37.3986524Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:37.3987401Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:37.3988314Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:37.3989153Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:37.3990067Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:37.3990840Z     return roi_align(
2024-08-20T23:29:37.3991651Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:37.3992739Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:37.3993921Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:37.3995111Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:37.3996348Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:37.3997200Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:37.3998095Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:37.3998888Z     return input[
2024-08-20T23:29:37.4001998Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 13541.47 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.76 GiB is free. Process 176214 has 3.20 GiB memory in use. Of the allocated memory 2.68 GiB is allocated by PyTorch, and 195.87 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:37.4004785Z 
2024-08-20T23:29:37.4005082Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:37.4005521Z 
2024-08-20T23:29:37.4005671Z Traceback (most recent call last):
2024-08-20T23:29:37.4006309Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:37.4006948Z     ) = runner.load_model(
2024-08-20T23:29:37.4007599Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:37.4008349Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:37.4009089Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:37.4009834Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:37.4010372Z RuntimeError: Eager run failed
2024-08-20T23:29:37.4010622Z 
2024-08-20T23:29:37.4010743Z eager_fail_to_run
2024-08-20T23:29:41.0740238Z 
2024-08-20T23:29:43.9762055Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:29:43.9762574Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:29:43.9763057Z cuda eval  detectron2_fasterrcnn_r_50_fpn     
2024-08-20T23:29:43.9775675Z Traceback (most recent call last):
2024-08-20T23:29:43.9776691Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:29:43.9777676Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:29:43.9778676Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:29:43.9779625Z     return mod(*inputs)
2024-08-20T23:29:43.9781011Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:43.9781910Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:43.9782858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:43.9783689Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:43.9784653Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:29:43.9785536Z     return self.inference(batched_inputs)
2024-08-20T23:29:43.9786513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:29:43.9787491Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:29:43.9788580Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:43.9789453Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:43.9790383Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:43.9791226Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:43.9792542Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:29:43.9793509Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:29:43.9794590Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:29:43.9795668Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:29:43.9796824Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:43.9797697Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:43.9798618Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:43.9799435Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:43.9800342Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-08-20T23:29:43.9801302Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-08-20T23:29:43.9802576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:29:43.9803460Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:29:43.9804389Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:29:43.9805206Z     return forward_call(*args, **kwargs)
2024-08-20T23:29:43.9806101Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:29:43.9806875Z     return roi_align(
2024-08-20T23:29:43.9807692Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:29:43.9808791Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:29:43.9809991Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:29:43.9811142Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:29:43.9812371Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:29:43.9813235Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:29:43.9814134Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:29:43.9814922Z     return input[
2024-08-20T23:29:43.9817994Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21403.23 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.93 GiB is free. Process 176336 has 3.03 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 93.33 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:29:43.9820643Z 
2024-08-20T23:29:43.9820935Z The above exception was the direct cause of the following exception:
2024-08-20T23:29:43.9821382Z 
2024-08-20T23:29:43.9821526Z Traceback (most recent call last):
2024-08-20T23:29:43.9822173Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:29:43.9822825Z     ) = runner.load_model(
2024-08-20T23:29:43.9823476Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:29:43.9824230Z     self.validate_model(model, example_inputs)
2024-08-20T23:29:43.9824984Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:29:43.9825748Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:29:43.9826209Z RuntimeError: Eager run failed
2024-08-20T23:29:43.9826591Z 
2024-08-20T23:29:43.9826710Z eager_fail_to_run
2024-08-20T23:29:47.6559531Z 
2024-08-20T23:29:49.9914019Z loading model: 0it [00:00, ?it/s]WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-08-20T23:29:50.6023066Z 
2024-08-20T23:29:50.6023662Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:29:50.6024386Z WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-08-20T23:29:50.6025146Z cuda eval  detectron2_fcos_r_50_fpn           
2024-08-20T23:30:03.0482857Z WARNING:common:Model detectron2_fcos_r_50_fpn does not support bfloat16, running with amp instead
2024-08-20T23:32:29.7960174Z pass
2024-08-20T23:32:29.8128003Z TIMING: entire_frame_compile:145.60855 backend_compile:21.15216 total_wall_time:145.60855
2024-08-20T23:32:29.8132668Z STATS: call_* op count: 886 | FakeTensorMode.__torch_dispatch__:29259 | FakeTensor.__torch_dispatch__:144 | ProxyTorchDispatchMode.__torch_dispatch__:7392 | attempt fast:848 | fast is_contiguous:843 | slow no contiguity match:5
2024-08-20T23:32:29.8135260Z Dynamo produced 38 graphs covering 886 ops with 21 graph breaks (4 unique)
2024-08-20T23:32:37.2738304Z 
2024-08-20T23:32:40.6953919Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:32:40.6954511Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:32:40.6954980Z cuda eval  detectron2_maskrcnn_r_101_c4       
2024-08-20T23:32:40.6969615Z Traceback (most recent call last):
2024-08-20T23:32:40.6970722Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:32:40.6971536Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:32:40.6972542Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:32:40.6973643Z     return mod(*inputs)
2024-08-20T23:32:40.6974975Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:40.6975904Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:40.6976871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:40.6977918Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:40.6978938Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:32:40.6979801Z     return self.inference(batched_inputs)
2024-08-20T23:32:40.6980869Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:32:40.6982156Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:32:40.6983568Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:40.6984699Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:40.6985935Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:40.6987042Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:40.6988340Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-08-20T23:32:40.6989503Z     box_features = self._shared_roi_transform(
2024-08-20T23:32:40.6991025Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-08-20T23:32:40.6992281Z     x = self.pooler(features, boxes)
2024-08-20T23:32:40.6993545Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:40.6994526Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:40.6995583Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:40.6997228Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:40.6998322Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:32:40.6999289Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:32:40.7000409Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:40.7001366Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:40.7002382Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:40.7003287Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:40.7004308Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:32:40.7005165Z     return roi_align(
2024-08-20T23:32:40.7006067Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:32:40.7007474Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:32:40.7008834Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:32:40.7009983Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:32:40.7011488Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:32:40.7012427Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:32:40.7013414Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:32:40.7014232Z     return input[
2024-08-20T23:32:40.7017488Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26943.76 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.46 GiB is free. Process 176750 has 3.50 GiB memory in use. Of the allocated memory 3.00 GiB is allocated by PyTorch, and 212.63 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:32:40.7020237Z 
2024-08-20T23:32:40.7020567Z The above exception was the direct cause of the following exception:
2024-08-20T23:32:40.7021046Z 
2024-08-20T23:32:40.7021196Z Traceback (most recent call last):
2024-08-20T23:32:40.7021848Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:32:40.7022508Z     ) = runner.load_model(
2024-08-20T23:32:40.7023162Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:32:40.7023930Z     self.validate_model(model, example_inputs)
2024-08-20T23:32:40.7024696Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:32:40.7025468Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:32:40.7025933Z RuntimeError: Eager run failed
2024-08-20T23:32:40.7026201Z 
2024-08-20T23:32:40.7026319Z eager_fail_to_run
2024-08-20T23:32:44.3729947Z 
2024-08-20T23:32:47.5826008Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:32:47.5826490Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:32:47.5826941Z cuda eval  detectron2_maskrcnn_r_101_fpn      
2024-08-20T23:32:47.5840052Z Traceback (most recent call last):
2024-08-20T23:32:47.5841311Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:32:47.5843685Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:32:47.5844684Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:32:47.5845534Z     return mod(*inputs)
2024-08-20T23:32:47.5846912Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:47.5847796Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:47.5848759Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:47.5849585Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:47.5850667Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:32:47.5851523Z     return self.inference(batched_inputs)
2024-08-20T23:32:47.5852491Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:32:47.5853467Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:32:47.5854551Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:47.5855440Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:47.5856543Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:47.5857367Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:47.5858353Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:32:47.5859312Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:32:47.5860398Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:32:47.5861537Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:32:47.5862696Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:47.5863572Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:47.5864508Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:47.5865330Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:47.5866243Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-08-20T23:32:47.5867466Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-08-20T23:32:47.5868590Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:47.5869467Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:47.5870398Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:47.5871223Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:47.5872124Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:32:47.5872912Z     return roi_align(
2024-08-20T23:32:47.5873737Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:32:47.5874840Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:32:47.5876046Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:32:47.5877114Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:32:47.5878342Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:32:47.5879216Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:32:47.5880122Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:32:47.5880927Z     return input[
2024-08-20T23:32:47.5884182Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 19183.01 GiB. GPU 0 has a total capacity of 21.98 GiB of which 17.75 GiB is free. Process 176885 has 4.22 GiB memory in use. Of the allocated memory 3.80 GiB is allocated by PyTorch, and 122.09 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:32:47.5886909Z 
2024-08-20T23:32:47.5887211Z The above exception was the direct cause of the following exception:
2024-08-20T23:32:47.5887656Z 
2024-08-20T23:32:47.5887805Z Traceback (most recent call last):
2024-08-20T23:32:47.5888440Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:32:47.5889089Z     ) = runner.load_model(
2024-08-20T23:32:47.5889747Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:32:47.5890691Z     self.validate_model(model, example_inputs)
2024-08-20T23:32:47.5891433Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:32:47.5892191Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:32:47.5892654Z RuntimeError: Eager run failed
2024-08-20T23:32:47.5892904Z 
2024-08-20T23:32:47.5893025Z eager_fail_to_run
2024-08-20T23:32:51.3089870Z 
2024-08-20T23:32:54.3669427Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:32:54.3670063Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:32:54.3670681Z cuda eval  detectron2_maskrcnn_r_50_c4        
2024-08-20T23:32:54.3685249Z Traceback (most recent call last):
2024-08-20T23:32:54.3686279Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:32:54.3687348Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:32:54.3688456Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:32:54.3689490Z     return mod(*inputs)
2024-08-20T23:32:54.3691173Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:54.3692466Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:54.3693834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:54.3695058Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:54.3696504Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:32:54.3697742Z     return self.inference(batched_inputs)
2024-08-20T23:32:54.3699190Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:32:54.3700637Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:32:54.3702248Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:54.3703573Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:54.3704885Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:54.3705996Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:54.3707408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-08-20T23:32:54.3708598Z     box_features = self._shared_roi_transform(
2024-08-20T23:32:54.3710195Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-08-20T23:32:54.3711559Z     x = self.pooler(features, boxes)
2024-08-20T23:32:54.3713282Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:54.3714536Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:54.3715853Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:54.3716985Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:54.3718272Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:32:54.3719466Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:32:54.3720940Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:32:54.3722182Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:32:54.3723563Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:32:54.3724723Z     return forward_call(*args, **kwargs)
2024-08-20T23:32:54.3726023Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:32:54.3727473Z     return roi_align(
2024-08-20T23:32:54.3728657Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:32:54.3730217Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:32:54.3732056Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:32:54.3733578Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:32:54.3735352Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:32:54.3736578Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:32:54.3737978Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:32:54.3739177Z     return input[
2024-08-20T23:32:54.3743775Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 27336.74 GiB. GPU 0 has a total capacity of 21.98 GiB of which 19.59 GiB is free. Process 177003 has 2.38 GiB memory in use. Of the allocated memory 1.86 GiB is allocated by PyTorch, and 222.09 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:32:54.3747705Z 
2024-08-20T23:32:54.3748131Z The above exception was the direct cause of the following exception:
2024-08-20T23:32:54.3748778Z 
2024-08-20T23:32:54.3749012Z Traceback (most recent call last):
2024-08-20T23:32:54.3749982Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:32:54.3750917Z     ) = runner.load_model(
2024-08-20T23:32:54.3751832Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:32:54.3752938Z     self.validate_model(model, example_inputs)
2024-08-20T23:32:54.3753982Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:32:54.3755052Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:32:54.3755715Z RuntimeError: Eager run failed
2024-08-20T23:32:54.3756069Z 
2024-08-20T23:32:54.3756242Z eager_fail_to_run
2024-08-20T23:32:58.0830837Z 
2024-08-20T23:33:01.0552674Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:33:01.0553261Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:33:01.0553828Z cuda eval  detectron2_maskrcnn_r_50_fpn       
2024-08-20T23:33:01.0566480Z Traceback (most recent call last):
2024-08-20T23:33:01.0567786Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:33:01.0569091Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:33:01.0570106Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:33:01.0571177Z     return mod(*inputs)
2024-08-20T23:33:01.0572447Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:33:01.0573344Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:33:01.0574288Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:33:01.0575115Z     return forward_call(*args, **kwargs)
2024-08-20T23:33:01.0576084Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:33:01.0576956Z     return self.inference(batched_inputs)
2024-08-20T23:33:01.0577949Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:33:01.0579170Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:33:01.0580264Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:33:01.0581156Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:33:01.0582100Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:33:01.0582933Z     return forward_call(*args, **kwargs)
2024-08-20T23:33:01.0583922Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 747, in forward
2024-08-20T23:33:01.0584934Z     pred_instances = self._forward_box(features, proposals)
2024-08-20T23:33:01.0586036Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 798, in _forward_box
2024-08-20T23:33:01.0587140Z     box_features = self.box_pooler(features, [x.proposal_boxes for x in proposals])
2024-08-20T23:33:01.0588313Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:33:01.0589209Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:33:01.0590151Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:33:01.0590986Z     return forward_call(*args, **kwargs)
2024-08-20T23:33:01.0591907Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 261, in forward
2024-08-20T23:33:01.0592871Z     output.index_put_((inds,), pooler(x[level], pooler_fmt_boxes_level))
2024-08-20T23:33:01.0593994Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:33:01.0594883Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:33:01.0595826Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:33:01.0596656Z     return forward_call(*args, **kwargs)
2024-08-20T23:33:01.0597571Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:33:01.0598366Z     return roi_align(
2024-08-20T23:33:01.0599199Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:33:01.0600305Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:33:01.0601513Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:33:01.0602595Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:33:01.0603930Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:33:01.0604813Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:33:01.0605717Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:33:01.0606520Z     return input[
2024-08-20T23:33:01.0609613Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20915.45 GiB. GPU 0 has a total capacity of 21.98 GiB of which 18.88 GiB is free. Process 177123 has 3.09 GiB memory in use. Of the allocated memory 2.65 GiB is allocated by PyTorch, and 145.05 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:33:01.0612412Z 
2024-08-20T23:33:01.0612713Z The above exception was the direct cause of the following exception:
2024-08-20T23:33:01.0613156Z 
2024-08-20T23:33:01.0613314Z Traceback (most recent call last):
2024-08-20T23:33:01.0614050Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:33:01.0614759Z     ) = runner.load_model(
2024-08-20T23:33:01.0615418Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:33:01.0616179Z     self.validate_model(model, example_inputs)
2024-08-20T23:33:01.0616939Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:33:01.0617704Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:33:01.0618179Z RuntimeError: Eager run failed
2024-08-20T23:33:01.0618439Z 
2024-08-20T23:33:01.0618564Z eager_fail_to_run
2024-08-20T23:33:04.7603755Z 
2024-08-20T23:33:13.6526899Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:33:13.6527384Z loading model: 0it [00:08, ?it/s]
2024-08-20T23:33:13.6531684Z cuda eval  dlrm                               
2024-08-20T23:33:14.9331036Z pass
2024-08-20T23:33:14.9448685Z TIMING: backend_compile:0.39601 entire_frame_compile:0.93784 total_wall_time:0.93784
2024-08-20T23:33:14.9449980Z STATS: call_* op count: 72 | FakeTensorMode.__torch_dispatch__:2064 | ProxyTorchDispatchMode.__torch_dispatch__:728
2024-08-20T23:33:14.9451074Z Dynamo produced 1 graphs covering 72 ops with 0 graph breaks (0 unique)
2024-08-20T23:33:18.5947516Z 
2024-08-20T23:33:20.1699940Z loading model: 0it [00:00, ?it/s]Downloading https://doctr-static.mindee.com/models?id=v0.7.0/db_resnet50-79bd7d70.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/db_resnet50-79bd7d70.pt
2024-08-20T23:33:20.4981443Z 
2024-08-20T23:33:20.4981763Z 
2024-08-20T23:33:20.5984121Z   0% 0/102021912 [00:00<?, ?it/s]
2024-08-20T23:33:20.5984623Z 
2024-08-20T23:33:20.6984344Z  27% 27682816/102021912 [00:00<00:00, 276826443.99it/s]
2024-08-20T23:33:20.6984935Z 
2024-08-20T23:33:20.7985079Z  57% 57773056/102021912 [00:00<00:00, 290965638.30it/s]
2024-08-20T23:33:20.7985664Z 
2024-08-20T23:33:20.8426356Z  87% 88619008/102021912 [00:00<00:00, 298948930.51it/s]
2024-08-20T23:33:20.8427132Z 102022144it [00:00, 296494948.27it/s]                  
2024-08-20T23:33:22.5358398Z Downloading https://doctr-static.mindee.com/models?id=v0.3.1/crnn_vgg16_bn-9762b0b0.pt&src=0 to /var/lib/jenkins/.cache/doctr/models/crnn_vgg16_bn-9762b0b0.pt
2024-08-20T23:33:22.8471218Z 
2024-08-20T23:33:22.8471225Z 
2024-08-20T23:33:22.9474005Z   0% 0/63286381 [00:00<?, ?it/s]
2024-08-20T23:33:22.9474340Z 
2024-08-20T23:33:23.0474093Z  45% 28580864/63286381 [00:00<00:00, 285808231.15it/s]
2024-08-20T23:33:23.0474536Z 
2024-08-20T23:33:23.0597057Z  94% 59724800/63286381 [00:00<00:00, 300883837.77it/s]
2024-08-20T23:33:23.0597621Z 63287296it [00:00, 298205191.91it/s]                  
2024-08-20T23:33:23.6416986Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-08-20T23:33:24.0961356Z 
2024-08-20T23:33:24.0962020Z loading model: 0it [00:05, ?it/s]
2024-08-20T23:33:24.0963256Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-08-20T23:33:24.0964027Z cuda eval  doctr_det_predictor                
2024-08-20T23:33:32.2768091Z WARNING:common:Model doctr_det_predictor does not support bfloat16, running with amp instead
2024-08-20T23:33:52.0819265Z pass
2024-08-20T23:33:52.0988201Z TIMING: entire_frame_compile:18.56785 backend_compile:4.33203 total_wall_time:18.56785
2024-08-20T23:33:52.0989434Z STATS: call_* op count: 234 | FakeTensorMode.__torch_dispatch__:25769 | ProxyTorchDispatchMode.__torch_dispatch__:5353
2024-08-20T23:33:52.0990407Z Dynamo produced 6 graphs covering 234 ops with 5 graph breaks (2 unique)
2024-08-20T23:33:56.1035190Z 
2024-08-20T23:33:59.3072493Z loading model: 0it [00:00, ?it/s]WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-08-20T23:33:59.5875676Z 
2024-08-20T23:33:59.5876277Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:33:59.5879972Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-08-20T23:33:59.5881230Z cuda eval  doctr_reco_predictor               
2024-08-20T23:33:59.7509203Z WARNING:common:Model doctr_reco_predictor does not support bfloat16, running with amp instead
2024-08-20T23:34:05.9203989Z W0820 23:34:05.919000 34899 torch/fx/experimental/symbolic_shapes.py:5131] [3/0] failed during evaluate_expr(Ne(u0, 123), hint=None, expect_rational=True, size_oblivious=False, forcing_spec=False
2024-08-20T23:34:05.9207070Z E0820 23:34:05.919000 34899 torch/fx/experimental/recording.py:298] [3/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne})
2024-08-20T23:34:05.9862098Z W0820 23:34:05.985000 34899 torch/fx/experimental/symbolic_shapes.py:5131] [4/0] failed during evaluate_expr(Ne(u0, 123), hint=None, expect_rational=True, size_oblivious=False, forcing_spec=False
2024-08-20T23:34:05.9865314Z E0820 23:34:05.985000 34899 torch/fx/experimental/recording.py:298] [4/0] failed while running evaluate_expr(*(Ne(u0, 123), None), **{'fx_node': ne})
2024-08-20T23:34:06.0652315Z pass
2024-08-20T23:34:06.0660491Z TIMING: entire_frame_compile:6.10628 backend_compile:2.18876 total_wall_time:6.10628
2024-08-20T23:34:06.0662515Z STATS: call_* op count: 58 | FakeTensorMode.__torch_dispatch__:6312 | attempt fast:312 | fast is_contiguous:312 | ProxyTorchDispatchMode.__torch_dispatch__:1059 | FakeTensor.__torch_dispatch__:66
2024-08-20T23:34:06.0664452Z Dynamo produced 3 graphs covering 58 ops with 4 graph breaks (4 unique)
2024-08-20T23:34:10.0536415Z 
2024-08-20T23:34:11.4360656Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:11.4361555Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:34:11.4365806Z cuda eval  drq                                
2024-08-20T23:34:12.3597674Z pass
2024-08-20T23:34:12.3598415Z TIMING: backend_compile:0.27334 entire_frame_compile:0.73825 total_wall_time:0.73825
2024-08-20T23:34:12.3599503Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:1035 | ProxyTorchDispatchMode.__torch_dispatch__:372
2024-08-20T23:34:12.3600460Z Dynamo produced 1 graphs covering 56 ops with 0 graph breaks (0 unique)
2024-08-20T23:34:16.0830335Z 
2024-08-20T23:34:17.8765032Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:17.8765577Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:34:17.8817799Z cuda eval  fastNLP_Bert                       
2024-08-20T23:34:20.0592121Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T23:34:20.0594459Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T23:34:20.0596039Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] or:
2024-08-20T23:34:20.0597017Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T23:34:20.0598517Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] to include these operations in the captured graph.
2024-08-20T23:34:20.0599513Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T23:34:20.0600413Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] Graph break: from user code at:
2024-08-20T23:34:20.0601815Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-20T23:34:20.0603295Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-20T23:34:20.0604765Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:34:20.0606083Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     return mod(*inputs)
2024-08-20T23:34:20.0607796Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward
2024-08-20T23:34:20.0609401Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     sequence_output = self.bert(words)
2024-08-20T23:34:20.0611155Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2024-08-20T23:34:20.0612624Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     outputs = self.model(words)
2024-08-20T23:34:20.0614261Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2024-08-20T23:34:20.0618184Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2024-08-20T23:34:20.0619413Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T23:34:20.0620146Z W0820 23:34:20.058000 35066 torch/_dynamo/variables/tensor.py:776] [0/0] 
2024-08-20T23:34:26.6489779Z pass
2024-08-20T23:34:26.6505569Z TIMING: entire_frame_compile:6.56367 backend_compile:2.54453 total_wall_time:6.56367
2024-08-20T23:34:26.6507119Z STATS: call_* op count: 440 | FakeTensorMode.__torch_dispatch__:11991 | ProxyTorchDispatchMode.__torch_dispatch__:4817
2024-08-20T23:34:26.6508570Z Dynamo produced 4 graphs covering 440 ops with 4 graph breaks (1 unique)
2024-08-20T23:34:30.4366970Z 
2024-08-20T23:34:31.2418896Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:31.2419395Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:31.2436614Z cuda eval  functorch_dp_cifar10               
2024-08-20T23:34:33.6539512Z pass
2024-08-20T23:34:33.6545772Z TIMING: backend_compile:0.89066 entire_frame_compile:2.07193 total_wall_time:2.07193
2024-08-20T23:34:33.6546915Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:2797 | ProxyTorchDispatchMode.__torch_dispatch__:1188
2024-08-20T23:34:33.6548021Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique)
2024-08-20T23:34:37.2952405Z 
2024-08-20T23:34:37.8427721Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:37.8428849Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:37.8431742Z cuda eval  functorch_maml_omniglot            
2024-08-20T23:34:38.6670473Z pass
2024-08-20T23:34:38.6671250Z TIMING: backend_compile:0.27629 entire_frame_compile:0.62007 total_wall_time:0.62007
2024-08-20T23:34:38.6672331Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1103 | ProxyTorchDispatchMode.__torch_dispatch__:304
2024-08-20T23:34:38.6673287Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique)
2024-08-20T23:34:42.3604874Z 
2024-08-20T23:34:43.6483295Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:34:43.6484176Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:34:43.6491419Z cuda eval  hf_Albert                          
2024-08-20T23:35:23.7961815Z pass
2024-08-20T23:35:23.8099826Z TIMING: backend_compile:27.00535 entire_frame_compile:37.85501 total_wall_time:37.85501
2024-08-20T23:35:23.8101321Z STATS: call_* op count: 874 | FakeTensorMode.__torch_dispatch__:39215 | attempt fast:2602 | fast is_contiguous:2602 | ProxyTorchDispatchMode.__torch_dispatch__:7936
2024-08-20T23:35:23.8102838Z Dynamo produced 1 graphs covering 874 ops with 0 graph breaks (0 unique)
2024-08-20T23:35:29.1982289Z 
2024-08-20T23:35:33.1514000Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:35:33.1514482Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:35:33.1573913Z cuda eval  hf_Bart                            
2024-08-20T23:36:15.4565098Z pass
2024-08-20T23:36:15.4822866Z TIMING: backend_compile:26.27789 entire_frame_compile:38.9143 total_wall_time:38.9143
2024-08-20T23:36:15.4824535Z STATS: call_* op count: 902 | FakeTensorMode.__torch_dispatch__:44420 | attempt fast:2611 | fast is_contiguous:2610 | slow no contiguity match:1 | FakeTensor.__torch_dispatch__:324 | ProxyTorchDispatchMode.__torch_dispatch__:9083
2024-08-20T23:36:15.4826554Z Dynamo produced 1 graphs covering 902 ops with 0 graph breaks (0 unique)
2024-08-20T23:36:20.7924866Z 
2024-08-20T23:36:23.3584962Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:36:23.3585434Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:36:23.3633630Z cuda eval  hf_Bert                            
2024-08-20T23:37:04.2663603Z pass
2024-08-20T23:37:04.2783500Z TIMING: backend_compile:25.93301 entire_frame_compile:38.24017 total_wall_time:38.24017
2024-08-20T23:37:04.2786200Z STATS: call_* op count: 734 | FakeTensorMode.__torch_dispatch__:39065 | attempt fast:2434 | fast is_contiguous:2434 | ProxyTorchDispatchMode.__torch_dispatch__:7998
2024-08-20T23:37:04.2787520Z Dynamo produced 1 graphs covering 734 ops with 0 graph breaks (0 unique)
2024-08-20T23:37:09.1007130Z 
2024-08-20T23:37:13.8166668Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:37:13.8167491Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:37:13.8257093Z cuda eval  hf_Bert_large                      
2024-08-20T23:38:35.6616523Z pass
2024-08-20T23:38:35.6830990Z TIMING: backend_compile:51.7113 entire_frame_compile:75.37235 total_wall_time:75.37235
2024-08-20T23:38:35.6832852Z STATS: call_* op count: 1430 | FakeTensorMode.__torch_dispatch__:76505 | attempt fast:4738 | fast is_contiguous:4738 | ProxyTorchDispatchMode.__torch_dispatch__:15654
2024-08-20T23:38:35.6834582Z Dynamo produced 1 graphs covering 1430 ops with 0 graph breaks (0 unique)
2024-08-20T23:38:41.5036801Z 
2024-08-20T23:38:44.7316068Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:38:44.7316536Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:38:44.7371504Z cuda eval  hf_BigBird                         
2024-08-20T23:39:41.8378165Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0] Error while creating guard:
2024-08-20T23:39:41.8379271Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0] Name: ''
2024-08-20T23:39:41.8380125Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Source: shape_env
2024-08-20T23:39:41.8381188Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Create Function: SHAPE_ENV
2024-08-20T23:39:41.8382087Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Guard Types: None
2024-08-20T23:39:41.8384265Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Code List: None
2024-08-20T23:39:41.8385073Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Object Weakref: None
2024-08-20T23:39:41.8385904Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     Guarded Class Weakref: None
2024-08-20T23:39:41.8386751Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0] Traceback (most recent call last):
2024-08-20T23:39:41.8388501Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-20T23:39:41.8390467Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     return self.create_fn(builder, self)
2024-08-20T23:39:41.8392183Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-20T23:39:41.8393687Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     guards = output_graph.shape_env.produce_guards(
2024-08-20T23:39:41.8395546Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-20T23:39:41.8397206Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]     raise ConstraintViolationError(
2024-08-20T23:39:41.8398889Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-20T23:39:41.8401291Z E0820 23:39:41.835000 35676 torch/_guards.py:282] [0/0]   - Not all values of RelaxedUnspecConstraint(L['inputs'][0].size()[0]) are valid because L['inputs'][0].size()[0] was inferred to be a constant (4).
2024-08-20T23:39:41.8402519Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0] Created at:
2024-08-20T23:39:41.8403815Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 603, in transform
2024-08-20T23:39:41.8405074Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]     tracer = InstructionTranslator(
2024-08-20T23:39:41.8406510Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2654, in __init__
2024-08-20T23:39:41.8407735Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]     output=OutputGraph(
2024-08-20T23:39:41.8409081Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 317, in __init__
2024-08-20T23:39:41.8410314Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]     self.init_ambient_guards()
2024-08-20T23:39:41.8411916Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 463, in init_ambient_guards
2024-08-20T23:39:41.8413391Z E0820 23:39:41.837000 35676 torch/_guards.py:284] [0/0]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
2024-08-20T23:39:41.8414172Z ERROR:common:
2024-08-20T23:39:41.8414487Z Traceback (most recent call last):
2024-08-20T23:39:41.8415189Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2819, in check_accuracy
2024-08-20T23:39:41.8416013Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2024-08-20T23:39:41.8417001Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn
2024-08-20T23:39:41.8417765Z     return fn(*args, **kwargs)
2024-08-20T23:39:41.8418637Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__
2024-08-20T23:39:41.8419457Z     return self._torchdynamo_orig_callable(
2024-08-20T23:39:41.8420373Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1039, in __call__
2024-08-20T23:39:41.8421202Z     result = self._inner_convert(
2024-08-20T23:39:41.8422073Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__
2024-08-20T23:39:41.8422896Z     return _compile(
2024-08-20T23:39:41.8423709Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile
2024-08-20T23:39:41.8424625Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2024-08-20T23:39:41.8425779Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner
2024-08-20T23:39:41.8426703Z     return _compile_inner(code, one_graph, hooks, transform)
2024-08-20T23:39:41.8427693Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
2024-08-20T23:39:41.8428505Z     return function(*args, **kwargs)
2024-08-20T23:39:41.8429438Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 783, in _compile_inner
2024-08-20T23:39:41.8430279Z     check_fn = CheckFunctionManager(
2024-08-20T23:39:41.8431145Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2200, in __init__
2024-08-20T23:39:41.8431905Z     guard.create(builder)
2024-08-20T23:39:41.8432660Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-20T23:39:41.8433402Z     return self.create_fn(builder, self)
2024-08-20T23:39:41.8434387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-20T23:39:41.8435227Z     guards = output_graph.shape_env.produce_guards(
2024-08-20T23:39:41.8436307Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-20T23:39:41.8437227Z     raise ConstraintViolationError(
2024-08-20T23:39:41.8438387Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-20T23:39:41.8440016Z   - Not all values of RelaxedUnspecConstraint(L['inputs'][0].size()[0]) are valid because L['inputs'][0].size()[0] was inferred to be a constant (4).
2024-08-20T23:39:41.8440791Z 
2024-08-20T23:39:41.8440796Z 
2024-08-20T23:39:41.8441084Z You can suppress this exception and fall back to eager by setting:
2024-08-20T23:39:41.8441643Z     import torch._dynamo
2024-08-20T23:39:41.8442066Z     torch._dynamo.config.suppress_errors = True
2024-08-20T23:39:41.8442398Z 
2024-08-20T23:39:41.8442692Z TorchDynamo optimized model failed to run because of following error
2024-08-20T23:39:41.8449717Z fail_to_run
2024-08-20T23:39:41.8637275Z TIMING: backend_compile:21.82866 total_wall_time:0.0
2024-08-20T23:39:41.8638541Z STATS: call_* op count: 6706 | FakeTensorMode.__torch_dispatch__:129138 | attempt fast:18 | slow both tensors nontrivially broadcast:1 | fast is_contiguous:17 | ProxyTorchDispatchMode.__torch_dispatch__:51076
2024-08-20T23:39:41.8639887Z Dynamo produced 1 graphs covering 6706 ops with 0 graph breaks (0 unique)
2024-08-20T23:39:46.3779159Z 
2024-08-20T23:39:48.4846808Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:39:48.4847744Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:39:48.4871515Z cuda eval  hf_DistilBert                      
2024-08-20T23:40:10.0671913Z pass
2024-08-20T23:40:10.0801579Z TIMING: backend_compile:13.72827 entire_frame_compile:19.93897 total_wall_time:19.93897
2024-08-20T23:40:10.0803262Z STATS: call_* op count: 408 | FakeTensorMode.__torch_dispatch__:21269 | attempt fast:1230 | fast is_contiguous:1194 | slow no contiguity match:36 | ProxyTorchDispatchMode.__torch_dispatch__:4225
2024-08-20T23:40:10.0804852Z Dynamo produced 1 graphs covering 408 ops with 0 graph breaks (0 unique)
2024-08-20T23:40:14.6181714Z 
2024-08-20T23:40:17.8582836Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:40:17.8583305Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:40:17.8619931Z cuda eval  hf_GPT2                            
2024-08-20T23:40:56.5746419Z pass
2024-08-20T23:40:56.6156185Z TIMING: backend_compile:24.3219 entire_frame_compile:35.35872 total_wall_time:35.35872
2024-08-20T23:40:56.6158652Z STATS: call_* op count: 1244 | FakeTensorMode.__torch_dispatch__:36622 | attempt fast:2124 | fast is_contiguous:2124 | ProxyTorchDispatchMode.__torch_dispatch__:7934
2024-08-20T23:40:56.6161431Z Dynamo produced 1 graphs covering 1244 ops with 0 graph breaks (0 unique)
2024-08-20T23:41:01.8867562Z 
2024-08-20T23:41:15.1620703Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:41:15.1621201Z loading model: 0it [00:13, ?it/s]
2024-08-20T23:41:15.1724317Z cuda eval  hf_GPT2_large                      
2024-08-20T23:41:15.1728046Z pass_due_to_skip
2024-08-20T23:41:15.5125502Z TIMING:
2024-08-20T23:41:15.5125831Z STATS: call_* op count: 0
2024-08-20T23:41:15.5126499Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T23:41:19.1017762Z 
2024-08-20T23:41:20.3727108Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:41:20.3728036Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:41:20.3748526Z cuda eval  hf_Reformer                        
2024-08-20T23:41:22.4522115Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T23:41:22.4523671Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T23:41:22.4525186Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] or:
2024-08-20T23:41:22.4526184Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T23:41:22.4527496Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] to include these operations in the captured graph.
2024-08-20T23:41:22.4528570Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] 
2024-08-20T23:41:22.4529467Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break: from user code at:
2024-08-20T23:41:22.4531681Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066
2024-08-20T23:41:22.4533444Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]     embedding_output = self.embeddings(
2024-08-20T23:41:22.4535204Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 266, in forward
2024-08-20T23:41:22.4536890Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]     position_embeddings = self.position_embeddings(position_ids)
2024-08-20T23:41:22.4538733Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 190, in forward
2024-08-20T23:41:22.4540381Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0]     max_position_id = position_ids.max().item()
2024-08-20T23:41:22.4541319Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] 
2024-08-20T23:41:22.4542048Z W0820 23:41:22.451000 36010 torch/_dynamo/variables/tensor.py:776] [5/0] 
2024-08-20T23:41:26.9283351Z pass
2024-08-20T23:41:26.9382476Z TIMING: entire_frame_compile:4.83113 backend_compile:1.90213 total_wall_time:4.83113
2024-08-20T23:41:26.9383815Z STATS: call_* op count: 483 | FakeTensorMode.__torch_dispatch__:9890 | ProxyTorchDispatchMode.__torch_dispatch__:3960 | FakeTensor.__torch_dispatch__:13 | attempt fast:2 | fast is_contiguous:2
2024-08-20T23:41:26.9385090Z Dynamo produced 7 graphs covering 483 ops with 5 graph breaks (2 unique)
2024-08-20T23:41:30.7281510Z 
2024-08-20T23:41:32.7176399Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:41:32.7176893Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:41:32.7212176Z cuda eval  hf_T5                              
2024-08-20T23:42:29.6924933Z pass
2024-08-20T23:42:29.7414911Z TIMING: backend_compile:33.54203 entire_frame_compile:46.92962 total_wall_time:46.92962
2024-08-20T23:42:29.7419961Z STATS: call_* op count: 1654 | FakeTensorMode.__torch_dispatch__:46924 | attempt fast:1302 | fast is_contiguous:1236 | slow both tensors nontrivially broadcast:18 | slow no contiguity match:48 | ProxyTorchDispatchMode.__torch_dispatch__:12999
2024-08-20T23:42:29.7421501Z Dynamo produced 1 graphs covering 1654 ops with 0 graph breaks (0 unique)
2024-08-20T23:42:35.5960703Z 
2024-08-20T23:42:39.6654232Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:42:39.6654712Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:42:39.6656302Z cuda eval  hf_T5_base                         
2024-08-20T23:42:39.6669671Z Traceback (most recent call last):
2024-08-20T23:42:39.6670492Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:42:39.6671327Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:42:39.6672174Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass
2024-08-20T23:42:39.6672893Z     return mod(*inputs)
2024-08-20T23:42:39.6674010Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6675227Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6676161Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6676998Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6677959Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward
2024-08-20T23:42:39.6679082Z     return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
2024-08-20T23:42:39.6680239Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6681110Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6682047Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6682870Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6683860Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward
2024-08-20T23:42:39.6684991Z     return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids)
2024-08-20T23:42:39.6686141Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6687026Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6687955Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6688783Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6689735Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1748, in forward
2024-08-20T23:42:39.6690792Z     decoder_outputs = self.decoder(
2024-08-20T23:42:39.6691774Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6692657Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6693576Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6694397Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6695358Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 1115, in forward
2024-08-20T23:42:39.6696203Z     layer_outputs = layer_module(
2024-08-20T23:42:39.6697136Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6698002Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6698924Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6699745Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6700863Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 695, in forward
2024-08-20T23:42:39.6701729Z     self_attention_outputs = self.layer[0](
2024-08-20T23:42:39.6702721Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6703599Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6704532Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6705348Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6706303Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 602, in forward
2024-08-20T23:42:39.6707169Z     attention_output = self.SelfAttention(
2024-08-20T23:42:39.6708158Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:42:39.6709033Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:42:39.6710052Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:42:39.6710880Z     return forward_call(*args, **kwargs)
2024-08-20T23:42:39.6711834Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/t5/modeling_t5.py", line 562, in forward
2024-08-20T23:42:39.6712911Z     attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(
2024-08-20T23:42:39.6716233Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 768.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 58.44 MiB is free. Process 179053 has 21.91 GiB memory in use. Of the allocated memory 20.94 GiB is allocated by PyTorch, and 666.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:42:39.6718947Z 
2024-08-20T23:42:39.6719256Z The above exception was the direct cause of the following exception:
2024-08-20T23:42:39.6719701Z 
2024-08-20T23:42:39.6719856Z Traceback (most recent call last):
2024-08-20T23:42:39.6720506Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:42:39.6721215Z     ) = runner.load_model(
2024-08-20T23:42:39.6721879Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:42:39.6722637Z     self.validate_model(model, example_inputs)
2024-08-20T23:42:39.6723390Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:42:39.6724163Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:42:39.6724643Z RuntimeError: Eager run failed
2024-08-20T23:42:39.6724901Z 
2024-08-20T23:42:39.6725026Z eager_fail_to_run
2024-08-20T23:42:43.2780688Z 
2024-08-20T23:42:50.3206991Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:42:50.3207494Z loading model: 0it [00:07, ?it/s]
2024-08-20T23:42:50.3240206Z cuda eval  hf_T5_generate                     
2024-08-20T23:43:17.1976214Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T23:43:17.1977484Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T23:43:17.1978481Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] or:
2024-08-20T23:43:17.1979601Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T23:43:17.1980795Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] to include these operations in the captured graph.
2024-08-20T23:43:17.1981780Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] 
2024-08-20T23:43:17.1983031Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] Graph break: from user code at:
2024-08-20T23:43:17.1985047Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 1392, in torch_dynamo_resume_in_generate_at_1348
2024-08-20T23:43:17.1986864Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]     model_kwargs["attention_mask"] = self._prepare_attention_mask_for_generation(
2024-08-20T23:43:17.1988845Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/generation/utils.py", line 476, in _prepare_attention_mask_for_generation
2024-08-20T23:43:17.1990654Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0]     is_pad_token_in_inputs = (pad_token_id is not None) and (pad_token_id in inputs)
2024-08-20T23:43:17.1991797Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] 
2024-08-20T23:43:17.1992684Z W0820 23:43:17.196000 36248 torch/_dynamo/variables/tensor.py:776] [4/0] 
2024-08-20T23:44:01.0774119Z pass
2024-08-20T23:44:01.1274433Z TIMING: entire_frame_compile:27.72488 backend_compile:10.96357 total_wall_time:27.72488
2024-08-20T23:44:01.1276114Z STATS: call_* op count: 1815 | FakeTensorMode.__torch_dispatch__:39701 | ProxyTorchDispatchMode.__torch_dispatch__:14586 | FakeTensor.__torch_dispatch__:82 | attempt fast:81 | fast is_contiguous:60 | slow both tensors nontrivially broadcast:9 | slow no contiguity match:12
2024-08-20T23:44:01.1277730Z Dynamo produced 4 graphs covering 1815 ops with 5 graph breaks (3 unique)
2024-08-20T23:44:07.1198558Z 
2024-08-20T23:44:17.3095921Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:44:17.3096387Z loading model: 0it [00:10, ?it/s]
2024-08-20T23:44:17.3501468Z cuda eval  hf_T5_large                        
2024-08-20T23:44:17.3507266Z pass_due_to_skip
2024-08-20T23:44:17.6474217Z TIMING:
2024-08-20T23:44:17.6475372Z STATS: call_* op count: 0
2024-08-20T23:44:17.6476508Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T23:44:21.2291158Z 
2024-08-20T23:44:22.3355361Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:44:22.3356239Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:44:22.3372626Z cuda eval  hf_Whisper                         
2024-08-20T23:44:25.8763283Z pass
2024-08-20T23:44:25.8790033Z TIMING: backend_compile:1.13619 entire_frame_compile:2.33536 total_wall_time:2.33536
2024-08-20T23:44:25.8791487Z STATS: call_* op count: 230 | FakeTensorMode.__torch_dispatch__:5871 | FakeTensor.__torch_dispatch__:126 | ProxyTorchDispatchMode.__torch_dispatch__:2362
2024-08-20T23:44:25.8792854Z Dynamo produced 1 graphs covering 230 ops with 0 graph breaks (0 unique)
2024-08-20T23:44:29.6457723Z 
2024-08-20T23:44:34.3374930Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:44:34.3375567Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:44:34.3470133Z cuda eval  hf_distil_whisper                  
2024-08-20T23:45:10.8037106Z pass
2024-08-20T23:45:10.8243931Z TIMING: backend_compile:6.61313 entire_frame_compile:12.97558 total_wall_time:12.97558
2024-08-20T23:45:10.8245141Z STATS: call_* op count: 1270 | FakeTensorMode.__torch_dispatch__:31871 | FakeTensor.__torch_dispatch__:726 | ProxyTorchDispatchMode.__torch_dispatch__:13102
2024-08-20T23:45:10.8246269Z Dynamo produced 1 graphs covering 1270 ops with 0 graph breaks (0 unique)
2024-08-20T23:45:14.7601572Z 
2024-08-20T23:45:15.1863458Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:45:15.1863936Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:45:15.1867280Z cuda eval  lennard_jones                      
2024-08-20T23:45:15.7251902Z pass
2024-08-20T23:45:15.7253041Z TIMING: backend_compile:0.16517 entire_frame_compile:0.35762 total_wall_time:0.35762
2024-08-20T23:45:15.7254252Z STATS: call_* op count: 18 | FakeTensorMode.__torch_dispatch__:537 | ProxyTorchDispatchMode.__torch_dispatch__:184
2024-08-20T23:45:15.7255904Z Dynamo produced 1 graphs covering 18 ops with 0 graph breaks (0 unique)
2024-08-20T23:45:16.5504669Z accuracy            pass_rate=66.67%
2024-08-20T23:45:16.5507656Z calls_captured      gmean=0.00x mean=580.542x
2024-08-20T23:45:16.5511875Z unique_graphs       gmean=0.00x mean=2.188x
2024-08-20T23:45:16.5514925Z graph_breaks        gmean=0.00x mean=1.188x
2024-08-20T23:45:16.5518822Z unique_graph_breaks gmean=0.00x mean=0.396x
2024-08-20T23:45:16.5522475Z autograd_captures   gmean=0.00x mean=0.000x
2024-08-20T23:45:16.5525913Z autograd_compiles   gmean=0.00x mean=0.000x
2024-08-20T23:45:16.5529245Z cudagraph_skips     gmean=0.00x mean=0.000x
2024-08-20T23:45:17.1444520Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_inference.csv
2024-08-20T23:45:17.5152527Z torchrec_dlrm                       XFAIL
2024-08-20T23:45:17.5155382Z BERT_pytorch                        PASS
2024-08-20T23:45:17.5160624Z Background_Matting                  XFAIL
2024-08-20T23:45:17.5165549Z LearningToPaint                     PASS
2024-08-20T23:45:17.5171327Z Super_SloMo                         PASS
2024-08-20T23:45:17.5176618Z alexnet                             PASS
2024-08-20T23:45:17.5181793Z basic_gnn_edgecnn                   PASS
2024-08-20T23:45:17.5186568Z basic_gnn_gcn                       PASS
2024-08-20T23:45:17.5191362Z basic_gnn_gin                       PASS
2024-08-20T23:45:17.5196057Z basic_gnn_sage                      PASS
2024-08-20T23:45:17.5200898Z cm3leon_generate                    PASS
2024-08-20T23:45:17.5205461Z dcgan                               PASS
2024-08-20T23:45:17.5210293Z demucs                              PASS
2024-08-20T23:45:17.5215206Z densenet121                         PASS
2024-08-20T23:45:17.5220026Z detectron2_fasterrcnn_r_101_c4      XFAIL
2024-08-20T23:45:17.5224801Z detectron2_fasterrcnn_r_101_dc5     XFAIL
2024-08-20T23:45:17.5229709Z detectron2_fasterrcnn_r_101_fpn     XFAIL
2024-08-20T23:45:17.5234330Z detectron2_fasterrcnn_r_50_c4       XFAIL
2024-08-20T23:45:17.5239010Z detectron2_fasterrcnn_r_50_dc5      XFAIL
2024-08-20T23:45:17.5243818Z detectron2_fasterrcnn_r_50_fpn      XFAIL
2024-08-20T23:45:17.5248515Z detectron2_fcos_r_50_fpn            PASS
2024-08-20T23:45:17.5253390Z detectron2_maskrcnn_r_101_c4        XFAIL
2024-08-20T23:45:17.5258152Z detectron2_maskrcnn_r_101_fpn       XFAIL
2024-08-20T23:45:17.5262771Z detectron2_maskrcnn_r_50_c4         XFAIL
2024-08-20T23:45:17.5267789Z detectron2_maskrcnn_r_50_fpn        XFAIL
2024-08-20T23:45:17.5272697Z dlrm                                PASS
2024-08-20T23:45:17.5277608Z doctr_det_predictor                 PASS
2024-08-20T23:45:17.5282398Z doctr_reco_predictor                PASS
2024-08-20T23:45:17.5287137Z drq                                 PASS
2024-08-20T23:45:17.5292047Z fastNLP_Bert                        PASS
2024-08-20T23:45:17.5296855Z functorch_dp_cifar10                PASS
2024-08-20T23:45:17.5301502Z functorch_maml_omniglot             PASS
2024-08-20T23:45:17.5306359Z hf_Albert                           PASS
2024-08-20T23:45:17.5310987Z hf_Bart                             PASS
2024-08-20T23:45:17.5315744Z hf_Bert                             PASS
2024-08-20T23:45:17.5320494Z hf_Bert_large                       PASS
2024-08-20T23:45:17.5325156Z hf_BigBird                          XFAIL
2024-08-20T23:45:17.5329956Z hf_DistilBert                       PASS
2024-08-20T23:45:17.5334786Z hf_GPT2                             PASS
2024-08-20T23:45:17.5339496Z hf_GPT2_large                       XFAIL
2024-08-20T23:45:17.5344287Z hf_Reformer                         PASS
2024-08-20T23:45:17.5348970Z hf_T5                               PASS
2024-08-20T23:45:17.5353669Z hf_T5_base                          XFAIL
2024-08-20T23:45:17.5358288Z hf_T5_generate                      PASS
2024-08-20T23:45:17.5363146Z hf_T5_large                         XFAIL
2024-08-20T23:45:17.5368401Z hf_Whisper                          PASS
2024-08-20T23:45:17.5373185Z hf_distil_whisper                   PASS
2024-08-20T23:45:17.5377832Z lennard_jones                       PASS
2024-08-20T23:45:17.5896564Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_inference.csv
2024-08-20T23:45:17.9569356Z torchrec_dlrm                       PASS
2024-08-20T23:45:17.9572547Z BERT_pytorch                        PASS
2024-08-20T23:45:17.9576985Z Background_Matting                  PASS
2024-08-20T23:45:17.9581183Z LearningToPaint                     PASS
2024-08-20T23:45:17.9585586Z Super_SloMo                         PASS
2024-08-20T23:45:17.9590035Z alexnet                             PASS
2024-08-20T23:45:17.9594509Z basic_gnn_edgecnn                   PASS
2024-08-20T23:45:17.9598710Z basic_gnn_gcn                       PASS
2024-08-20T23:45:17.9603195Z basic_gnn_gin                       PASS
2024-08-20T23:45:17.9607721Z basic_gnn_sage                      PASS
2024-08-20T23:45:17.9612095Z cm3leon_generate                    PASS
2024-08-20T23:45:17.9616504Z dcgan                               PASS
2024-08-20T23:45:17.9620769Z demucs                              PASS
2024-08-20T23:45:17.9625136Z densenet121                         PASS
2024-08-20T23:45:17.9629460Z detectron2_fasterrcnn_r_101_c4      PASS
2024-08-20T23:45:17.9633955Z detectron2_fasterrcnn_r_101_dc5     PASS
2024-08-20T23:45:17.9638220Z detectron2_fasterrcnn_r_101_fpn     PASS
2024-08-20T23:45:17.9642771Z detectron2_fasterrcnn_r_50_c4       PASS
2024-08-20T23:45:17.9647055Z detectron2_fasterrcnn_r_50_dc5      PASS
2024-08-20T23:45:17.9651612Z detectron2_fasterrcnn_r_50_fpn      PASS
2024-08-20T23:45:17.9655839Z detectron2_fcos_r_50_fpn            PASS
2024-08-20T23:45:17.9660390Z detectron2_maskrcnn_r_101_c4        PASS
2024-08-20T23:45:17.9664566Z detectron2_maskrcnn_r_101_fpn       PASS
2024-08-20T23:45:17.9669397Z detectron2_maskrcnn_r_50_c4         PASS
2024-08-20T23:45:17.9674084Z detectron2_maskrcnn_r_50_fpn        PASS
2024-08-20T23:45:17.9678365Z dlrm                                PASS
2024-08-20T23:45:17.9682677Z doctr_det_predictor                 PASS
2024-08-20T23:45:17.9687181Z doctr_reco_predictor                PASS
2024-08-20T23:45:17.9691433Z drq                                 PASS
2024-08-20T23:45:17.9696070Z fastNLP_Bert                        PASS
2024-08-20T23:45:17.9700293Z functorch_dp_cifar10                PASS
2024-08-20T23:45:17.9704816Z functorch_maml_omniglot             PASS
2024-08-20T23:45:17.9709005Z hf_Albert                           PASS
2024-08-20T23:45:17.9713468Z hf_Bart                             PASS
2024-08-20T23:45:17.9717651Z hf_Bert                             PASS
2024-08-20T23:45:17.9722193Z hf_Bert_large                       PASS
2024-08-20T23:45:17.9726418Z hf_BigBird                          PASS
2024-08-20T23:45:17.9730948Z hf_DistilBert                       PASS
2024-08-20T23:45:17.9735370Z hf_GPT2                             PASS
2024-08-20T23:45:17.9739745Z hf_GPT2_large                       PASS
2024-08-20T23:45:17.9744088Z hf_Reformer                         PASS
2024-08-20T23:45:17.9748627Z hf_T5                               PASS
2024-08-20T23:45:17.9752858Z hf_T5_base                          PASS
2024-08-20T23:45:17.9757412Z hf_T5_generate                      PASS
2024-08-20T23:45:17.9761633Z hf_T5_large                         PASS
2024-08-20T23:45:17.9766133Z hf_Whisper                          PASS
2024-08-20T23:45:17.9771543Z hf_distil_whisper                   PASS
2024-08-20T23:45:17.9776177Z lennard_jones                       PASS
2024-08-20T23:45:18.0301836Z + test_single_dynamo_benchmark training torchbench 0 --training --amp
2024-08-20T23:45:18.0304374Z ++ pwd
2024-08-20T23:45:18.0307860Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-08-20T23:45:18.0308953Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-08-20T23:45:18.0331482Z + local name=training
2024-08-20T23:45:18.0331980Z + shift
2024-08-20T23:45:18.0332369Z + local suite=torchbench
2024-08-20T23:45:18.0332827Z + shift
2024-08-20T23:45:18.0333162Z + local shard_id=0
2024-08-20T23:45:18.0335542Z + shift
2024-08-20T23:45:18.0335943Z + partition_flags=()
2024-08-20T23:45:18.0336406Z + local partition_flags
2024-08-20T23:45:18.0336958Z + [[ -n 2 ]]
2024-08-20T23:45:18.0337281Z + [[ -n 0 ]]
2024-08-20T23:45:18.0338095Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-08-20T23:45:18.0339019Z + [[ dynamic_aot_eager_torchbench == *perf_compare* ]]
2024-08-20T23:45:18.0339548Z + [[ dynamic_aot_eager_torchbench == *perf* ]]
2024-08-20T23:45:18.0340065Z + [[ dynamic_aot_eager_torchbench == *aot_inductor* ]]
2024-08-20T23:45:18.0340586Z + [[ dynamic_aot_eager_torchbench == *_avx2* ]]
2024-08-20T23:45:18.0341082Z + [[ dynamic_aot_eager_torchbench == *_avx512* ]]
2024-08-20T23:45:18.0342930Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --dynamic-shapes --dynamic-batch-only --device cuda --training --amp --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv
2024-08-20T23:45:23.7573019Z 
2024-08-20T23:45:24.3738201Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead.
2024-08-20T23:45:24.3740186Z   from torch.distributed.optim import _apply_optimizer_in_backward
2024-08-20T23:45:24.7669586Z 
2024-08-20T23:45:24.7670298Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:45:24.7670902Z cuda train torchrec_dlrm                      
2024-08-20T23:45:30.1846770Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1] Error while creating guard:
2024-08-20T23:45:30.1848131Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1] Name: ''
2024-08-20T23:45:30.1848925Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Source: shape_env
2024-08-20T23:45:30.1849920Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Create Function: SHAPE_ENV
2024-08-20T23:45:30.1851237Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Guard Types: None
2024-08-20T23:45:30.1852141Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Code List: None
2024-08-20T23:45:30.1853116Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Object Weakref: None
2024-08-20T23:45:30.1854323Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     Guarded Class Weakref: None
2024-08-20T23:45:30.1855567Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1] Traceback (most recent call last):
2024-08-20T23:45:30.1857510Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-20T23:45:30.1859073Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     return self.create_fn(builder, self)
2024-08-20T23:45:30.1860576Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-20T23:45:30.1861963Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     guards = output_graph.shape_env.produce_guards(
2024-08-20T23:45:30.1863723Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-20T23:45:30.1865472Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]     raise ConstraintViolationError(
2024-08-20T23:45:30.1868281Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0][0].size()[0], L['cloned_inputs'][0][2].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-20T23:45:30.1870775Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0][0].size()[0]) are valid because L['cloned_inputs'][0][0].size()[0] was inferred to be a constant (4).
2024-08-20T23:45:30.1872999Z [rank0]:E0820 23:45:30.182000 36701 torch/_guards.py:282] [3/0_1]   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0][2].size()[0]) are valid because L['cloned_inputs'][0][2].size()[0] was inferred to be a constant (4).
2024-08-20T23:45:30.1874395Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1] Created at:
2024-08-20T23:45:30.1875785Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 603, in transform
2024-08-20T23:45:30.1877373Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]     tracer = InstructionTranslator(
2024-08-20T23:45:30.1878897Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2654, in __init__
2024-08-20T23:45:30.1880204Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]     output=OutputGraph(
2024-08-20T23:45:30.1881644Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 317, in __init__
2024-08-20T23:45:30.1882952Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]     self.init_ambient_guards()
2024-08-20T23:45:30.1884489Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 463, in init_ambient_guards
2024-08-20T23:45:30.1886064Z [rank0]:E0820 23:45:30.184000 36701 torch/_guards.py:284] [3/0_1]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
2024-08-20T23:45:30.1886902Z ERROR:common:
2024-08-20T23:45:30.1887226Z Traceback (most recent call last):
2024-08-20T23:45:30.1887938Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2819, in check_accuracy
2024-08-20T23:45:30.1888790Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2024-08-20T23:45:30.1889782Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn
2024-08-20T23:45:30.1890621Z     return fn(*args, **kwargs)
2024-08-20T23:45:30.1891313Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-20T23:45:30.1892128Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-20T23:45:30.1893037Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 437, in forward_and_backward_pass
2024-08-20T23:45:30.1893881Z     cloned_inputs = clone_inputs(inputs)
2024-08-20T23:45:30.1895167Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 438, in torch_dynamo_resume_in_forward_and_backward_pass_at_437
2024-08-20T23:45:30.1896230Z     self.optimizer_zero_grad(mod)
2024-08-20T23:45:30.1897246Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__
2024-08-20T23:45:30.1898090Z     return self._torchdynamo_orig_callable(
2024-08-20T23:45:30.1899015Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1039, in __call__
2024-08-20T23:45:30.1899818Z     result = self._inner_convert(
2024-08-20T23:45:30.1900765Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__
2024-08-20T23:45:30.1901675Z     return _compile(
2024-08-20T23:45:30.1902513Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile
2024-08-20T23:45:30.1903435Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2024-08-20T23:45:30.1904493Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner
2024-08-20T23:45:30.1905416Z     return _compile_inner(code, one_graph, hooks, transform)
2024-08-20T23:45:30.1906420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
2024-08-20T23:45:30.1907233Z     return function(*args, **kwargs)
2024-08-20T23:45:30.1908271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 783, in _compile_inner
2024-08-20T23:45:30.1909134Z     check_fn = CheckFunctionManager(
2024-08-20T23:45:30.1910032Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2200, in __init__
2024-08-20T23:45:30.1910905Z     guard.create(builder)
2024-08-20T23:45:30.1911676Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-20T23:45:30.1912432Z     return self.create_fn(builder, self)
2024-08-20T23:45:30.1913333Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-20T23:45:30.1914187Z     guards = output_graph.shape_env.produce_guards(
2024-08-20T23:45:30.1915291Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-20T23:45:30.1916231Z     raise ConstraintViolationError(
2024-08-20T23:45:30.1917649Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0][0].size()[0], L['cloned_inputs'][0][2].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-20T23:45:30.1919544Z   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0][0].size()[0]) are valid because L['cloned_inputs'][0][0].size()[0] was inferred to be a constant (4).
2024-08-20T23:45:30.1921298Z   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0][2].size()[0]) are valid because L['cloned_inputs'][0][2].size()[0] was inferred to be a constant (4).
2024-08-20T23:45:30.1922154Z 
2024-08-20T23:45:30.1922160Z 
2024-08-20T23:45:30.1922445Z You can suppress this exception and fall back to eager by setting:
2024-08-20T23:45:30.1923008Z     import torch._dynamo
2024-08-20T23:45:30.1923421Z     torch._dynamo.config.suppress_errors = True
2024-08-20T23:45:30.1923750Z 
2024-08-20T23:45:30.1924052Z TorchDynamo optimized model failed to run because of following error
2024-08-20T23:45:30.1924611Z fail_to_run
2024-08-20T23:45:30.3326078Z TIMING: entire_frame_compile:0.02343 backend_compile:2.32259 total_wall_time:0.02343
2024-08-20T23:45:30.3327576Z STATS: call_* op count: 123 | FakeTensorMode.__torch_dispatch__:6438 | attempt fast:14 | fast is_contiguous:14 | FakeTensor.__torch_dispatch__:314 | ProxyTorchDispatchMode.__torch_dispatch__:2474
2024-08-20T23:45:30.3328878Z Dynamo produced 1 graphs covering 123 ops with 3 graph breaks (3 unique)
2024-08-20T23:45:30.9737396Z [rank0]:[W820 23:45:30.826120745 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present,  but this warning has only been added since PyTorch 2.4 (function operator())
2024-08-20T23:45:34.2435406Z 
2024-08-20T23:45:36.4138664Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:45:36.4139294Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:45:36.4139875Z cuda train BERT_pytorch                       
2024-08-20T23:46:19.6233957Z pass
2024-08-20T23:46:19.7105193Z TIMING: entire_frame_compile:40.92425 backend_compile:27.56273 total_wall_time:40.92425
2024-08-20T23:46:19.7106839Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:45785 | attempt fast:1964 | fast is_contiguous:1964 | ProxyTorchDispatchMode.__torch_dispatch__:15567 | FakeTensor.__torch_dispatch__:22
2024-08-20T23:46:19.7108354Z Dynamo produced 2 graphs covering 542 ops with 6 graph breaks (5 unique)
2024-08-20T23:46:24.7820099Z 
2024-08-20T23:46:29.5997598Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:46:29.5998692Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:46:29.5999737Z cuda train Background_Matting                 
2024-08-20T23:46:29.6000855Z pass_due_to_skip
2024-08-20T23:46:29.6277510Z TIMING:
2024-08-20T23:46:29.6277923Z STATS: call_* op count: 0
2024-08-20T23:46:29.6278479Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-20T23:46:33.2674588Z 
2024-08-20T23:46:36.3463167Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:46:36.3464144Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:46:36.3465036Z cuda train LearningToPaint                    
2024-08-20T23:46:46.7106412Z pass
2024-08-20T23:46:46.7175845Z TIMING: entire_frame_compile:9.2586 backend_compile:5.20051 total_wall_time:9.2586
2024-08-20T23:46:46.7177277Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:10316 | attempt fast:434 | fast is_contiguous:434 | ProxyTorchDispatchMode.__torch_dispatch__:2365 | FakeTensor.__torch_dispatch__:22
2024-08-20T23:46:46.7178554Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique)
2024-08-20T23:46:51.0036113Z 
2024-08-20T23:46:53.5624765Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:46:53.5625244Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:46:53.5625686Z cuda train Super_SloMo                        
2024-08-20T23:49:11.2063276Z W0820 23:49:11.205000 37060 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:49:22.0449566Z pass
2024-08-20T23:49:22.0903779Z TIMING: entire_frame_compile:128.9833 backend_compile:27.68338 total_wall_time:128.9833
2024-08-20T23:49:22.0905326Z STATS: call_* op count: 671 | FakeTensorMode.__torch_dispatch__:45082 | attempt fast:1715 | fast is_contiguous:1553 | slow no contiguity match:162 | ProxyTorchDispatchMode.__torch_dispatch__:13583 | FakeTensor.__torch_dispatch__:1774
2024-08-20T23:49:22.0906853Z Dynamo produced 3 graphs covering 671 ops with 7 graph breaks (5 unique)
2024-08-20T23:49:28.0558561Z 
2024-08-20T23:49:29.5570704Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:49:29.5571285Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:49:29.5571715Z cuda train alexnet                            
2024-08-20T23:49:33.6911845Z pass
2024-08-20T23:49:33.7228306Z TIMING: entire_frame_compile:2.76315 backend_compile:1.96458 total_wall_time:2.76315
2024-08-20T23:49:33.7229894Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:2050 | attempt fast:24 | fast is_contiguous:24 | ProxyTorchDispatchMode.__torch_dispatch__:674 | FakeTensor.__torch_dispatch__:22
2024-08-20T23:49:33.7231636Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique)
2024-08-20T23:49:37.5855537Z 
2024-08-20T23:49:39.5571296Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:49:39.5571778Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:49:39.5572217Z cuda train basic_gnn_edgecnn                  
2024-08-20T23:49:42.3091005Z W0820 23:49:42.308000 37257 torch/_logging/_internal.py:1058] [24/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:49:43.3794872Z pass
2024-08-20T23:49:43.3964573Z TIMING: entire_frame_compile:2.87648 backend_compile:1.03012 total_wall_time:2.87648
2024-08-20T23:49:43.3967977Z STATS: call_* op count: 102 | FakeTensorMode.__torch_dispatch__:3640 | ProxyTorchDispatchMode.__torch_dispatch__:1462 | FakeTensor.__torch_dispatch__:398
2024-08-20T23:49:43.3969633Z Dynamo produced 8 graphs covering 102 ops with 20 graph breaks (6 unique)
2024-08-20T23:49:47.2445004Z 
2024-08-20T23:49:49.2139538Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:49:49.2140045Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:49:49.2140515Z cuda train basic_gnn_gcn                      
2024-08-20T23:49:50.8704598Z W0820 23:49:50.869000 37347 torch/_logging/_internal.py:1058] [18/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:49:51.4942412Z pass
2024-08-20T23:49:51.5056638Z TIMING: entire_frame_compile:1.66284 backend_compile:0.69081 total_wall_time:1.66284
2024-08-20T23:49:51.5057806Z STATS: call_* op count: 98 | FakeTensorMode.__torch_dispatch__:2157 | FakeTensor.__torch_dispatch__:192 | ProxyTorchDispatchMode.__torch_dispatch__:866
2024-08-20T23:49:51.5058903Z Dynamo produced 10 graphs covering 98 ops with 13 graph breaks (6 unique)
2024-08-20T23:49:55.3344565Z 
2024-08-20T23:49:57.2853082Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:49:57.2853580Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:49:57.2854513Z cuda train basic_gnn_gin                      
2024-08-20T23:49:58.9457188Z W0820 23:49:58.944000 37437 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:50:00.0047506Z pass
2024-08-20T23:50:00.0135324Z TIMING: entire_frame_compile:2.12955 backend_compile:0.92972 total_wall_time:2.12955
2024-08-20T23:50:00.0137680Z STATS: call_* op count: 109 | FakeTensorMode.__torch_dispatch__:3059 | ProxyTorchDispatchMode.__torch_dispatch__:1395 | FakeTensor.__torch_dispatch__:248
2024-08-20T23:50:00.0139866Z Dynamo produced 3 graphs covering 109 ops with 7 graph breaks (5 unique)
2024-08-20T23:50:03.8107887Z 
2024-08-20T23:50:05.7205312Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:50:05.7206011Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:50:05.7206681Z cuda train basic_gnn_sage                     
2024-08-20T23:50:07.1968787Z W0820 23:50:07.195000 37527 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:50:08.0604193Z pass
2024-08-20T23:50:08.0669822Z TIMING: entire_frame_compile:1.76331 backend_compile:0.76894 total_wall_time:1.76331
2024-08-20T23:50:08.0672277Z STATS: call_* op count: 106 | FakeTensorMode.__torch_dispatch__:2513 | ProxyTorchDispatchMode.__torch_dispatch__:1102 | FakeTensor.__torch_dispatch__:194
2024-08-20T23:50:08.0674499Z Dynamo produced 3 graphs covering 106 ops with 7 graph breaks (5 unique)
2024-08-20T23:50:11.8465227Z 
2024-08-20T23:50:12.4350421Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:50:12.4350912Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:50:12.4351349Z cuda train dcgan                              
2024-08-20T23:50:15.7919978Z pass
2024-08-20T23:50:15.7936603Z TIMING: entire_frame_compile:3.05798 backend_compile:2.01673 total_wall_time:3.05798
2024-08-20T23:50:15.7938131Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:1977 | attempt fast:80 | fast is_contiguous:80 | FakeTensor.__torch_dispatch__:38 | ProxyTorchDispatchMode.__torch_dispatch__:427
2024-08-20T23:50:15.7939407Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique)
2024-08-20T23:50:19.5810629Z 
2024-08-20T23:50:23.2544067Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:50:23.2544549Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:50:23.2544996Z cuda train demucs                             
2024-08-20T23:51:22.9256993Z pass
2024-08-20T23:51:22.9485439Z TIMING: entire_frame_compile:5.06591 backend_compile:2.23098 total_wall_time:5.06591
2024-08-20T23:51:22.9487486Z STATS: call_* op count: 89 | FakeTensorMode.__torch_dispatch__:4933 | attempt fast:25 | fast is_contiguous:10 | slow no contiguity match:10 | slow both tensors nontrivially broadcast:5 | ProxyTorchDispatchMode.__torch_dispatch__:1789 | FakeTensor.__torch_dispatch__:61
2024-08-20T23:51:22.9489628Z Dynamo produced 6 graphs covering 89 ops with 9 graph breaks (6 unique)
2024-08-20T23:51:26.8568813Z 
2024-08-20T23:51:29.1326252Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:51:29.1326930Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:51:29.1327477Z cuda train densenet121                        
2024-08-20T23:52:47.1527544Z pass
2024-08-20T23:52:47.1649206Z TIMING: entire_frame_compile:72.19497 backend_compile:47.68525 total_wall_time:72.19497
2024-08-20T23:52:47.1651164Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:70067 | attempt fast:2842 | fast is_contiguous:2311 | ProxyTorchDispatchMode.__torch_dispatch__:18440 | slow no contiguity match:531 | FakeTensor.__torch_dispatch__:22
2024-08-20T23:52:47.1653022Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique)
2024-08-20T23:52:57.3891837Z 
2024-08-20T23:53:02.6869179Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:02.6869656Z loading model: 0it [00:05, ?it/s]
2024-08-20T23:53:02.6870102Z cuda train detectron2_maskrcnn_r_50_c4        
2024-08-20T23:53:02.6883392Z Traceback (most recent call last):
2024-08-20T23:53:02.6884164Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model
2024-08-20T23:53:02.6885298Z     self.model_iter_fn(model, example_inputs)
2024-08-20T23:53:02.6886306Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass
2024-08-20T23:53:02.6887283Z     pred = mod(*cloned_inputs)
2024-08-20T23:53:02.6888487Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:53:02.6889493Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:53:02.6890522Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:53:02.6891600Z     return forward_call(*args, **kwargs)
2024-08-20T23:53:02.6892954Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
2024-08-20T23:53:02.6894027Z     return self.inference(batched_inputs)
2024-08-20T23:53:02.6895018Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference
2024-08-20T23:53:02.6896014Z     results, _ = self.roi_heads(images, features, proposals, None)
2024-08-20T23:53:02.6897090Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:53:02.6897962Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:53:02.6898886Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:53:02.6899702Z     return forward_call(*args, **kwargs)
2024-08-20T23:53:02.6900685Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward
2024-08-20T23:53:02.6901589Z     box_features = self._shared_roi_transform(
2024-08-20T23:53:02.6902687Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform
2024-08-20T23:53:02.6903635Z     x = self.pooler(features, boxes)
2024-08-20T23:53:02.6904589Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:53:02.6905457Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:53:02.6906387Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:53:02.6907209Z     return forward_call(*args, **kwargs)
2024-08-20T23:53:02.6908112Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward
2024-08-20T23:53:02.6908980Z     return self.level_poolers[0](x[0], pooler_fmt_boxes)
2024-08-20T23:53:02.6910007Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
2024-08-20T23:53:02.6911159Z     return self._call_impl(*args, **kwargs)
2024-08-20T23:53:02.6912155Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
2024-08-20T23:53:02.6912965Z     return forward_call(*args, **kwargs)
2024-08-20T23:53:02.6913867Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward
2024-08-20T23:53:02.6914648Z     return roi_align(
2024-08-20T23:53:02.6915472Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2024-08-20T23:53:02.6916565Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2024-08-20T23:53:02.6917756Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align
2024-08-20T23:53:02.6918822Z     val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask)  # [K, C, PH, PW, IY, IX]
2024-08-20T23:53:02.6920053Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate
2024-08-20T23:53:02.6921007Z     v1 = masked_index(y_low, x_low)
2024-08-20T23:53:02.6921904Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index
2024-08-20T23:53:02.6922711Z     return input[
2024-08-20T23:53:02.6925787Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 181102 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-20T23:53:02.6928448Z 
2024-08-20T23:53:02.6928748Z The above exception was the direct cause of the following exception:
2024-08-20T23:53:02.6929203Z 
2024-08-20T23:53:02.6929348Z Traceback (most recent call last):
2024-08-20T23:53:02.6929991Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-20T23:53:02.6930637Z     ) = runner.load_model(
2024-08-20T23:53:02.6931373Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model
2024-08-20T23:53:02.6932126Z     self.validate_model(model, example_inputs)
2024-08-20T23:53:02.6932880Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model
2024-08-20T23:53:02.6933639Z     raise RuntimeError("Eager run failed") from e
2024-08-20T23:53:02.6934098Z RuntimeError: Eager run failed
2024-08-20T23:53:02.6934354Z 
2024-08-20T23:53:02.6934471Z eager_fail_to_run
2024-08-20T23:53:06.3997808Z 
2024-08-20T23:53:15.2962761Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:15.2963226Z loading model: 0it [00:08, ?it/s]
2024-08-20T23:53:18.0541246Z cuda train dlrm                               
2024-08-20T23:53:18.0541744Z pass
2024-08-20T23:53:18.0940760Z TIMING: entire_frame_compile:2.07461 backend_compile:1.34409 total_wall_time:2.07461
2024-08-20T23:53:18.0942140Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:2480 | ProxyTorchDispatchMode.__torch_dispatch__:1006 | FakeTensor.__torch_dispatch__:22
2024-08-20T23:53:18.0943244Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique)
2024-08-20T23:53:21.8487895Z 
2024-08-20T23:53:23.2424269Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:23.2424972Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:53:23.2425445Z cuda train drq                                
2024-08-20T23:53:25.4809196Z W0820 23:53:25.479000 38129 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:53:28.0199906Z pass
2024-08-20T23:53:28.0228780Z TIMING: entire_frame_compile:4.44117 backend_compile:2.61053 total_wall_time:4.44117
2024-08-20T23:53:28.0230081Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:5656 | ProxyTorchDispatchMode.__torch_dispatch__:2562 | FakeTensor.__torch_dispatch__:420
2024-08-20T23:53:28.0231181Z Dynamo produced 5 graphs covering 136 ops with 7 graph breaks (5 unique)
2024-08-20T23:53:31.9606887Z 
2024-08-20T23:53:33.7952619Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:33.7953497Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:53:33.7954373Z cuda train fastNLP_Bert                       
2024-08-20T23:53:39.6082870Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting:
2024-08-20T23:53:39.6084356Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     torch._dynamo.config.capture_scalar_outputs = True
2024-08-20T23:53:39.6085358Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] or:
2024-08-20T23:53:39.6086337Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2024-08-20T23:53:39.6087910Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph.
2024-08-20T23:53:39.6089157Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T23:53:39.6090078Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at:
2024-08-20T23:53:39.6091785Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438
2024-08-20T23:53:39.6093452Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     pred = mod(*cloned_inputs)
2024-08-20T23:53:39.6095192Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward
2024-08-20T23:53:39.6096619Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     sequence_output = self.bert(words)
2024-08-20T23:53:39.6098260Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward
2024-08-20T23:53:39.6099714Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     outputs = self.model(words)
2024-08-20T23:53:39.6101342Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward
2024-08-20T23:53:39.6103389Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0]     max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item()  # 表示word piece的长度(包括padding)
2024-08-20T23:53:39.6106464Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T23:53:39.6107194Z W0820 23:53:39.607000 38235 torch/_dynamo/variables/tensor.py:776] [3/0] 
2024-08-20T23:53:50.8549955Z pass
2024-08-20T23:53:50.9445328Z TIMING: entire_frame_compile:10.94352 backend_compile:6.94225 total_wall_time:10.94352
2024-08-20T23:53:50.9446528Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:25118 | ProxyTorchDispatchMode.__torch_dispatch__:13734 | FakeTensor.__torch_dispatch__:52
2024-08-20T23:53:50.9447652Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique)
2024-08-20T23:53:55.0361557Z 
2024-08-20T23:53:55.9455479Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:55.9455993Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:53:55.9456450Z cuda train functorch_dp_cifar10               
2024-08-20T23:54:09.0627192Z W0820 23:54:09.061000 38334 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:54:15.3654017Z pass
2024-08-20T23:54:15.3761651Z TIMING: entire_frame_compile:18.78756 backend_compile:11.91718 total_wall_time:18.78756
2024-08-20T23:54:15.3763424Z STATS: call_* op count: 276 | FakeTensorMode.__torch_dispatch__:18866 | attempt fast:852 | fast is_contiguous:832 | ProxyTorchDispatchMode.__torch_dispatch__:5405 | slow both tensors nontrivially broadcast:20 | FakeTensor.__torch_dispatch__:1208
2024-08-20T23:54:15.3764941Z Dynamo produced 3 graphs covering 276 ops with 7 graph breaks (5 unique)
2024-08-20T23:54:19.6948913Z 
2024-08-20T23:54:20.3352739Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:54:20.3353221Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:54:20.3353668Z cuda train functorch_maml_omniglot            
2024-08-20T23:54:22.2013420Z W0820 23:54:22.200000 38433 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:54:23.6122543Z pass
2024-08-20T23:54:23.6123559Z TIMING: entire_frame_compile:2.91759 backend_compile:1.70254 total_wall_time:2.91759
2024-08-20T23:54:23.6125291Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:2830 | ProxyTorchDispatchMode.__torch_dispatch__:1104 | FakeTensor.__torch_dispatch__:296
2024-08-20T23:54:23.6126898Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique)
2024-08-20T23:54:27.3877171Z 
2024-08-20T23:54:28.7638001Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:54:28.7638625Z loading model: 0it [00:01, ?it/s]
2024-08-20T23:54:28.7640617Z cuda train hf_Albert                          
2024-08-20T23:55:10.9225939Z W0820 23:55:10.921000 38530 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:55:13.2602468Z pass
2024-08-20T23:55:13.3164302Z TIMING: entire_frame_compile:37.6476 backend_compile:26.16289 total_wall_time:37.6476
2024-08-20T23:55:13.3165698Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:44180 | attempt fast:2272 | fast is_contiguous:2272 | ProxyTorchDispatchMode.__torch_dispatch__:15170 | FakeTensor.__torch_dispatch__:572
2024-08-20T23:55:13.3167223Z Dynamo produced 3 graphs covering 542 ops with 6 graph breaks (5 unique)
2024-08-20T23:55:18.7608783Z 
2024-08-20T23:55:22.7491084Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:55:22.7491549Z loading model: 0it [00:03, ?it/s]
2024-08-20T23:55:22.7491978Z cuda train hf_Bart                            
2024-08-20T23:56:13.6916550Z W0820 23:56:13.690000 38629 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:56:36.8587827Z pass
2024-08-20T23:56:36.9179502Z TIMING: entire_frame_compile:63.96421 backend_compile:39.1727 total_wall_time:63.96421
2024-08-20T23:56:36.9182747Z STATS: call_* op count: 1249 | FakeTensorMode.__torch_dispatch__:72424 | attempt fast:2296 | fast is_contiguous:2292 | slow no contiguity match:4 | FakeTensor.__torch_dispatch__:4980 | ProxyTorchDispatchMode.__torch_dispatch__:26116
2024-08-20T23:56:36.9184365Z Dynamo produced 3 graphs covering 1249 ops with 6 graph breaks (5 unique)
2024-08-20T23:56:42.9658210Z 
2024-08-20T23:56:45.5616961Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:56:45.5617503Z loading model: 0it [00:02, ?it/s]
2024-08-20T23:56:45.5618009Z cuda train hf_Bert                            
2024-08-20T23:57:30.7286910Z W0820 23:57:30.727000 38728 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-20T23:57:49.5060664Z pass
2024-08-20T23:57:49.5552382Z TIMING: entire_frame_compile:55.93915 backend_compile:33.13532 total_wall_time:55.93915
2024-08-20T23:57:49.5555590Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:60510 | attempt fast:2121 | fast is_contiguous:2121 | ProxyTorchDispatchMode.__torch_dispatch__:21388 | FakeTensor.__torch_dispatch__:3692
2024-08-20T23:57:49.5556905Z Dynamo produced 3 graphs covering 994 ops with 6 graph breaks (5 unique)
2024-08-20T23:57:55.4219225Z 
2024-08-20T23:58:00.2870525Z loading model: 0it [00:00, ?it/s]
2024-08-20T23:58:00.2871001Z loading model: 0it [00:04, ?it/s]
2024-08-20T23:58:00.2872885Z cuda train hf_Bert_large                      
2024-08-20T23:59:33.6332070Z W0820 23:59:33.632000 38827 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:00:10.3424569Z pass
2024-08-21T00:00:10.5181948Z TIMING: entire_frame_compile:109.8999 backend_compile:65.38806 total_wall_time:109.8999
2024-08-21T00:00:10.5183360Z STATS: call_* op count: 1918 | FakeTensorMode.__torch_dispatch__:118326 | attempt fast:4125 | fast is_contiguous:4125 | ProxyTorchDispatchMode.__torch_dispatch__:41896 | FakeTensor.__torch_dispatch__:7148
2024-08-21T00:00:10.5184683Z Dynamo produced 3 graphs covering 1918 ops with 6 graph breaks (5 unique)
2024-08-21T00:00:18.6558371Z 
2024-08-21T00:00:22.2794587Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:00:22.2795199Z loading model: 0it [00:03, ?it/s]
2024-08-21T00:00:22.2797752Z cuda train hf_BigBird                         
2024-08-21T00:00:33.5611344Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine
2024-08-21T00:01:15.8243504Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1] Error while creating guard:
2024-08-21T00:01:15.8244680Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1] Name: ''
2024-08-21T00:01:15.8246814Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Source: shape_env
2024-08-21T00:01:15.8247882Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Create Function: SHAPE_ENV
2024-08-21T00:01:15.8248845Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Guard Types: None
2024-08-21T00:01:15.8249606Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Code List: None
2024-08-21T00:01:15.8250587Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Object Weakref: None
2024-08-21T00:01:15.8251733Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     Guarded Class Weakref: None
2024-08-21T00:01:15.8252646Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1] Traceback (most recent call last):
2024-08-21T00:01:15.8254060Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-21T00:01:15.8255398Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     return self.create_fn(builder, self)
2024-08-21T00:01:15.8257120Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-21T00:01:15.8258598Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     guards = output_graph.shape_env.produce_guards(
2024-08-21T00:01:15.8260525Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-21T00:01:15.8262093Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]     raise ConstraintViolationError(
2024-08-21T00:01:15.8263872Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-21T00:01:15.8266214Z E0821 00:01:15.823000 38926 torch/_guards.py:282] [3/0_1]   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4).
2024-08-21T00:01:15.8268172Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1] Created at:
2024-08-21T00:01:15.8269529Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 603, in transform
2024-08-21T00:01:15.8271114Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]     tracer = InstructionTranslator(
2024-08-21T00:01:15.8272586Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2654, in __init__
2024-08-21T00:01:15.8273814Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]     output=OutputGraph(
2024-08-21T00:01:15.8275220Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 317, in __init__
2024-08-21T00:01:15.8276448Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]     self.init_ambient_guards()
2024-08-21T00:01:15.8277898Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 463, in init_ambient_guards
2024-08-21T00:01:15.8279390Z E0821 00:01:15.823000 38926 torch/_guards.py:284] [3/0_1]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
2024-08-21T00:01:15.8280365Z ERROR:common:
2024-08-21T00:01:15.8280676Z Traceback (most recent call last):
2024-08-21T00:01:15.8281366Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2819, in check_accuracy
2024-08-21T00:01:15.8282195Z     new_result = optimized_model_iter_fn(model_copy, example_inputs)
2024-08-21T00:01:15.8283172Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn
2024-08-21T00:01:15.8283922Z     return fn(*args, **kwargs)
2024-08-21T00:01:15.8284593Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-21T00:01:15.8285441Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-21T00:01:15.8286321Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 437, in forward_and_backward_pass
2024-08-21T00:01:15.8287132Z     cloned_inputs = clone_inputs(inputs)
2024-08-21T00:01:15.8288095Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 438, in torch_dynamo_resume_in_forward_and_backward_pass_at_437
2024-08-21T00:01:15.8289039Z     self.optimizer_zero_grad(mod)
2024-08-21T00:01:15.8289929Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__
2024-08-21T00:01:15.8290841Z     return self._torchdynamo_orig_callable(
2024-08-21T00:01:15.8291756Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1039, in __call__
2024-08-21T00:01:15.8292547Z     result = self._inner_convert(
2024-08-21T00:01:15.8293452Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__
2024-08-21T00:01:15.8294238Z     return _compile(
2024-08-21T00:01:15.8295049Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile
2024-08-21T00:01:15.8295969Z     guarded_code = compile_inner(code, one_graph, hooks, transform)
2024-08-21T00:01:15.8297028Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner
2024-08-21T00:01:15.8297941Z     return _compile_inner(code, one_graph, hooks, transform)
2024-08-21T00:01:15.8298928Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function
2024-08-21T00:01:15.8299731Z     return function(*args, **kwargs)
2024-08-21T00:01:15.8300652Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 783, in _compile_inner
2024-08-21T00:01:15.8301484Z     check_fn = CheckFunctionManager(
2024-08-21T00:01:15.8302345Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2200, in __init__
2024-08-21T00:01:15.8303097Z     guard.create(builder)
2024-08-21T00:01:15.8303946Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create
2024-08-21T00:01:15.8304725Z     return self.create_fn(builder, self)
2024-08-21T00:01:15.8305621Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV
2024-08-21T00:01:15.8306446Z     guards = output_graph.shape_env.produce_guards(
2024-08-21T00:01:15.8307521Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards
2024-08-21T00:01:15.8308437Z     raise ConstraintViolationError(
2024-08-21T00:01:15.8309614Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic".
2024-08-21T00:01:15.8311400Z   - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4).
2024-08-21T00:01:15.8312233Z 
2024-08-21T00:01:15.8312239Z 
2024-08-21T00:01:15.8312525Z You can suppress this exception and fall back to eager by setting:
2024-08-21T00:01:15.8313185Z     import torch._dynamo
2024-08-21T00:01:15.8313593Z     torch._dynamo.config.suppress_errors = True
2024-08-21T00:01:15.8313919Z 
2024-08-21T00:01:15.8314215Z TorchDynamo optimized model failed to run because of following error
2024-08-21T00:01:15.8314776Z fail_to_run
2024-08-21T00:01:15.8934970Z TIMING: entire_frame_compile:0.02399 backend_compile:25.35047 total_wall_time:0.02399
2024-08-21T00:01:15.8936676Z STATS: call_* op count: 3361 | FakeTensorMode.__torch_dispatch__:128049 | attempt fast:102 | slow both tensors nontrivially broadcast:2 | fast is_contiguous:100 | ProxyTorchDispatchMode.__torch_dispatch__:58466 | FakeTensor.__torch_dispatch__:4
2024-08-21T00:01:15.8938297Z Dynamo produced 1 graphs covering 3361 ops with 3 graph breaks (3 unique)
2024-08-21T00:01:21.0888016Z 
2024-08-21T00:01:23.2033556Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:01:23.2034464Z loading model: 0it [00:02, ?it/s]
2024-08-21T00:01:23.2035385Z cuda train hf_DistilBert                      
2024-08-21T00:01:47.4626765Z W0821 00:01:47.461000 39023 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:01:56.9286624Z pass
2024-08-21T00:01:56.9556750Z TIMING: entire_frame_compile:28.92237 backend_compile:18.04711 total_wall_time:28.92237
2024-08-21T00:01:56.9559800Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:32093 | attempt fast:1085 | fast is_contiguous:1061 | slow no contiguity match:24 | ProxyTorchDispatchMode.__torch_dispatch__:11112 | FakeTensor.__torch_dispatch__:1942
2024-08-21T00:01:56.9562713Z Dynamo produced 3 graphs covering 540 ops with 6 graph breaks (5 unique)
2024-08-21T00:02:01.6934661Z 
2024-08-21T00:02:04.9439019Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:02:04.9439490Z loading model: 0it [00:03, ?it/s]
2024-08-21T00:02:04.9439923Z cuda train hf_GPT2                            
2024-08-21T00:02:44.2737532Z W0821 00:02:44.272000 39122 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:02:57.7818712Z pass
2024-08-21T00:02:57.8808243Z TIMING: entire_frame_compile:47.78749 backend_compile:30.36379 total_wall_time:47.78749
2024-08-21T00:02:57.8810613Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:51246 | attempt fast:1931 | fast is_contiguous:1919 | ProxyTorchDispatchMode.__torch_dispatch__:17951 | slow no contiguity match:12 | FakeTensor.__torch_dispatch__:2768
2024-08-21T00:02:57.8812857Z Dynamo produced 3 graphs covering 1087 ops with 6 graph breaks (5 unique)
2024-08-21T00:03:03.3241207Z 
2024-08-21T00:03:16.7965770Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:03:16.7966854Z loading model: 0it [00:13, ?it/s]
2024-08-21T00:03:16.7967787Z cuda train hf_GPT2_large                      
2024-08-21T00:03:16.7968799Z pass_due_to_skip
2024-08-21T00:03:17.3007470Z TIMING:
2024-08-21T00:03:17.3007786Z STATS: call_* op count: 0
2024-08-21T00:03:17.3008632Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-21T00:03:20.8589230Z 
2024-08-21T00:03:22.5085016Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:03:22.5085560Z loading model: 0it [00:01, ?it/s]
2024-08-21T00:03:22.5086019Z cuda train hf_Reformer                        
2024-08-21T00:03:37.1744125Z W0821 00:03:37.173000 39264 torch/_logging/_internal.py:1058] [32/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:03:44.5766956Z pass
2024-08-21T00:03:44.6442709Z TIMING: entire_frame_compile:14.0116 backend_compile:6.68758 total_wall_time:14.0116
2024-08-21T00:03:44.6446269Z STATS: call_* op count: 688 | FakeTensorMode.__torch_dispatch__:19628 | ProxyTorchDispatchMode.__torch_dispatch__:7563 | attempt fast:43 | fast is_contiguous:37 | slow no contiguity match:6 | FakeTensor.__torch_dispatch__:1520
2024-08-21T00:03:44.6447715Z Dynamo produced 17 graphs covering 688 ops with 25 graph breaks (9 unique)
2024-08-21T00:03:48.8388006Z 
2024-08-21T00:03:53.1595111Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:03:53.1595640Z loading model: 0it [00:04, ?it/s]
2024-08-21T00:03:53.1596076Z cuda train hf_T5_base                         
2024-08-21T00:03:56.2914271Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine
2024-08-21T00:03:58.3998676Z ERROR:common:
2024-08-21T00:03:58.3999079Z Traceback (most recent call last):
2024-08-21T00:03:58.3999806Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2753, in check_accuracy
2024-08-21T00:03:58.4000570Z     correct_rerun_result = self.run_n_iterations(
2024-08-21T00:03:58.4004551Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations
2024-08-21T00:03:58.4005428Z     self.model_iter_fn(mod, inputs, collect_outputs=False)
2024-08-21T00:03:58.4006341Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass
2024-08-21T00:03:58.4007224Z     self.grad_scaler.scale(loss).backward()
2024-08-21T00:03:58.4008312Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward
2024-08-21T00:03:58.4009067Z     torch.autograd.backward(
2024-08-21T00:03:58.4009919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
2024-08-21T00:03:58.4010691Z     _engine_run_backward(
2024-08-21T00:03:58.4011698Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward
2024-08-21T00:03:58.4012850Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2024-08-21T00:03:58.4016340Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 182903 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
2024-08-21T00:03:58.4879427Z eager_2nd_run_OOM
2024-08-21T00:03:58.4909192Z TIMING:
2024-08-21T00:03:58.4909501Z STATS: call_* op count: 0
2024-08-21T00:03:58.4910054Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-21T00:04:02.1283362Z 
2024-08-21T00:04:12.4734956Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:12.4735433Z loading model: 0it [00:10, ?it/s]
2024-08-21T00:04:12.4735865Z cuda train hf_T5_large                        
2024-08-21T00:04:12.4739058Z pass_due_to_skip
2024-08-21T00:04:12.9459375Z TIMING:
2024-08-21T00:04:12.9459756Z STATS: call_* op count: 0
2024-08-21T00:04:12.9460442Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2024-08-21T00:04:16.5576686Z 
2024-08-21T00:04:17.7340914Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:17.7341449Z loading model: 0it [00:01, ?it/s]
2024-08-21T00:04:17.7341970Z cuda train hf_Whisper                         
2024-08-21T00:04:31.9386896Z W0821 00:04:31.937000 39449 torch/_logging/_internal.py:1058] [5/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:04:38.4094232Z pass
2024-08-21T00:04:38.4159686Z TIMING: entire_frame_compile:17.7283 backend_compile:10.58747 total_wall_time:17.7283
2024-08-21T00:04:38.4161191Z STATS: call_* op count: 346 | FakeTensorMode.__torch_dispatch__:19642 | attempt fast:657 | fast is_contiguous:620 | slow no contiguity match:37 | FakeTensor.__torch_dispatch__:1372 | ProxyTorchDispatchMode.__torch_dispatch__:6891
2024-08-21T00:04:38.4162625Z Dynamo produced 3 graphs covering 346 ops with 6 graph breaks (5 unique)
2024-08-21T00:04:42.7168675Z 
2024-08-21T00:04:43.0146907Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:43.0147436Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:43.0148223Z cuda train hf_distil_whisper                  
2024-08-21T00:04:43.0166164Z Traceback (most recent call last):
2024-08-21T00:04:43.0166866Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run
2024-08-21T00:04:43.0167819Z     ) = runner.load_model(
2024-08-21T00:04:43.0168490Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model
2024-08-21T00:04:43.0169213Z     benchmark = benchmark_cls(
2024-08-21T00:04:43.0169935Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__
2024-08-21T00:04:43.0171976Z     obj = type.__call__(cls, *args, **kwargs)
2024-08-21T00:04:43.0172912Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__
2024-08-21T00:04:43.0173920Z     raise NotImplementedError("Training is not implemented.")
2024-08-21T00:04:43.0174537Z NotImplementedError: Training is not implemented.
2024-08-21T00:04:43.0174906Z 
2024-08-21T00:04:43.0175036Z model_fail_to_load
2024-08-21T00:04:46.4480146Z 
2024-08-21T00:04:46.9264725Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:46.9265180Z loading model: 0it [00:00, ?it/s]
2024-08-21T00:04:46.9265631Z cuda train lennard_jones                      
2024-08-21T00:04:49.0076071Z W0821 00:04:49.006000 39582 torch/_logging/_internal.py:1058] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2024-08-21T00:04:50.0328208Z pass
2024-08-21T00:04:50.0328803Z TIMING: entire_frame_compile:2.8451 backend_compile:1.75914 total_wall_time:2.8451
2024-08-21T00:04:50.0330132Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:2580 | attempt fast:52 | fast is_contiguous:52 | ProxyTorchDispatchMode.__torch_dispatch__:921 | FakeTensor.__torch_dispatch__:224
2024-08-21T00:04:50.0331493Z Dynamo produced 3 graphs covering 60 ops with 7 graph breaks (5 unique)
2024-08-21T00:04:50.9842984Z accuracy            pass_rate=75.76%
2024-08-21T00:04:50.9843548Z calls_captured      gmean=0.00x mean=428.939x
2024-08-21T00:04:50.9845375Z unique_graphs       gmean=0.00x mean=3.212x
2024-08-21T00:04:50.9849322Z graph_breaks        gmean=0.00x mean=6.364x
2024-08-21T00:04:50.9853437Z unique_graph_breaks gmean=0.00x mean=4.212x
2024-08-21T00:04:50.9857351Z autograd_captures   gmean=0.00x mean=0.000x
2024-08-21T00:04:50.9861294Z autograd_compiles   gmean=0.00x mean=0.000x
2024-08-21T00:04:50.9865149Z cudagraph_skips     gmean=0.00x mean=0.000x
2024-08-21T00:04:51.5872996Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_training.csv
2024-08-21T00:04:51.9226847Z torchrec_dlrm                       XFAIL
2024-08-21T00:04:51.9230688Z BERT_pytorch                        PASS
2024-08-21T00:04:51.9235045Z Background_Matting                  XFAIL
2024-08-21T00:04:51.9240545Z LearningToPaint                     PASS
2024-08-21T00:04:51.9244920Z Super_SloMo                         PASS
2024-08-21T00:04:51.9249994Z alexnet                             PASS
2024-08-21T00:04:51.9254806Z basic_gnn_edgecnn                   PASS
2024-08-21T00:04:51.9259526Z basic_gnn_gcn                       PASS
2024-08-21T00:04:51.9264220Z basic_gnn_gin                       PASS
2024-08-21T00:04:51.9269146Z basic_gnn_sage                      PASS
2024-08-21T00:04:51.9274833Z dcgan                               PASS
2024-08-21T00:04:51.9279348Z demucs                              PASS
2024-08-21T00:04:51.9283583Z densenet121                         PASS
2024-08-21T00:04:51.9287838Z detectron2_maskrcnn_r_50_c4         XFAIL
2024-08-21T00:04:51.9292038Z dlrm                                PASS
2024-08-21T00:04:51.9296569Z drq                                 PASS
2024-08-21T00:04:51.9300830Z fastNLP_Bert                        PASS
2024-08-21T00:04:51.9305209Z functorch_dp_cifar10                PASS
2024-08-21T00:04:51.9309401Z functorch_maml_omniglot             PASS
2024-08-21T00:04:51.9313699Z hf_Albert                           PASS
2024-08-21T00:04:51.9318021Z hf_Bart                             PASS
2024-08-21T00:04:51.9322321Z hf_Bert                             PASS
2024-08-21T00:04:51.9326560Z hf_Bert_large                       PASS
2024-08-21T00:04:51.9330890Z hf_BigBird                          XFAIL
2024-08-21T00:04:51.9335371Z hf_DistilBert                       PASS
2024-08-21T00:04:51.9339567Z hf_GPT2                             PASS
2024-08-21T00:04:51.9343863Z hf_GPT2_large                       XFAIL
2024-08-21T00:04:51.9348099Z hf_Reformer                         PASS
2024-08-21T00:04:51.9352463Z hf_T5_base                          XFAIL
2024-08-21T00:04:51.9356720Z hf_T5_large                         XFAIL
2024-08-21T00:04:51.9361119Z hf_Whisper                          PASS
2024-08-21T00:04:51.9365376Z hf_distil_whisper                   XFAIL
2024-08-21T00:04:51.9370120Z lennard_jones                       PASS
2024-08-21T00:04:51.9891659Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_aot_eager_torchbench_training.csv
2024-08-21T00:04:52.3240081Z torchrec_dlrm                       PASS
2024-08-21T00:04:52.3243052Z BERT_pytorch                        PASS
2024-08-21T00:04:52.3247200Z Background_Matting                  PASS
2024-08-21T00:04:52.3251731Z LearningToPaint                     PASS
2024-08-21T00:04:52.3256472Z Super_SloMo                         PASS
2024-08-21T00:04:52.3261203Z alexnet                             PASS
2024-08-21T00:04:52.3265926Z basic_gnn_edgecnn                   PASS
2024-08-21T00:04:52.3271168Z basic_gnn_gcn                       PASS
2024-08-21T00:04:52.3275926Z basic_gnn_gin                       PASS
2024-08-21T00:04:52.3280562Z basic_gnn_sage                      PASS
2024-08-21T00:04:52.3285305Z dcgan                               PASS
2024-08-21T00:04:52.3289830Z demucs                              PASS
2024-08-21T00:04:52.3294968Z densenet121                         PASS
2024-08-21T00:04:52.3299800Z detectron2_maskrcnn_r_50_c4         PASS
2024-08-21T00:04:52.3304425Z dlrm                                PASS
2024-08-21T00:04:52.3309369Z drq                                 PASS
2024-08-21T00:04:52.3313748Z fastNLP_Bert                        PASS
2024-08-21T00:04:52.3318706Z functorch_dp_cifar10                PASS
2024-08-21T00:04:52.3323427Z functorch_maml_omniglot             PASS
2024-08-21T00:04:52.3328092Z hf_Albert                           PASS
2024-08-21T00:04:52.3332926Z hf_Bart                             PASS
2024-08-21T00:04:52.3337474Z hf_Bert                             PASS
2024-08-21T00:04:52.3342098Z hf_Bert_large                       PASS
2024-08-21T00:04:52.3346859Z hf_BigBird                          PASS
2024-08-21T00:04:52.3351553Z hf_DistilBert                       PASS
2024-08-21T00:04:52.3356601Z hf_GPT2                             PASS
2024-08-21T00:04:52.3360975Z hf_GPT2_large                       PASS
2024-08-21T00:04:52.3365575Z hf_Reformer                         PASS
2024-08-21T00:04:52.3371018Z hf_T5_base                          PASS
2024-08-21T00:04:52.3375890Z hf_T5_large                         PASS
2024-08-21T00:04:52.3380505Z hf_Whisper                          PASS
2024-08-21T00:04:52.3385121Z hf_distil_whisper                   PASS
2024-08-21T00:04:52.3390065Z lennard_jones                       PASS
2024-08-21T00:04:52.3912582Z + cleanup_workspace
2024-08-21T00:04:52.3913670Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-08-21T00:04:52.3914746Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-08-21T00:04:52.3915845Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-08-21T00:04:52.3916526Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-08-21T00:04:52.3917700Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-08-21T00:04:52.3918580Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-08-21T00:04:52.3919287Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-08-21T00:04:53.1657284Z ##[group]Run cat test/**/*_toprint.log || true
2024-08-21T00:04:53.1657800Z cat test/**/*_toprint.log || true
2024-08-21T00:04:53.1671435Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.1671944Z env:
2024-08-21T00:04:53.1672218Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.1672675Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.1673416Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.1674061Z ##[endgroup]
2024-08-21T00:04:53.1767485Z cat: 'test/**/*_toprint.log': No such file or directory
2024-08-21T00:04:53.1800135Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-08-21T00:04:53.1800612Z kill "$MONITOR_SCRIPT_PID"
2024-08-21T00:04:53.1808712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.1809215Z env:
2024-08-21T00:04:53.1809496Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.1809956Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.1810838Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.1811506Z   MONITOR_SCRIPT_PID: 139405
2024-08-21T00:04:53.1811868Z ##[endgroup]
2024-08-21T00:04:53.2013007Z Prepare all required actions
2024-08-21T00:04:53.2013526Z Getting action download info
2024-08-21T00:04:53.4273400Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
2024-08-21T00:04:53.6146935Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-08-21T00:04:53.6147590Z with:
2024-08-21T00:04:53.6148270Z   file-suffix: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882
2024-08-21T00:04:53.6149076Z   s3-bucket: gha-artifacts
2024-08-21T00:04:53.6149572Z env:
2024-08-21T00:04:53.6149958Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.6150506Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.6151440Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.6152184Z ##[endgroup]
2024-08-21T00:04:53.6200800Z ##[group]Run # Remove any previous test jsons if they exist
2024-08-21T00:04:53.6201445Z # Remove any previous test jsons if they exist
2024-08-21T00:04:53.6201964Z rm -f test-jsons-*.zip
2024-08-21T00:04:53.6202497Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2024-08-21T00:04:53.6213424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.6213933Z env:
2024-08-21T00:04:53.6214222Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.6214680Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.6215424Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.6216374Z   FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882
2024-08-21T00:04:53.6217065Z ##[endgroup]
2024-08-21T00:04:53.6505598Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2024-08-21T00:04:53.6542503Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2024-08-21T00:04:53.6547808Z   adding: test/minioptest_failures_dict.json (deflated 70%)
2024-08-21T00:04:53.6564263Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2024-08-21T00:04:53.6581373Z   adding: test/slow_tests.json (deflated 82%)
2024-08-21T00:04:53.6614488Z ##[group]Run # Remove any previous test reports if they exist
2024-08-21T00:04:53.6615133Z # Remove any previous test reports if they exist
2024-08-21T00:04:53.6615659Z rm -f test-reports-*.zip
2024-08-21T00:04:53.6616255Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2024-08-21T00:04:53.6625421Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.6625924Z env:
2024-08-21T00:04:53.6626205Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.6626663Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.6627395Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.6628330Z   FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882
2024-08-21T00:04:53.6629031Z ##[endgroup]
2024-08-21T00:04:53.6886484Z   adding: test/test-reports/inference_torchbench.csv (deflated 73%)
2024-08-21T00:04:53.6891479Z   adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%)
2024-08-21T00:04:53.6893771Z   adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%)
2024-08-21T00:04:53.6894987Z   adding: test/test-reports/training_torchbench.csv (deflated 64%)
2024-08-21T00:04:53.6901665Z   adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%)
2024-08-21T00:04:53.6903299Z   adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%)
2024-08-21T00:04:53.6935323Z ##[group]Run # Remove any previous usage logs if they exist
2024-08-21T00:04:53.6935950Z # Remove any previous usage logs if they exist
2024-08-21T00:04:53.6936452Z rm -f logs-*.zip
2024-08-21T00:04:53.6937297Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2024-08-21T00:04:53.6938084Z # so check to see if the file exists first
2024-08-21T00:04:53.6938583Z if [ -f 'usage_log.txt' ]; then
2024-08-21T00:04:53.6939115Z     zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'
2024-08-21T00:04:53.6939601Z fi
2024-08-21T00:04:53.6939968Z if ls test/**/*.log 1> /dev/null 2>&1; then
2024-08-21T00:04:53.6940539Z     zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log'
2024-08-21T00:04:53.6941037Z fi
2024-08-21T00:04:53.6949613Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.6950151Z env:
2024-08-21T00:04:53.6950436Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.6950919Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.6951742Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.6952805Z   FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882
2024-08-21T00:04:53.6953611Z ##[endgroup]
2024-08-21T00:04:53.7057918Z   adding: usage_log.txt (deflated 92%)
2024-08-21T00:04:53.7145702Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-08-21T00:04:53.7146410Z # Remove any previous debugging artifacts if they exist
2024-08-21T00:04:53.7146958Z rm -f debug-*.zip
2024-08-21T00:04:53.7147348Z if [ -d 'test/debug' ]; then
2024-08-21T00:04:53.7147855Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2024-08-21T00:04:53.7148345Z fi
2024-08-21T00:04:53.7156483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:53.7156996Z env:
2024-08-21T00:04:53.7157283Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.7157746Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.7158488Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.7159444Z   FILE_SUFFIX: test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882
2024-08-21T00:04:53.7160131Z ##[endgroup]
2024-08-21T00:04:53.7253795Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-21T00:04:53.7254239Z with:
2024-08-21T00:04:53.7254526Z   s3-bucket: gha-artifacts
2024-08-21T00:04:53.7254962Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-21T00:04:53.7255431Z   retention-days: 14
2024-08-21T00:04:53.7255758Z   if-no-files-found: warn
2024-08-21T00:04:53.7256116Z   path: test-jsons-*.zip
2024-08-21T00:04:53.7256655Z   name: artifact
2024-08-21T00:04:53.7256966Z   region: us-east-1
2024-08-21T00:04:53.7257268Z env:
2024-08-21T00:04:53.7257556Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:53.7258015Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:53.7258757Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:53.7259472Z ##[endgroup]
2024-08-21T00:04:54.0991297Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-21T00:04:54.0991954Z With the provided path, there will be 1 file uploaded
2024-08-21T00:04:54.0992665Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-21T00:04:54.1050434Z Starting upload of test-jsons-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:54.2318706Z Finished upload of test-jsons-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:54.2600488Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-21T00:04:54.2600953Z with:
2024-08-21T00:04:54.2601245Z   s3-bucket: gha-artifacts
2024-08-21T00:04:54.2601690Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-21T00:04:54.2602174Z   retention-days: 14
2024-08-21T00:04:54.2602513Z   if-no-files-found: error
2024-08-21T00:04:54.2602888Z   path: test-reports-*.zip
2024-08-21T00:04:54.2603250Z   name: artifact
2024-08-21T00:04:54.2603556Z   region: us-east-1
2024-08-21T00:04:54.2603865Z env:
2024-08-21T00:04:54.2604341Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:54.2604805Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:54.2605545Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:54.2606198Z ##[endgroup]
2024-08-21T00:04:54.6189353Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-21T00:04:54.6190232Z With the provided path, there will be 1 file uploaded
2024-08-21T00:04:54.6191043Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-21T00:04:54.6248011Z Starting upload of test-reports-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:54.7448142Z Finished upload of test-reports-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:54.7734036Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-21T00:04:54.7734486Z with:
2024-08-21T00:04:54.7734778Z   s3-bucket: gha-artifacts
2024-08-21T00:04:54.7735230Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-21T00:04:54.7735705Z   retention-days: 14
2024-08-21T00:04:54.7736040Z   if-no-files-found: ignore
2024-08-21T00:04:54.7736404Z   path: logs-*.zip
2024-08-21T00:04:54.7736715Z   name: artifact
2024-08-21T00:04:54.7737018Z   region: us-east-1
2024-08-21T00:04:54.7737329Z env:
2024-08-21T00:04:54.7737615Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:54.7738076Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:54.7738818Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:54.7739474Z ##[endgroup]
2024-08-21T00:04:55.1288637Z NOTE: s3-prefix specified, ignoring name parameter
2024-08-21T00:04:55.1289231Z With the provided path, there will be 1 file uploaded
2024-08-21T00:04:55.1289847Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact
2024-08-21T00:04:55.1345582Z Starting upload of logs-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:55.2559198Z Finished upload of logs-test-dynamic_aot_eager_torchbench-1-2-linux.g5.4xlarge.nvidia.gpu_29026299882.zip
2024-08-21T00:04:55.2838838Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-08-21T00:04:55.2839288Z with:
2024-08-21T00:04:55.2839577Z   s3-bucket: gha-artifacts
2024-08-21T00:04:55.2840023Z   s3-prefix: pytorch/pytorch/10479310721/1/artifact

2024-08-21T00:04:55.2840642Z   retention-days: 14
2024-08-21T00:04:55.2840998Z   if-no-files-found: ignore
2024-08-21T00:04:55.2841578Z   path: debug-*.zip
2024-08-21T00:04:55.2841907Z   name: artifact
2024-08-21T00:04:55.2842222Z   region: us-east-1
2024-08-21T00:04:55.2842543Z env:
2024-08-21T00:04:55.2842839Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:55.2843299Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:55.2844039Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:55.2844696Z ##[endgroup]
2024-08-21T00:04:55.6307412Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-08-21T00:04:55.6588081Z ##[group]Run # shellcheck disable=SC2156
2024-08-21T00:04:55.6588573Z # shellcheck disable=SC2156
2024-08-21T00:04:55.6589367Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2024-08-21T00:04:55.6598485Z shell: /usr/bin/bash -e {0}
2024-08-21T00:04:55.6598847Z env:
2024-08-21T00:04:55.6599130Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:55.6599603Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:55.6600339Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:55.6600988Z ##[endgroup]
2024-08-21T00:04:55.9291974Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2024-08-21T00:04:55.9292537Z with:
2024-08-21T00:04:55.9292817Z env:
2024-08-21T00:04:55.9293191Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:55.9293780Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:55.9294726Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:55.9295371Z ##[endgroup]
2024-08-21T00:04:55.9314732Z ##[group]Run set -eou pipefail
2024-08-21T00:04:55.9315139Z set -eou pipefail
2024-08-21T00:04:55.9315489Z 
2024-08-21T00:04:55.9316004Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2024-08-21T00:04:55.9316634Z for _ in $(seq 1440); do
2024-08-21T00:04:55.9317123Z     # Break if no ssh session exists anymore
2024-08-21T00:04:55.9317622Z     if [ "$(who)" = "" ]; then
2024-08-21T00:04:55.9318033Z       break
2024-08-21T00:04:55.9318375Z     fi
2024-08-21T00:04:55.9318671Z     echo "."
2024-08-21T00:04:55.9318999Z     sleep 5
2024-08-21T00:04:55.9319314Z done
2024-08-21T00:04:55.9328076Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:55.9328578Z env:
2024-08-21T00:04:55.9328862Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:55.9329322Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:55.9330060Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:55.9330765Z ##[endgroup]
2024-08-21T00:04:55.9361052Z Holding runner for 2 hours until all ssh sessions have logged out
2024-08-21T00:04:55.9431372Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-08-21T00:04:55.9432145Z # ignore expansion of "docker ps -q" since it could be empty
2024-08-21T00:04:55.9432741Z # shellcheck disable=SC2046
2024-08-21T00:04:55.9433221Z docker stop $(docker ps -q) || true
2024-08-21T00:04:55.9433704Z # Prune all of the docker images
2024-08-21T00:04:55.9434214Z docker system prune -af
2024-08-21T00:04:55.9442373Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:04:55.9442889Z env:
2024-08-21T00:04:55.9443168Z   GIT_DEFAULT_BRANCH: main
2024-08-21T00:04:55.9443635Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-08-21T00:04:55.9444436Z   DOCKER_CONTAINER_ID: 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:04:55.9445092Z ##[endgroup]
2024-08-21T00:04:56.7030121Z 55aa4f098e1d
2024-08-21T00:05:00.6730830Z Deleted Containers:
2024-08-21T00:05:00.6731384Z 55aa4f098e1dd063e36883ff3da892cfe7436558eabafe00f5544c9d098fb271
2024-08-21T00:05:00.6732166Z 
2024-08-21T00:05:09.4290320Z Deleted Images:
2024-08-21T00:05:09.4292441Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9
2024-08-21T00:05:09.4295013Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402
2024-08-21T00:05:09.4296532Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb
2024-08-21T00:05:09.4297382Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d
2024-08-21T00:05:09.4298232Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5
2024-08-21T00:05:09.4299050Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61
2024-08-21T00:05:09.4299865Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6
2024-08-21T00:05:09.4300684Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c
2024-08-21T00:05:09.4301526Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411
2024-08-21T00:05:09.4302674Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967
2024-08-21T00:05:09.4303495Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c
2024-08-21T00:05:09.4304332Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90
2024-08-21T00:05:09.4305150Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08
2024-08-21T00:05:09.4305953Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9
2024-08-21T00:05:09.4306757Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34
2024-08-21T00:05:09.4307570Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3
2024-08-21T00:05:09.4308386Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1
2024-08-21T00:05:09.4309218Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497
2024-08-21T00:05:09.4310046Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62
2024-08-21T00:05:09.4310861Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8
2024-08-21T00:05:09.4311677Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf
2024-08-21T00:05:09.4312490Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e
2024-08-21T00:05:09.4313307Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688
2024-08-21T00:05:09.4314134Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5
2024-08-21T00:05:09.4314952Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0
2024-08-21T00:05:09.4315762Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd
2024-08-21T00:05:09.4316577Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453
2024-08-21T00:05:09.4317415Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef
2024-08-21T00:05:09.4318276Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04
2024-08-21T00:05:09.4319102Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251
2024-08-21T00:05:09.4319922Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475
2024-08-21T00:05:09.4320729Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36
2024-08-21T00:05:09.4321545Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a
2024-08-21T00:05:09.4322352Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0
2024-08-21T00:05:09.4323161Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a
2024-08-21T00:05:09.4324153Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506
2024-08-21T00:05:09.4324970Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e
2024-08-21T00:05:09.4325786Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7
2024-08-21T00:05:09.4326597Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924
2024-08-21T00:05:09.4327515Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a
2024-08-21T00:05:09.4328562Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf
2024-08-21T00:05:09.4329381Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc
2024-08-21T00:05:09.4330200Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9
2024-08-21T00:05:09.4331171Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40
2024-08-21T00:05:09.4331988Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59
2024-08-21T00:05:09.4332787Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a
2024-08-21T00:05:09.4333737Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328
2024-08-21T00:05:09.4334571Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0
2024-08-21T00:05:09.4335381Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc
2024-08-21T00:05:09.4336170Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d
2024-08-21T00:05:09.4336969Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579
2024-08-21T00:05:09.4337837Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013
2024-08-21T00:05:09.4338647Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf
2024-08-21T00:05:09.4339446Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b
2024-08-21T00:05:09.4340247Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd
2024-08-21T00:05:09.4341066Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69
2024-08-21T00:05:09.4341920Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d
2024-08-21T00:05:09.4342741Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f
2024-08-21T00:05:09.4343549Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e
2024-08-21T00:05:09.4344358Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7
2024-08-21T00:05:09.4345164Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34
2024-08-21T00:05:09.4345965Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e
2024-08-21T00:05:09.4346789Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98
2024-08-21T00:05:09.4347628Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad
2024-08-21T00:05:09.4348457Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1
2024-08-21T00:05:09.4349261Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144
2024-08-21T00:05:09.4350072Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb
2024-08-21T00:05:09.4350891Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597
2024-08-21T00:05:09.4351702Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8
2024-08-21T00:05:09.4352517Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83
2024-08-21T00:05:09.4353669Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8
2024-08-21T00:05:09.4354479Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00
2024-08-21T00:05:09.4355372Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f
2024-08-21T00:05:09.4356190Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64
2024-08-21T00:05:09.4357024Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88
2024-08-21T00:05:09.4357847Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d
2024-08-21T00:05:09.4358678Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25
2024-08-21T00:05:09.4359506Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9
2024-08-21T00:05:09.4360339Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29
2024-08-21T00:05:09.4361169Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037
2024-08-21T00:05:09.4361980Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9
2024-08-21T00:05:09.4362463Z 
2024-08-21T00:05:09.4362629Z Total reclaimed space: 49.69GB
2024-08-21T00:05:09.4429946Z Post job cleanup.
2024-08-21T00:05:09.4480342Z Post job cleanup.
2024-08-21T00:05:09.5416382Z [command]/usr/bin/git version
2024-08-21T00:05:09.5474723Z git version 2.40.1
2024-08-21T00:05:09.5512721Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2be9f182-5a97-4ee1-abfd-6210f47d0816' before making global git config changes
2024-08-21T00:05:09.5513953Z Adding repository directory to the temporary git global config as a safe directory
2024-08-21T00:05:09.5517866Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-08-21T00:05:09.5567695Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-08-21T00:05:09.5614827Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-08-21T00:05:09.6005600Z Entering 'android/libs/fbjni'
2024-08-21T00:05:09.6076536Z Entering 'third_party/FP16'
2024-08-21T00:05:09.6146686Z Entering 'third_party/FXdiv'
2024-08-21T00:05:09.6216401Z Entering 'third_party/NNPACK'
2024-08-21T00:05:09.6290727Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-21T00:05:09.6360389Z Entering 'third_party/XNNPACK'
2024-08-21T00:05:09.6450032Z Entering 'third_party/benchmark'
2024-08-21T00:05:09.6521163Z Entering 'third_party/cpp-httplib'
2024-08-21T00:05:09.6592712Z Entering 'third_party/cpuinfo'
2024-08-21T00:05:09.6660842Z Entering 'third_party/cudnn_frontend'
2024-08-21T00:05:09.6729583Z Entering 'third_party/cutlass'
2024-08-21T00:05:09.6804764Z Entering 'third_party/eigen'
2024-08-21T00:05:09.6876519Z Entering 'third_party/fbgemm'
2024-08-21T00:05:09.6943604Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-21T00:05:09.7011178Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-21T00:05:09.7078621Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-21T00:05:09.7149649Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-21T00:05:09.7216015Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-21T00:05:09.7284675Z Entering 'third_party/flatbuffers'
2024-08-21T00:05:09.7356328Z Entering 'third_party/fmt'
2024-08-21T00:05:09.7424741Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-21T00:05:09.7493130Z Entering 'third_party/gloo'
2024-08-21T00:05:09.7561458Z Entering 'third_party/googletest'
2024-08-21T00:05:09.7629780Z Entering 'third_party/ideep'
2024-08-21T00:05:09.7696815Z Entering 'third_party/ideep/mkl-dnn'
2024-08-21T00:05:09.7769766Z Entering 'third_party/ittapi'
2024-08-21T00:05:09.7837590Z Entering 'third_party/kineto'
2024-08-21T00:05:09.7904223Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-21T00:05:09.7969439Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-21T00:05:09.8041766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-21T00:05:09.8115816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-21T00:05:09.8182262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-21T00:05:09.8246838Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-21T00:05:09.8317869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-21T00:05:09.8385927Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-21T00:05:09.8453207Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-21T00:05:09.8521316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-21T00:05:09.8590876Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-21T00:05:09.8655981Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-21T00:05:09.8725563Z Entering 'third_party/mimalloc'
2024-08-21T00:05:09.8795335Z Entering 'third_party/nccl/nccl'
2024-08-21T00:05:09.8865285Z Entering 'third_party/nlohmann'
2024-08-21T00:05:09.8936213Z Entering 'third_party/onnx'
2024-08-21T00:05:09.9018585Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-21T00:05:09.9087759Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-21T00:05:09.9164515Z Entering 'third_party/opentelemetry-cpp'
2024-08-21T00:05:09.9233466Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-21T00:05:09.9301592Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-21T00:05:09.9366571Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-21T00:05:09.9432173Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-21T00:05:09.9499713Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-21T00:05:09.9564847Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-21T00:05:09.9632044Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-21T00:05:09.9696616Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-21T00:05:09.9765826Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-21T00:05:09.9836038Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-21T00:05:09.9924671Z Entering 'third_party/pocketfft'
2024-08-21T00:05:09.9993435Z Entering 'third_party/protobuf'
2024-08-21T00:05:10.0063888Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-21T00:05:10.0130532Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-21T00:05:10.0203426Z Entering 'third_party/psimd'
2024-08-21T00:05:10.0272696Z Entering 'third_party/pthreadpool'
2024-08-21T00:05:10.0342108Z Entering 'third_party/pybind11'
2024-08-21T00:05:10.0411643Z Entering 'third_party/python-peachpy'
2024-08-21T00:05:10.0481109Z Entering 'third_party/sleef'
2024-08-21T00:05:10.0550051Z Entering 'third_party/tensorpipe'
2024-08-21T00:05:10.0619434Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-21T00:05:10.0686492Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-21T00:05:10.0751515Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-21T00:05:10.0819332Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-21T00:05:10.0887415Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-21T00:05:10.0982926Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-08-21T00:05:10.1019052Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1028790Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-08-21T00:05:10.1076183Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-08-21T00:05:10.1442285Z Entering 'android/libs/fbjni'
2024-08-21T00:05:10.1486368Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1530454Z Entering 'third_party/FP16'
2024-08-21T00:05:10.1574426Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1618336Z Entering 'third_party/FXdiv'
2024-08-21T00:05:10.1662379Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1706636Z Entering 'third_party/NNPACK'
2024-08-21T00:05:10.1750648Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1795148Z Entering 'third_party/VulkanMemoryAllocator'
2024-08-21T00:05:10.1839536Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1886332Z Entering 'third_party/XNNPACK'
2024-08-21T00:05:10.1929944Z http.https://github.com/.extraheader
2024-08-21T00:05:10.1990697Z Entering 'third_party/benchmark'
2024-08-21T00:05:10.2034551Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2079881Z Entering 'third_party/cpp-httplib'
2024-08-21T00:05:10.2123879Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2167913Z Entering 'third_party/cpuinfo'
2024-08-21T00:05:10.2214080Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2259442Z Entering 'third_party/cudnn_frontend'
2024-08-21T00:05:10.2304123Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2348155Z Entering 'third_party/cutlass'
2024-08-21T00:05:10.2392587Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2444407Z Entering 'third_party/eigen'
2024-08-21T00:05:10.2488521Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2534155Z Entering 'third_party/fbgemm'
2024-08-21T00:05:10.2578359Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2621846Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-08-21T00:05:10.2664304Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2708582Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-08-21T00:05:10.2755551Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2798448Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-08-21T00:05:10.2840983Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2891552Z Entering 'third_party/fbgemm/third_party/googletest'
2024-08-21T00:05:10.2934399Z http.https://github.com/.extraheader
2024-08-21T00:05:10.2977134Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-08-21T00:05:10.3019679Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3066497Z Entering 'third_party/flatbuffers'
2024-08-21T00:05:10.3111581Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3158191Z Entering 'third_party/fmt'
2024-08-21T00:05:10.3203180Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3248338Z Entering 'third_party/gemmlowp/gemmlowp'
2024-08-21T00:05:10.3293068Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3337809Z Entering 'third_party/gloo'
2024-08-21T00:05:10.3382603Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3426182Z Entering 'third_party/googletest'
2024-08-21T00:05:10.3470508Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3514338Z Entering 'third_party/ideep'
2024-08-21T00:05:10.3557574Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3601818Z Entering 'third_party/ideep/mkl-dnn'
2024-08-21T00:05:10.3643778Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3700832Z Entering 'third_party/ittapi'
2024-08-21T00:05:10.3745131Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3789192Z Entering 'third_party/kineto'
2024-08-21T00:05:10.3833183Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3876721Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-08-21T00:05:10.3920211Z http.https://github.com/.extraheader
2024-08-21T00:05:10.3962419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-08-21T00:05:10.4006047Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4051393Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-08-21T00:05:10.4094727Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4138706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-08-21T00:05:10.4182461Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4227144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-08-21T00:05:10.4270312Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4312674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-08-21T00:05:10.4356287Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4404174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-08-21T00:05:10.4446987Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4492424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-08-21T00:05:10.4535251Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4580192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-08-21T00:05:10.4622954Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4668787Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-08-21T00:05:10.4711996Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4759521Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-08-21T00:05:10.4806246Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4849134Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-08-21T00:05:10.4891687Z http.https://github.com/.extraheader
2024-08-21T00:05:10.4937674Z Entering 'third_party/mimalloc'
2024-08-21T00:05:10.4982677Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5027057Z Entering 'third_party/nccl/nccl'
2024-08-21T00:05:10.5072621Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5117429Z Entering 'third_party/nlohmann'
2024-08-21T00:05:10.5160958Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5206414Z Entering 'third_party/onnx'
2024-08-21T00:05:10.5250424Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5312110Z Entering 'third_party/onnx/third_party/benchmark'
2024-08-21T00:05:10.5357424Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5400966Z Entering 'third_party/onnx/third_party/pybind11'
2024-08-21T00:05:10.5446634Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5495312Z Entering 'third_party/opentelemetry-cpp'
2024-08-21T00:05:10.5539285Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5584261Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-08-21T00:05:10.5626439Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5669333Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-08-21T00:05:10.5711405Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5755335Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-08-21T00:05:10.5798945Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5841990Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-08-21T00:05:10.5884471Z http.https://github.com/.extraheader
2024-08-21T00:05:10.5928132Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-08-21T00:05:10.5971672Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6014173Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-08-21T00:05:10.6056228Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6101640Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-08-21T00:05:10.6143993Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6185687Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-08-21T00:05:10.6228742Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6274925Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-08-21T00:05:10.6317350Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6364128Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-08-21T00:05:10.6408546Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6473250Z Entering 'third_party/pocketfft'
2024-08-21T00:05:10.6517157Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6564242Z Entering 'third_party/protobuf'
2024-08-21T00:05:10.6608118Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6653564Z Entering 'third_party/protobuf/third_party/benchmark'
2024-08-21T00:05:10.6696355Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6738809Z Entering 'third_party/protobuf/third_party/googletest'
2024-08-21T00:05:10.6783045Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6829129Z Entering 'third_party/psimd'
2024-08-21T00:05:10.6872600Z http.https://github.com/.extraheader
2024-08-21T00:05:10.6915905Z Entering 'third_party/pthreadpool'
2024-08-21T00:05:10.6958781Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7002354Z Entering 'third_party/pybind11'
2024-08-21T00:05:10.7045489Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7089153Z Entering 'third_party/python-peachpy'
2024-08-21T00:05:10.7132308Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7176790Z Entering 'third_party/sleef'
2024-08-21T00:05:10.7218882Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7263982Z Entering 'third_party/tensorpipe'
2024-08-21T00:05:10.7305809Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7348648Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-08-21T00:05:10.7391463Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7435367Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-08-21T00:05:10.7478692Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7520970Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-08-21T00:05:10.7563012Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7606598Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-08-21T00:05:10.7653541Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7695013Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-08-21T00:05:10.7737280Z http.https://github.com/.extraheader
2024-08-21T00:05:10.7896814Z A job completed hook has been configured by the self-hosted runner administrator
2024-08-21T00:05:10.7920181Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2024-08-21T00:05:10.7928205Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-08-21T00:05:10.7928732Z ##[endgroup]
2024-08-21T00:05:18.0388450Z Cleaning up orphan processes