2024-08-20T22:09:51.2167044Z Current runner version: '2.319.1' 2024-08-20T22:09:51.2173333Z Runner name: 'i-07cd86c6b6ea02560' 2024-08-20T22:09:51.2174148Z Runner group name: 'Default' 2024-08-20T22:09:51.2175085Z Machine name: 'ip-10-0-44-42' 2024-08-20T22:09:51.2191788Z Testing runner upgrade compatibility 2024-08-20T22:09:51.3574721Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:51.3576869Z Actions: read 2024-08-20T22:09:51.3577388Z Attestations: read 2024-08-20T22:09:51.3577830Z Checks: read 2024-08-20T22:09:51.3578373Z Contents: read 2024-08-20T22:09:51.3578871Z Deployments: read 2024-08-20T22:09:51.3579406Z Discussions: read 2024-08-20T22:09:51.3579860Z Issues: read 2024-08-20T22:09:51.3580307Z Metadata: read 2024-08-20T22:09:51.3580822Z Packages: read 2024-08-20T22:09:51.3581269Z Pages: read 2024-08-20T22:09:51.3581720Z PullRequests: read 2024-08-20T22:09:51.3582285Z RepositoryProjects: read 2024-08-20T22:09:51.3582852Z SecurityEvents: read 2024-08-20T22:09:51.3583311Z Statuses: read 2024-08-20T22:09:51.3584046Z ##[endgroup] 2024-08-20T22:09:51.3587299Z Secret source: Actions 2024-08-20T22:09:51.3588176Z Prepare workflow directory 2024-08-20T22:09:51.8937801Z Prepare all required actions 2024-08-20T22:09:51.9119299Z Getting action download info 2024-08-20T22:09:52.0842248Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:52.5604832Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:09:56.6279227Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:09:56.8196341Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:09:57.0820589Z Getting action download info 2024-08-20T22:09:57.1732496Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:09:57.3476804Z Getting action download info 2024-08-20T22:09:57.4312114Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:09:57.5827656Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:09:57.5830300Z ##[group] Inputs 2024-08-20T22:09:57.5830849Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:09:57.5842192Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:09:57.5852726Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:09:57.5853919Z sync-tag: 2024-08-20T22:09:57.5854750Z timeout-minutes: 240 2024-08-20T22:09:57.5855102Z use-gha: 2024-08-20T22:09:57.5855410Z dashboard-tag: 2024-08-20T22:09:57.5855738Z s3-bucket: gha-artifacts 2024-08-20T22:09:57.5856109Z aws-role-to-assume: 2024-08-20T22:09:57.5856445Z ##[endgroup] 2024-08-20T22:09:57.5857358Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:09:57.6526586Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:09:57.6667095Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:09:57.6679016Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:09:57.6679557Z ##[endgroup] 2024-08-20T22:09:59.5235688Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:09:59.5236274Z Instance Type: g5.4xlarge 2024-08-20T22:09:59.5237013Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:09:59.5237562Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:05.3372770Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:05.3373333Z with: 2024-08-20T22:10:05.3374068Z github-secret: *** 2024-08-20T22:10:05.3375037Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:05.3376070Z activate-with-label: false 2024-08-20T22:10:05.3376451Z label: with-ssh 2024-08-20T22:10:05.3376796Z remove-existing-keys: true 2024-08-20T22:10:05.3377186Z fail-silently: true 2024-08-20T22:10:05.3377517Z env: 2024-08-20T22:10:05.3377811Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:05.3378175Z ##[endgroup] 2024-08-20T22:10:05.4277956Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:05.4280600Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:05.7364853Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:05.8115034Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:05.8115664Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:05.8855476Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:05.8869568Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:05.8893662Z Login using: ssh ec2-user@ec2-44-201-194-2.compute-1.amazonaws.com 2024-08-20T22:10:05.8894448Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:05.8895299Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:05.9028206Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:05.9029123Z with: 2024-08-20T22:10:05.9029424Z submodules: recursive 2024-08-20T22:10:05.9029771Z fetch-depth: 0 2024-08-20T22:10:05.9030072Z env: 2024-08-20T22:10:05.9030357Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:05.9030712Z ##[endgroup] 2024-08-20T22:10:05.9220771Z ##[group]Run retry () { 2024-08-20T22:10:05.9221177Z retry () { 2024-08-20T22:10:05.9221913Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:05.9222500Z } 2024-08-20T22:10:05.9222832Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:05.9223281Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:05.9223784Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:05.9224264Z else 2024-08-20T22:10:05.9224631Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:05.9225084Z fi 2024-08-20T22:10:05.9225450Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:05.9235189Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:05.9235723Z env: 2024-08-20T22:10:05.9236020Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:05.9236377Z NO_SUDO: 2024-08-20T22:10:05.9236674Z ##[endgroup] 2024-08-20T22:10:05.9267248Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:10.5885775Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:10.5886232Z with: 2024-08-20T22:10:10.5886571Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:10.5887030Z fetch-depth: 0 2024-08-20T22:10:10.5887352Z submodules: recursive 2024-08-20T22:10:10.5887701Z quiet-checkout: true 2024-08-20T22:10:10.5888061Z repository: pytorch/pytorch 2024-08-20T22:10:10.5888555Z token: *** 2024-08-20T22:10:10.5888856Z ssh-strict: true 2024-08-20T22:10:10.5889195Z persist-credentials: true 2024-08-20T22:10:10.5889565Z clean: true 2024-08-20T22:10:10.5889915Z sparse-checkout-cone-mode: true 2024-08-20T22:10:10.5890315Z lfs: false 2024-08-20T22:10:10.5890621Z set-safe-directory: true 2024-08-20T22:10:10.5890974Z env: 2024-08-20T22:10:10.5891266Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:10.5891616Z ##[endgroup] 2024-08-20T22:10:10.6880627Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:10.6882388Z ##[group]Getting Git version info 2024-08-20T22:10:10.6883224Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:10.6884151Z [command]/usr/bin/git version 2024-08-20T22:10:10.6884550Z git version 2.40.1 2024-08-20T22:10:10.6898874Z ##[endgroup] 2024-08-20T22:10:10.6915535Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/96d98c88-b57b-4dfd-b0b1-10d3e2516af0' before making global git config changes 2024-08-20T22:10:10.6916931Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:10.6920921Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:10.6971969Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:10.6976769Z ##[group]Initializing the repository 2024-08-20T22:10:10.6979965Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:10.7023064Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:10.7024301Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:10.7025459Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:10.7026253Z hint: 2024-08-20T22:10:10.7027127Z hint: git config --global init.defaultBranch 2024-08-20T22:10:10.7027780Z hint: 2024-08-20T22:10:10.7028525Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:10.7030082Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:10.7031280Z hint: 2024-08-20T22:10:10.7031760Z hint: git branch -m 2024-08-20T22:10:10.7032628Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:10.7039566Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:10.7079322Z ##[endgroup] 2024-08-20T22:10:10.7080004Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:10.7082910Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:10.7119869Z ##[endgroup] 2024-08-20T22:10:10.7120473Z ##[group]Setting up auth 2024-08-20T22:10:10.7125614Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:10.7170486Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:10.7502664Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:10.7540771Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:10.7896221Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:10.7958757Z ##[endgroup] 2024-08-20T22:10:10.7959512Z ##[group]Fetching the repository 2024-08-20T22:10:10.7964949Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:13.7401006Z remote: Enumerating objects: 1008518 2024-08-20T22:10:13.7402346Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:13.7404946Z remote: Counting objects: 0% (1/1711) 2024-08-20T22:10:13.7405843Z remote: Counting objects: 1% (18/1711) 2024-08-20T22:10:13.7406782Z remote: Counting objects: 2% (35/1711) 2024-08-20T22:10:13.7407608Z remote: Counting objects: 3% (52/1711) 2024-08-20T22:10:13.7408546Z remote: Counting objects: 4% (69/1711) 2024-08-20T22:10:13.7409297Z remote: Counting objects: 5% (86/1711) 2024-08-20T22:10:13.7410007Z remote: Counting objects: 6% (103/1711) 2024-08-20T22:10:13.7410639Z remote: Counting objects: 7% (120/1711) 2024-08-20T22:10:13.7411174Z remote: Counting objects: 8% (137/1711) 2024-08-20T22:10:13.7411735Z remote: Counting objects: 9% (154/1711) 2024-08-20T22:10:13.7412325Z remote: Counting objects: 10% (172/1711) 2024-08-20T22:10:13.7412921Z remote: Counting objects: 11% (189/1711) 2024-08-20T22:10:13.7413475Z remote: Counting objects: 12% (206/1711) 2024-08-20T22:10:13.7414003Z remote: Counting objects: 13% (223/1711) 2024-08-20T22:10:13.7414517Z remote: Counting objects: 14% (240/1711) 2024-08-20T22:10:13.7415091Z remote: Counting objects: 15% (257/1711) 2024-08-20T22:10:13.7415612Z remote: Counting objects: 16% (274/1711) 2024-08-20T22:10:13.7416172Z remote: Counting objects: 17% (291/1711) 2024-08-20T22:10:13.7416683Z remote: Counting objects: 18% (308/1711) 2024-08-20T22:10:13.7417194Z remote: Counting objects: 19% (326/1711) 2024-08-20T22:10:13.7417707Z remote: Counting objects: 20% (343/1711) 2024-08-20T22:10:13.7418281Z remote: Counting objects: 21% (360/1711) 2024-08-20T22:10:13.7418798Z remote: Counting objects: 22% (377/1711) 2024-08-20T22:10:13.7419312Z remote: Counting objects: 23% (394/1711) 2024-08-20T22:10:13.7419817Z remote: Counting objects: 24% (411/1711) 2024-08-20T22:10:13.7420340Z remote: Counting objects: 25% (428/1711) 2024-08-20T22:10:13.7420909Z remote: Counting objects: 26% (445/1711) 2024-08-20T22:10:13.7421415Z remote: Counting objects: 27% (462/1711) 2024-08-20T22:10:13.7422216Z remote: Counting objects: 28% (480/1711) 2024-08-20T22:10:13.7422798Z remote: Counting objects: 29% (497/1711) 2024-08-20T22:10:13.7423309Z remote: Counting objects: 30% (514/1711) 2024-08-20T22:10:13.7423815Z remote: Counting objects: 31% (531/1711) 2024-08-20T22:10:13.7424333Z remote: Counting objects: 32% (548/1711) 2024-08-20T22:10:13.7424896Z remote: Counting objects: 33% (565/1711) 2024-08-20T22:10:13.7425560Z remote: Counting objects: 34% (582/1711) 2024-08-20T22:10:13.7426064Z remote: Counting objects: 35% (599/1711) 2024-08-20T22:10:13.7426710Z remote: Counting objects: 36% (616/1711) 2024-08-20T22:10:13.7427221Z remote: Counting objects: 37% (634/1711) 2024-08-20T22:10:13.7427731Z remote: Counting objects: 38% (651/1711) 2024-08-20T22:10:13.7428235Z remote: Counting objects: 39% (668/1711) 2024-08-20T22:10:13.7428992Z remote: Counting objects: 40% (685/1711) 2024-08-20T22:10:13.7429506Z remote: Counting objects: 41% (702/1711) 2024-08-20T22:10:13.7430011Z remote: Counting objects: 42% (719/1711) 2024-08-20T22:10:13.7430542Z remote: Counting objects: 43% (736/1711) 2024-08-20T22:10:13.7431042Z remote: Counting objects: 44% (753/1711) 2024-08-20T22:10:13.7431548Z remote: Counting objects: 45% (770/1711) 2024-08-20T22:10:13.7432056Z remote: Counting objects: 46% (788/1711) 2024-08-20T22:10:13.7432565Z remote: Counting objects: 47% (805/1711) 2024-08-20T22:10:13.7433065Z remote: Counting objects: 48% (822/1711) 2024-08-20T22:10:13.7433572Z remote: Counting objects: 49% (839/1711) 2024-08-20T22:10:13.7434077Z remote: Counting objects: 50% (856/1711) 2024-08-20T22:10:13.7434585Z remote: Counting objects: 51% (873/1711) 2024-08-20T22:10:13.7435095Z remote: Counting objects: 52% (890/1711) 2024-08-20T22:10:13.7435608Z remote: Counting objects: 53% (907/1711) 2024-08-20T22:10:13.7436125Z remote: Counting objects: 54% (924/1711) 2024-08-20T22:10:13.7436638Z remote: Counting objects: 55% (942/1711) 2024-08-20T22:10:13.7437143Z remote: Counting objects: 56% (959/1711) 2024-08-20T22:10:13.7437657Z remote: Counting objects: 57% (976/1711) 2024-08-20T22:10:13.7438174Z remote: Counting objects: 58% (993/1711) 2024-08-20T22:10:13.7438692Z remote: Counting objects: 59% (1010/1711) 2024-08-20T22:10:13.7439241Z remote: Counting objects: 60% (1027/1711) 2024-08-20T22:10:13.7439781Z remote: Counting objects: 61% (1044/1711) 2024-08-20T22:10:13.7440318Z remote: Counting objects: 62% (1061/1711) 2024-08-20T22:10:13.7440868Z remote: Counting objects: 63% (1078/1711) 2024-08-20T22:10:13.7441405Z remote: Counting objects: 64% (1096/1711) 2024-08-20T22:10:13.7441932Z remote: Counting objects: 65% (1113/1711) 2024-08-20T22:10:13.7442521Z remote: Counting objects: 66% (1130/1711) 2024-08-20T22:10:13.7443053Z remote: Counting objects: 67% (1147/1711) 2024-08-20T22:10:13.7443577Z remote: Counting objects: 68% (1164/1711) 2024-08-20T22:10:13.7444113Z remote: Counting objects: 69% (1181/1711) 2024-08-20T22:10:13.7444643Z remote: Counting objects: 70% (1198/1711) 2024-08-20T22:10:13.7445175Z remote: Counting objects: 71% (1215/1711) 2024-08-20T22:10:13.7445702Z remote: Counting objects: 72% (1232/1711) 2024-08-20T22:10:13.7446238Z remote: Counting objects: 73% (1250/1711) 2024-08-20T22:10:13.7446766Z remote: Counting objects: 74% (1267/1711) 2024-08-20T22:10:13.7447296Z remote: Counting objects: 75% (1284/1711) 2024-08-20T22:10:13.7447820Z remote: Counting objects: 76% (1301/1711) 2024-08-20T22:10:13.7448351Z remote: Counting objects: 77% (1318/1711) 2024-08-20T22:10:13.7448880Z remote: Counting objects: 78% (1335/1711) 2024-08-20T22:10:13.7449540Z remote: Counting objects: 79% (1352/1711) 2024-08-20T22:10:13.7450065Z remote: Counting objects: 80% (1369/1711) 2024-08-20T22:10:13.7450599Z remote: Counting objects: 81% (1386/1711) 2024-08-20T22:10:13.7451132Z remote: Counting objects: 82% (1404/1711) 2024-08-20T22:10:13.7451665Z remote: Counting objects: 83% (1421/1711) 2024-08-20T22:10:13.7452189Z remote: Counting objects: 84% (1438/1711) 2024-08-20T22:10:13.7452719Z remote: Counting objects: 85% (1455/1711) 2024-08-20T22:10:13.7453361Z remote: Counting objects: 86% (1472/1711) 2024-08-20T22:10:13.7453889Z remote: Counting objects: 87% (1489/1711) 2024-08-20T22:10:13.7454414Z remote: Counting objects: 88% (1506/1711) 2024-08-20T22:10:13.7454939Z remote: Counting objects: 89% (1523/1711) 2024-08-20T22:10:13.7455466Z remote: Counting objects: 90% (1540/1711) 2024-08-20T22:10:13.7455994Z remote: Counting objects: 91% (1558/1711) 2024-08-20T22:10:13.7456522Z remote: Counting objects: 92% (1575/1711) 2024-08-20T22:10:13.7457053Z remote: Counting objects: 93% (1592/1711) 2024-08-20T22:10:13.7457582Z remote: Counting objects: 94% (1609/1711) 2024-08-20T22:10:13.7458113Z remote: Counting objects: 95% (1626/1711) 2024-08-20T22:10:13.7458634Z remote: Counting objects: 96% (1643/1711) 2024-08-20T22:10:13.7459159Z remote: Counting objects: 97% (1660/1711) 2024-08-20T22:10:13.7459693Z remote: Counting objects: 98% (1677/1711) 2024-08-20T22:10:13.7460220Z remote: Counting objects: 99% (1694/1711) 2024-08-20T22:10:13.7460741Z remote: Counting objects: 100% (1711/1711) 2024-08-20T22:10:13.7461304Z remote: Counting objects: 100% (1711/1711), done. 2024-08-20T22:10:13.7527165Z remote: Compressing objects: 0% (1/885) 2024-08-20T22:10:13.7864994Z remote: Compressing objects: 1% (9/885) 2024-08-20T22:10:13.8031863Z remote: Compressing objects: 2% (18/885) 2024-08-20T22:10:13.8537600Z remote: Compressing objects: 3% (27/885) 2024-08-20T22:10:13.9179490Z remote: Compressing objects: 4% (36/885) 2024-08-20T22:10:13.9865654Z remote: Compressing objects: 5% (45/885) 2024-08-20T22:10:14.0301213Z remote: Compressing objects: 6% (54/885) 2024-08-20T22:10:14.0738778Z remote: Compressing objects: 7% (62/885) 2024-08-20T22:10:14.1185505Z remote: Compressing objects: 8% (71/885) 2024-08-20T22:10:14.1497576Z remote: Compressing objects: 9% (80/885) 2024-08-20T22:10:14.1775903Z remote: Compressing objects: 10% (89/885) 2024-08-20T22:10:14.2078500Z remote: Compressing objects: 11% (98/885) 2024-08-20T22:10:14.2297971Z remote: Compressing objects: 12% (107/885) 2024-08-20T22:10:14.2473097Z remote: Compressing objects: 13% (116/885) 2024-08-20T22:10:14.2641659Z remote: Compressing objects: 14% (124/885) 2024-08-20T22:10:14.2767521Z remote: Compressing objects: 15% (133/885) 2024-08-20T22:10:14.2851375Z remote: Compressing objects: 16% (142/885) 2024-08-20T22:10:14.2946037Z remote: Compressing objects: 17% (151/885) 2024-08-20T22:10:14.3011325Z remote: Compressing objects: 18% (160/885) 2024-08-20T22:10:14.3061438Z remote: Compressing objects: 19% (169/885) 2024-08-20T22:10:14.3105034Z remote: Compressing objects: 20% (177/885) 2024-08-20T22:10:14.3130941Z remote: Compressing objects: 21% (186/885) 2024-08-20T22:10:14.3149327Z remote: Compressing objects: 22% (195/885) 2024-08-20T22:10:14.3152654Z remote: Compressing objects: 23% (204/885) 2024-08-20T22:10:14.3160219Z remote: Compressing objects: 24% (213/885) 2024-08-20T22:10:14.3169070Z remote: Compressing objects: 25% (222/885) 2024-08-20T22:10:14.3174617Z remote: Compressing objects: 26% (231/885) 2024-08-20T22:10:14.3181860Z remote: Compressing objects: 27% (239/885) 2024-08-20T22:10:14.3195707Z remote: Compressing objects: 28% (248/885) 2024-08-20T22:10:14.3213335Z remote: Compressing objects: 29% (257/885) 2024-08-20T22:10:14.3224560Z remote: Compressing objects: 30% (266/885) 2024-08-20T22:10:14.3236045Z remote: Compressing objects: 31% (275/885) 2024-08-20T22:10:14.3247657Z remote: Compressing objects: 32% (284/885) 2024-08-20T22:10:14.3254929Z remote: Compressing objects: 33% (293/885) 2024-08-20T22:10:14.3260648Z remote: Compressing objects: 34% (301/885) 2024-08-20T22:10:14.3270484Z remote: Compressing objects: 35% (310/885) 2024-08-20T22:10:14.3276876Z remote: Compressing objects: 36% (319/885) 2024-08-20T22:10:14.3287167Z remote: Compressing objects: 37% (328/885) 2024-08-20T22:10:14.3294516Z remote: Compressing objects: 38% (337/885) 2024-08-20T22:10:14.3299031Z remote: Compressing objects: 39% (346/885) 2024-08-20T22:10:14.3305754Z remote: Compressing objects: 40% (354/885) 2024-08-20T22:10:14.3311060Z remote: Compressing objects: 41% (363/885) 2024-08-20T22:10:14.3318730Z remote: Compressing objects: 42% (372/885) 2024-08-20T22:10:14.3323562Z remote: Compressing objects: 43% (381/885) 2024-08-20T22:10:14.3331161Z remote: Compressing objects: 44% (390/885) 2024-08-20T22:10:14.3336909Z remote: Compressing objects: 45% (399/885) 2024-08-20T22:10:14.3340245Z remote: Compressing objects: 46% (408/885) 2024-08-20T22:10:14.3344405Z remote: Compressing objects: 47% (416/885) 2024-08-20T22:10:14.3348164Z remote: Compressing objects: 48% (425/885) 2024-08-20T22:10:14.3351523Z remote: Compressing objects: 49% (434/885) 2024-08-20T22:10:14.3354092Z remote: Compressing objects: 50% (443/885) 2024-08-20T22:10:14.3357590Z remote: Compressing objects: 51% (452/885) 2024-08-20T22:10:14.3360453Z remote: Compressing objects: 52% (461/885) 2024-08-20T22:10:14.3362364Z remote: Compressing objects: 53% (470/885) 2024-08-20T22:10:14.3363655Z remote: Compressing objects: 54% (478/885) 2024-08-20T22:10:14.3365873Z remote: Compressing objects: 55% (487/885) 2024-08-20T22:10:14.3367090Z remote: Compressing objects: 56% (496/885) 2024-08-20T22:10:14.3368725Z remote: Compressing objects: 57% (505/885) 2024-08-20T22:10:14.3370054Z remote: Compressing objects: 58% (514/885) 2024-08-20T22:10:14.3370609Z remote: Compressing objects: 59% (523/885) 2024-08-20T22:10:14.3371205Z remote: Compressing objects: 60% (531/885) 2024-08-20T22:10:14.3371754Z remote: Compressing objects: 61% (540/885) 2024-08-20T22:10:14.3372347Z remote: Compressing objects: 62% (549/885) 2024-08-20T22:10:14.3372894Z remote: Compressing objects: 63% (558/885) 2024-08-20T22:10:14.3376702Z remote: Compressing objects: 64% (567/885) 2024-08-20T22:10:14.3384217Z remote: Compressing objects: 65% (576/885) 2024-08-20T22:10:14.3389855Z remote: Compressing objects: 66% (585/885) 2024-08-20T22:10:14.3394628Z remote: Compressing objects: 67% (593/885) 2024-08-20T22:10:14.3399185Z remote: Compressing objects: 68% (602/885) 2024-08-20T22:10:14.3404235Z remote: Compressing objects: 69% (611/885) 2024-08-20T22:10:14.3408568Z remote: Compressing objects: 70% (620/885) 2024-08-20T22:10:14.3412094Z remote: Compressing objects: 71% (629/885) 2024-08-20T22:10:14.3415544Z remote: Compressing objects: 72% (638/885) 2024-08-20T22:10:14.3419945Z remote: Compressing objects: 73% (647/885) 2024-08-20T22:10:14.3422768Z remote: Compressing objects: 74% (655/885) 2024-08-20T22:10:14.3425650Z remote: Compressing objects: 75% (664/885) 2024-08-20T22:10:14.3428184Z remote: Compressing objects: 76% (673/885) 2024-08-20T22:10:14.3432124Z remote: Compressing objects: 77% (682/885) 2024-08-20T22:10:14.3433816Z remote: Compressing objects: 78% (691/885) 2024-08-20T22:10:14.3435678Z remote: Compressing objects: 79% (700/885) 2024-08-20T22:10:14.3438266Z remote: Compressing objects: 80% (708/885) 2024-08-20T22:10:14.3440097Z remote: Compressing objects: 81% (717/885) 2024-08-20T22:10:14.3442660Z remote: Compressing objects: 82% (726/885) 2024-08-20T22:10:14.3444269Z remote: Compressing objects: 83% (735/885) 2024-08-20T22:10:14.3445912Z remote: Compressing objects: 84% (744/885) 2024-08-20T22:10:14.3448048Z remote: Compressing objects: 85% (753/885) 2024-08-20T22:10:14.3449972Z remote: Compressing objects: 86% (762/885) 2024-08-20T22:10:14.3452370Z remote: Compressing objects: 87% (770/885) 2024-08-20T22:10:14.3454207Z remote: Compressing objects: 88% (779/885) 2024-08-20T22:10:14.3456619Z remote: Compressing objects: 89% (788/885) 2024-08-20T22:10:14.3457507Z remote: Compressing objects: 90% (797/885) 2024-08-20T22:10:14.3459393Z remote: Compressing objects: 91% (806/885) 2024-08-20T22:10:14.3459981Z remote: Compressing objects: 92% (815/885) 2024-08-20T22:10:14.3460764Z remote: Compressing objects: 93% (824/885) 2024-08-20T22:10:14.3462418Z remote: Compressing objects: 94% (832/885) 2024-08-20T22:10:14.3463305Z remote: Compressing objects: 95% (841/885) 2024-08-20T22:10:14.3464503Z remote: Compressing objects: 96% (850/885) 2024-08-20T22:10:14.3466126Z remote: Compressing objects: 97% (859/885) 2024-08-20T22:10:14.3467201Z remote: Compressing objects: 98% (868/885) 2024-08-20T22:10:14.3468406Z remote: Compressing objects: 99% (877/885) 2024-08-20T22:10:14.3469507Z remote: Compressing objects: 100% (885/885) 2024-08-20T22:10:14.3470094Z remote: Compressing objects: 100% (885/885), done. 2024-08-20T22:10:36.6347570Z remote: Total 1010229 (delta 1101), reused 1314 (delta 823), pack-reused 1008518 (from 1) 2024-08-20T22:10:58.3703543Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:10:58.3740672Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:58.3746445Z ##[endgroup] 2024-08-20T22:10:58.3748622Z ##[group]Determining the checkout info 2024-08-20T22:10:58.3749286Z ##[endgroup] 2024-08-20T22:10:58.3749825Z ##[group]Checking out the ref 2024-08-20T22:10:58.3752006Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:00.1109704Z ##[endgroup] 2024-08-20T22:11:00.1110914Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:00.1113824Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:00.1176030Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:00.1217537Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:00.1259727Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:00.1301119Z ##[endgroup] 2024-08-20T22:11:00.1305608Z ##[group]Fetching submodules 2024-08-20T22:11:00.1306430Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:00.1692206Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:00.2060807Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:00.2062920Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:00.2065378Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:00.2068499Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:00.2072205Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:00.2074872Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:00.2078417Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:00.2082013Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:00.2085344Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:00.2089229Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:00.2092759Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:00.2096444Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:00.2100253Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:00.2104815Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:00.2108832Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:00.2113200Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:00.2120446Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:00.2124714Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:00.2129073Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:00.2134620Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:00.2138377Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:00.2142724Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:00.2147292Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:00.2151962Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:00.2156480Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:00.2161577Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:00.2166142Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:00.2171031Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:00.2175921Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:00.2181076Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:00.2186093Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:00.2191686Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:00.2196625Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:00.2205178Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:00.2241770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:00.5257409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:00.6873370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:00.8590217Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:01.1054564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:03.0576521Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:14.5186122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:14.9645720Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:15.3772007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:15.9883943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:17.2447821Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:19.3229151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:25.3987995Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:26.8371830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:28.6626153Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:29.9499356Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:30.3787328Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:30.7303726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:31.9203238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:32.3212726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:32.5495426Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:34.2734432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:34.9642744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:35.6688494Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:41.7180636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:43.6789413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:47.9551531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:48.2045375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:11:57.3778666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:11:57.5584162Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:11:57.7485766Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:11:58.6705473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:11:59.0053030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:11:59.6305964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:00.0838347Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:00.0989639Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:00.1108664Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:00.1421752Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:00.1870209Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:01.3689628Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:01.3978715Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:01.4509721Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:01.5643464Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:01.6048567Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:02.2184344Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:02.5066181Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:02.6012051Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:02.6036960Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:02.6040281Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:02.6043676Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:02.6047337Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:02.6051023Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:02.6082410Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:03.7125632Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:04.3105100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:06.4322202Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:07.4522573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:07.7800391Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:07.8918388Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:08.4002002Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:08.4711349Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:08.4860141Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:08.6394890Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:08.6856195Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:08.7320611Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:08.7641114Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:08.8172561Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:08.8333889Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:08.8353028Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:08.8389598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:21.5962741Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:21.6187035Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:21.7193455Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:21.7217517Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:21.7220769Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:21.7224309Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:21.7258849Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:22.2694591Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:23.5513003Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:24.6723977Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:24.6743034Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:24.6746660Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:24.6750355Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:24.6753913Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:24.6757608Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:24.6761489Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:24.6765450Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:24.6769171Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:24.6803357Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:25.5554684Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:25.9197971Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:27.1385898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:27.4233417Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:27.9594455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:29.0229075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:36.2987075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:36.7659684Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:36.7888869Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:36.8319265Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:36.8488054Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:36.8507835Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:36.8541251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:37.1711729Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:37.1943314Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:37.2426187Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:37.3623583Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:37.3826633Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:37.4278519Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:37.4942028Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:37.5395608Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:37.5693877Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:37.6925082Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:38.2054403Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:38.2091559Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:38.2095290Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:38.2127431Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:38.6622078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:39.6165692Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:39.6572124Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:39.7481100Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:39.7505710Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:39.7509224Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:39.7512701Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:39.7516272Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:39.7520071Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:39.7523712Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:39.7527453Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:39.7533012Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:39.7564382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:40.2046981Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:41.2480030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:41.5959922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:47.6221638Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:47.8949918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:48.1161064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:48.4117833Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:55.1833278Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:55.2307857Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:55.2496622Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:55.3727720Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:55.3886503Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:55.4075865Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:55.4282784Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:55.4302580Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:55.4305983Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:55.4338147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:12:57.3038742Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:12:58.6872000Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:12:58.7406668Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:59.3922631Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:12:59.4065763Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:12:59.7254470Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:12:59.7282445Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:59.7285677Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:59.7319454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:00.1706420Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:01.2477099Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:01.3293084Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:01.3412080Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:01.3570381Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:01.4018005Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:01.4363737Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:01.4850017Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:01.5195224Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:01.5217217Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:01.5220559Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:01.5224391Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:01.5227902Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:01.5262068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:02.5663964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:02.7994907Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:04.0106409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:04.9666067Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:04.9861399Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:05.0647148Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:05.0998847Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:05.1018080Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:05.1049241Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:05.3195198Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:05.3244579Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:05.3611823Z Entering 'android/libs/fbjni' 2024-08-20T22:13:05.3663092Z Entering 'third_party/FP16' 2024-08-20T22:13:05.3713811Z Entering 'third_party/FXdiv' 2024-08-20T22:13:05.3771196Z Entering 'third_party/NNPACK' 2024-08-20T22:13:05.3821740Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:05.3872395Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:05.3939394Z Entering 'third_party/benchmark' 2024-08-20T22:13:05.3989734Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:05.4040018Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:05.4092815Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:05.4144747Z Entering 'third_party/cutlass' 2024-08-20T22:13:05.4203829Z Entering 'third_party/eigen' 2024-08-20T22:13:05.4258660Z Entering 'third_party/fbgemm' 2024-08-20T22:13:05.4309070Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:05.4360384Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:05.4414952Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:05.4473262Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:05.4525060Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:05.4582090Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:05.4637007Z Entering 'third_party/fmt' 2024-08-20T22:13:05.4686814Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:05.4738779Z Entering 'third_party/gloo' 2024-08-20T22:13:05.4789790Z Entering 'third_party/googletest' 2024-08-20T22:13:05.4840222Z Entering 'third_party/ideep' 2024-08-20T22:13:05.4888122Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:05.4952229Z Entering 'third_party/ittapi' 2024-08-20T22:13:05.5002943Z Entering 'third_party/kineto' 2024-08-20T22:13:05.5053414Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:05.5103586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:05.5157622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:05.5211079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:05.5262898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:05.5312250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:05.5367339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:05.5418247Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:05.5469741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:05.5521274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:05.5576560Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:05.5627575Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:05.5681769Z Entering 'third_party/mimalloc' 2024-08-20T22:13:05.5734332Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:05.5787759Z Entering 'third_party/nlohmann' 2024-08-20T22:13:05.5840997Z Entering 'third_party/onnx' 2024-08-20T22:13:05.5906669Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:05.5958686Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:05.6014952Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:05.6068461Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:05.6123018Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:05.6174162Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:05.6224148Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:05.6276703Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:05.6331301Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:05.6380869Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:05.6429839Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:05.6482701Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:05.6536323Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:05.6614382Z Entering 'third_party/pocketfft' 2024-08-20T22:13:05.6666946Z Entering 'third_party/protobuf' 2024-08-20T22:13:05.6720974Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:05.6773117Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:05.6829320Z Entering 'third_party/psimd' 2024-08-20T22:13:05.6881373Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:05.6932849Z Entering 'third_party/pybind11' 2024-08-20T22:13:05.6984250Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:05.7036810Z Entering 'third_party/sleef' 2024-08-20T22:13:05.7087284Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:05.7138035Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:05.7189832Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:05.7240760Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:05.7294566Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:05.7343456Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:05.7419443Z ##[endgroup] 2024-08-20T22:13:05.7423293Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:05.7425055Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:05.7790938Z Entering 'android/libs/fbjni' 2024-08-20T22:13:05.7859524Z Entering 'third_party/FP16' 2024-08-20T22:13:05.7927086Z Entering 'third_party/FXdiv' 2024-08-20T22:13:05.8000730Z Entering 'third_party/NNPACK' 2024-08-20T22:13:05.8069424Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:05.8141520Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:05.8224840Z Entering 'third_party/benchmark' 2024-08-20T22:13:05.8293792Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:05.8362037Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:05.8431247Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:05.8499044Z Entering 'third_party/cutlass' 2024-08-20T22:13:05.8578698Z Entering 'third_party/eigen' 2024-08-20T22:13:05.8648519Z Entering 'third_party/fbgemm' 2024-08-20T22:13:05.8714377Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:05.8782826Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:05.8849647Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:05.8923691Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:05.8991418Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:05.9060807Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:05.9131091Z Entering 'third_party/fmt' 2024-08-20T22:13:05.9200314Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:05.9269550Z Entering 'third_party/gloo' 2024-08-20T22:13:05.9337337Z Entering 'third_party/googletest' 2024-08-20T22:13:05.9404541Z Entering 'third_party/ideep' 2024-08-20T22:13:05.9473284Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:05.9548446Z Entering 'third_party/ittapi' 2024-08-20T22:13:05.9620556Z Entering 'third_party/kineto' 2024-08-20T22:13:05.9690410Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:05.9754899Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:05.9823015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:05.9891044Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:05.9958381Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:06.0022558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:06.0094454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:06.0162005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:06.0231713Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:06.0304888Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:06.0378500Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:06.0444529Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:06.0513364Z Entering 'third_party/mimalloc' 2024-08-20T22:13:06.0581878Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:06.0650061Z Entering 'third_party/nlohmann' 2024-08-20T22:13:06.0718628Z Entering 'third_party/onnx' 2024-08-20T22:13:06.0800184Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:06.0867659Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:06.0939668Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:06.1011412Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:06.1077712Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:06.1142942Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:06.1207841Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:06.1277143Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:06.1342911Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:06.1408773Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:06.1472534Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:06.1542200Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:06.1612285Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:06.1702357Z Entering 'third_party/pocketfft' 2024-08-20T22:13:06.1769403Z Entering 'third_party/protobuf' 2024-08-20T22:13:06.1844493Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:06.1910547Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:06.1980257Z Entering 'third_party/psimd' 2024-08-20T22:13:06.2051177Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:06.2120460Z Entering 'third_party/pybind11' 2024-08-20T22:13:06.2188171Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:06.2255408Z Entering 'third_party/sleef' 2024-08-20T22:13:06.2322154Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:06.2388091Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:06.2454738Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:06.2520420Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:06.2588518Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:06.2653945Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:06.2750694Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:06.3115602Z Entering 'android/libs/fbjni' 2024-08-20T22:13:06.3177446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:06.3199118Z Entering 'third_party/FP16' 2024-08-20T22:13:06.3261746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:06.3283618Z Entering 'third_party/FXdiv' 2024-08-20T22:13:06.3345535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:06.3366669Z Entering 'third_party/NNPACK' 2024-08-20T22:13:06.3427438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:06.3448764Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:06.3509517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:06.3530716Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:06.3592592Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:06.3627648Z Entering 'third_party/benchmark' 2024-08-20T22:13:06.3689084Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:06.3716016Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:06.3778383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:06.3799723Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:06.3862606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:06.3884664Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:06.3946357Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:06.3967803Z Entering 'third_party/cutlass' 2024-08-20T22:13:06.4029384Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:06.4060211Z Entering 'third_party/eigen' 2024-08-20T22:13:06.4121580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:06.4145141Z Entering 'third_party/fbgemm' 2024-08-20T22:13:06.4207317Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:06.4227356Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:06.4288905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:06.4313220Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:06.4375475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:06.4396094Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:06.4457622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:06.4484087Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:06.4544823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.4564918Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:06.4625941Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:06.4648523Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:06.4712040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:06.4734783Z Entering 'third_party/fmt' 2024-08-20T22:13:06.4796257Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:06.4817674Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:06.4879861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:06.4900832Z Entering 'third_party/gloo' 2024-08-20T22:13:06.4961432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:06.4982583Z Entering 'third_party/googletest' 2024-08-20T22:13:06.5044509Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.5069407Z Entering 'third_party/ideep' 2024-08-20T22:13:06.5132935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:06.5152365Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:06.5213431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:06.5242636Z Entering 'third_party/ittapi' 2024-08-20T22:13:06.5304499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:06.5325877Z Entering 'third_party/kineto' 2024-08-20T22:13:06.5388626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:06.5408459Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:06.5472000Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:06.5490464Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:06.5554977Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:06.5576803Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:06.5639486Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:06.5660304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:06.5727361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:06.5748315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:06.5810796Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:06.5829898Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:06.5895331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:06.5917118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:06.5979339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:06.6000184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:06.6062673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.6084745Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:06.6147343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:06.6168642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:06.6231957Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:06.6254834Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:06.6316118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:06.6336802Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:06.6398432Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.6421143Z Entering 'third_party/mimalloc' 2024-08-20T22:13:06.6483687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:06.6504950Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:06.6570650Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:06.6593662Z Entering 'third_party/nlohmann' 2024-08-20T22:13:06.6654746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:06.6677144Z Entering 'third_party/onnx' 2024-08-20T22:13:06.6740887Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:06.6776607Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:06.6844612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:06.6866000Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:06.6928928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:06.6960295Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:06.7025120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:06.7046195Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:06.7107971Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:06.7132950Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:06.7196584Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.7216394Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:06.7277833Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:06.7297756Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:06.7359604Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:06.7381759Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:06.7445169Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:06.7464703Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:06.7525763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:06.7545659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:06.7607655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:06.7630956Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:06.7697191Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:06.7724534Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:06.7787548Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:06.7810745Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:06.7875097Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:06.7917167Z Entering 'third_party/pocketfft' 2024-08-20T22:13:06.7980875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:06.8002225Z Entering 'third_party/protobuf' 2024-08-20T22:13:06.8064194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:06.8087230Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:06.8149673Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:06.8169220Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:06.8235930Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.8261081Z Entering 'third_party/psimd' 2024-08-20T22:13:06.8322242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:06.8344167Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:06.8405476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:06.8426502Z Entering 'third_party/pybind11' 2024-08-20T22:13:06.8487689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:06.8508965Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:06.8571470Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:06.8591729Z Entering 'third_party/sleef' 2024-08-20T22:13:06.8653482Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:06.8675623Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:06.8738998Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:06.8759648Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:06.8820750Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:06.8843420Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:06.8905803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:06.8925286Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:06.8989502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:06.9009445Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:06.9071556Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:06.9090554Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:06.9155407Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:07.0051572Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:07.0420118Z Entering 'android/libs/fbjni' 2024-08-20T22:13:07.0477154Z Entering 'third_party/FP16' 2024-08-20T22:13:07.0528432Z Entering 'third_party/FXdiv' 2024-08-20T22:13:07.0580719Z Entering 'third_party/NNPACK' 2024-08-20T22:13:07.0633565Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:07.0687373Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:07.0759475Z Entering 'third_party/benchmark' 2024-08-20T22:13:07.0808858Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:07.0860596Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:07.0912216Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:07.0961887Z Entering 'third_party/cutlass' 2024-08-20T22:13:07.1019403Z Entering 'third_party/eigen' 2024-08-20T22:13:07.1073699Z Entering 'third_party/fbgemm' 2024-08-20T22:13:07.1123386Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:07.1172816Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:07.1221967Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:07.1287048Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:07.1339230Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:07.1391743Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:07.1457589Z Entering 'third_party/fmt' 2024-08-20T22:13:07.1514219Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:07.1578734Z Entering 'third_party/gloo' 2024-08-20T22:13:07.1635765Z Entering 'third_party/googletest' 2024-08-20T22:13:07.1693694Z Entering 'third_party/ideep' 2024-08-20T22:13:07.1742964Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:07.1815079Z Entering 'third_party/ittapi' 2024-08-20T22:13:07.1870388Z Entering 'third_party/kineto' 2024-08-20T22:13:07.1920211Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:07.1977292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:07.2031946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:07.2085211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:07.2139609Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:07.2191410Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:07.2255578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:07.2307065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:07.2363071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:07.2416889Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:07.2477269Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:07.2528510Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:07.2595513Z Entering 'third_party/mimalloc' 2024-08-20T22:13:07.2649963Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:07.2703267Z Entering 'third_party/nlohmann' 2024-08-20T22:13:07.2759033Z Entering 'third_party/onnx' 2024-08-20T22:13:07.2824520Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:07.2878985Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:07.2935955Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:07.2989702Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:07.3040244Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:07.3090851Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:07.3142334Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:07.3194015Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:07.3244231Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:07.3294996Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:07.3343187Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:07.3396581Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:07.3450768Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:07.3523396Z Entering 'third_party/pocketfft' 2024-08-20T22:13:07.3576679Z Entering 'third_party/protobuf' 2024-08-20T22:13:07.3631906Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:07.3683619Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:07.3742810Z Entering 'third_party/psimd' 2024-08-20T22:13:07.3794635Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:07.3847152Z Entering 'third_party/pybind11' 2024-08-20T22:13:07.3899563Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:07.3953277Z Entering 'third_party/sleef' 2024-08-20T22:13:07.4005604Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:07.4057783Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:07.4111520Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:07.4163464Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:07.4213971Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:07.4263828Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:07.4341545Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:07.4706966Z Entering 'android/libs/fbjni' 2024-08-20T22:13:07.4760411Z Entering 'third_party/FP16' 2024-08-20T22:13:07.4814241Z Entering 'third_party/FXdiv' 2024-08-20T22:13:07.4867517Z Entering 'third_party/NNPACK' 2024-08-20T22:13:07.4919858Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:07.4972821Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:07.5040260Z Entering 'third_party/benchmark' 2024-08-20T22:13:07.5091859Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:07.5147857Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:07.5199673Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:07.5251662Z Entering 'third_party/cutlass' 2024-08-20T22:13:07.5310351Z Entering 'third_party/eigen' 2024-08-20T22:13:07.5364365Z Entering 'third_party/fbgemm' 2024-08-20T22:13:07.5415077Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:07.5466563Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:07.5522223Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:07.5579975Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:07.5635590Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:07.5690887Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:07.5750899Z Entering 'third_party/fmt' 2024-08-20T22:13:07.5805163Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:07.5858327Z Entering 'third_party/gloo' 2024-08-20T22:13:07.5911134Z Entering 'third_party/googletest' 2024-08-20T22:13:07.5964652Z Entering 'third_party/ideep' 2024-08-20T22:13:07.6014980Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:07.6078787Z Entering 'third_party/ittapi' 2024-08-20T22:13:07.6135221Z Entering 'third_party/kineto' 2024-08-20T22:13:07.6185735Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:07.6235854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:07.6289027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:07.6343223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:07.6393838Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:07.6441505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:07.6497957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:07.6550120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:07.6601551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:07.6654128Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:07.6708146Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:07.6759084Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:07.6813038Z Entering 'third_party/mimalloc' 2024-08-20T22:13:07.6865532Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:07.6917726Z Entering 'third_party/nlohmann' 2024-08-20T22:13:07.6973824Z Entering 'third_party/onnx' 2024-08-20T22:13:07.7039819Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:07.7093052Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:07.7154016Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:07.7206076Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:07.7257587Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:07.7308422Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:07.7357834Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:07.7414219Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:07.7472927Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:07.7524829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:07.7573790Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:07.7627754Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:07.7681608Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:07.7757584Z Entering 'third_party/pocketfft' 2024-08-20T22:13:07.7810141Z Entering 'third_party/protobuf' 2024-08-20T22:13:07.7867656Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:07.7918252Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:07.7977314Z Entering 'third_party/psimd' 2024-08-20T22:13:07.8030204Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:07.8081909Z Entering 'third_party/pybind11' 2024-08-20T22:13:07.8136196Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:07.8187523Z Entering 'third_party/sleef' 2024-08-20T22:13:07.8239431Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:07.8289735Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:07.8340155Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:07.8395185Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:07.8445661Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:07.8493769Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:07.8566029Z ##[endgroup] 2024-08-20T22:13:07.8618955Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:07.8655831Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:07.8869772Z Prepare all required actions 2024-08-20T22:13:07.8870340Z Getting action download info 2024-08-20T22:13:08.0275404Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:08.0275846Z env: 2024-08-20T22:13:08.0276135Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:08.0276478Z ##[endgroup] 2024-08-20T22:13:08.0337664Z ##[group]Run set -euo pipefail 2024-08-20T22:13:08.0338105Z set -euo pipefail 2024-08-20T22:13:08.0338508Z function get_ec2_metadata() { 2024-08-20T22:13:08.0339066Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:08.0339981Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:08.0340773Z  category=$1 2024-08-20T22:13:08.0341339Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:08.0342005Z  runner_name_str=i-07cd86c6b6ea02560 2024-08-20T22:13:08.0342519Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:08.0343038Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:08.0343633Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:08.0344349Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:08.0344973Z  else 2024-08-20T22:13:08.0345492Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:08.0346173Z  fi 2024-08-20T22:13:08.0346479Z } 2024-08-20T22:13:08.0346851Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:08.0347472Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:08.0348155Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:08.0348754Z echo "system info $(uname -a)" 2024-08-20T22:13:08.0360110Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:08.0360627Z env: 2024-08-20T22:13:08.0360919Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:08.0361274Z ##[endgroup] 2024-08-20T22:13:08.0463386Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:08.0529619Z instance-id: i-07cd86c6b6ea02560 2024-08-20T22:13:08.0586873Z instance-type: g5.4xlarge 2024-08-20T22:13:08.0600541Z system info Linux ip-10-0-44-42.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:08.0625077Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:08.0626167Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:08.0638146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:08.0638809Z env: 2024-08-20T22:13:08.0639093Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:08.0639445Z ##[endgroup] 2024-08-20T22:13:08.0716602Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:08.0717236Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:08.0717803Z  echo "Docker daemon is running..."; 2024-08-20T22:13:08.0718271Z else 2024-08-20T22:13:08.0718782Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:08.0719382Z fi 2024-08-20T22:13:08.0728019Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:08.0728523Z env: 2024-08-20T22:13:08.0729222Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:08.0729577Z ##[endgroup] 2024-08-20T22:13:08.0816086Z Docker daemon is running... 2024-08-20T22:13:08.0865372Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:08.0865940Z with: 2024-08-20T22:13:08.0866318Z shell: bash 2024-08-20T22:13:08.0866615Z timeout_minutes: 5 2024-08-20T22:13:08.0866943Z max_attempts: 3 2024-08-20T22:13:08.0867272Z retry_wait_seconds: 30 2024-08-20T22:13:08.0871497Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:08.0874561Z polling_interval_seconds: 1 2024-08-20T22:13:08.0874953Z warning_on_retry: true 2024-08-20T22:13:08.0875309Z continue_on_error: false 2024-08-20T22:13:08.0875655Z env: 2024-08-20T22:13:08.0875940Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:08.0876303Z AWS_RETRY_MODE: standard 2024-08-20T22:13:08.0876654Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:08.0877007Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:08.0877393Z ##[endgroup] 2024-08-20T22:13:09.2961191Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:09.2962078Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:09.2962959Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:09.2963491Z 2024-08-20T22:13:09.2963626Z Login Succeeded 2024-08-20T22:13:10.1459577Z Command completed after 1 attempt(s). 2024-08-20T22:13:10.1522647Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:10.1523384Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:10.1524054Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:10.1533609Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.1534112Z env: 2024-08-20T22:13:10.1534404Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.1534762Z ##[endgroup] 2024-08-20T22:13:10.1630925Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:10.1631710Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:10.1632317Z # shellcheck disable=SC2046 2024-08-20T22:13:10.1632797Z docker stop $(docker ps -q) || true 2024-08-20T22:13:10.1633286Z # Prune all of the docker images 2024-08-20T22:13:10.1633752Z docker system prune -af 2024-08-20T22:13:10.1643050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.1643559Z env: 2024-08-20T22:13:10.1643842Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.1644202Z ##[endgroup] 2024-08-20T22:13:10.1967828Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:10.1968834Z See 'docker stop --help'. 2024-08-20T22:13:10.1969635Z 2024-08-20T22:13:10.1969937Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:10.1970503Z 2024-08-20T22:13:10.1970710Z Stop one or more running containers 2024-08-20T22:13:10.2142211Z Total reclaimed space: 0B 2024-08-20T22:13:10.2188269Z ##[group]Run set +e 2024-08-20T22:13:10.2188644Z set +e 2024-08-20T22:13:10.2188960Z set -x 2024-08-20T22:13:10.2189261Z  2024-08-20T22:13:10.2189590Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:10.2190396Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:10.2191494Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:10.2192270Z # one is returned at random 2024-08-20T22:13:10.2192837Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:10.2193358Z  2024-08-20T22:13:10.2193709Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:10.2194352Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:10.2195357Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:10.2195927Z  2024-08-20T22:13:10.2196271Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:10.2196846Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:10.2197375Z  exit 1 2024-08-20T22:13:10.2197700Z  fi 2024-08-20T22:13:10.2198001Z fi 2024-08-20T22:13:10.2198297Z  2024-08-20T22:13:10.2198677Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:10.2199208Z  # Clean up any old records first 2024-08-20T22:13:10.2199741Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:10.2200215Z fi 2024-08-20T22:13:10.2200508Z  2024-08-20T22:13:10.2200961Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:10.2201538Z cat /etc/hosts 2024-08-20T22:13:10.2210657Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.2211163Z env: 2024-08-20T22:13:10.2211465Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.2211825Z ##[endgroup] 2024-08-20T22:13:10.2241894Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:10.2248601Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:10.2249078Z ++ tail -n1 2024-08-20T22:13:10.2472132Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:10.2472607Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:10.2473164Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:10.2485895Z 108.138.64.116 download.pytorch.org 2024-08-20T22:13:10.2488077Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T22:13:10.3633769Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:10.3634522Z + sudo tee -a /etc/hosts 2024-08-20T22:13:10.3963291Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:10.3986332Z + cat /etc/hosts 2024-08-20T22:13:10.3996374Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:10.4009888Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:10.4010624Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:10.4169218Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:10.4169834Z with: 2024-08-20T22:13:10.4171009Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4172258Z docker-build-dir: .ci/docker 2024-08-20T22:13:10.4172653Z working-directory: . 2024-08-20T22:13:10.4173130Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.4173663Z force-push: false 2024-08-20T22:13:10.4173971Z env: 2024-08-20T22:13:10.4174250Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.4174598Z ##[endgroup] 2024-08-20T22:13:10.4193793Z ##[group]Run set -ex 2024-08-20T22:13:10.4194170Z set -ex 2024-08-20T22:13:10.4194468Z  2024-08-20T22:13:10.4195046Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:10.4196052Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:10.4196879Z # job could then download the pre-built image as usual 2024-08-20T22:13:10.4197648Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:10.4198347Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4199009Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4199586Z  2024-08-20T22:13:10.4200118Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:10.4200769Z  exit 0 2024-08-20T22:13:10.4201092Z else 2024-08-20T22:13:10.4201473Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4201942Z fi 2024-08-20T22:13:10.4202234Z  2024-08-20T22:13:10.4202724Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:10.4203597Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:10.4204405Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:10.4205137Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:10.4205883Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4206606Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4207182Z else 2024-08-20T22:13:10.4207638Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:10.4208333Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4209272Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4210052Z fi 2024-08-20T22:13:10.4220429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.4220990Z env: 2024-08-20T22:13:10.4221278Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.4221640Z REPO_NAME: pytorch 2024-08-20T22:13:10.4222788Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4224021Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:10.4224530Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.4225056Z ##[endgroup] 2024-08-20T22:13:10.4253795Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:10.4254222Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:10.4254605Z + echo skip=false 2024-08-20T22:13:10.4256669Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:10.4261837Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4263099Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:10.4288401Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4289231Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4290781Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4325458Z ##[group]Run set +e 2024-08-20T22:13:10.4326133Z set +e 2024-08-20T22:13:10.4326558Z set -x 2024-08-20T22:13:10.4326941Z  2024-08-20T22:13:10.4327335Z login() { 2024-08-20T22:13:10.4328158Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:10.4329273Z } 2024-08-20T22:13:10.4329748Z  2024-08-20T22:13:10.4330240Z retry () { 2024-08-20T22:13:10.4330731Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:10.4331305Z } 2024-08-20T22:13:10.4331726Z  2024-08-20T22:13:10.4332168Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:10.4332647Z  2024-08-20T22:13:10.4333273Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:10.4334099Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:10.4334748Z  exit 0 2024-08-20T22:13:10.4335117Z fi 2024-08-20T22:13:10.4335525Z  2024-08-20T22:13:10.4336194Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:10.4337153Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:10.4337970Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:10.4338809Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:10.4339531Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:10.4340061Z else 2024-08-20T22:13:10.4340674Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:10.4341499Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:10.4342102Z fi 2024-08-20T22:13:10.4342522Z  2024-08-20T22:13:10.4342929Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:10.4343538Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4344176Z  2024-08-20T22:13:10.4344939Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:10.4345784Z  exit 0 2024-08-20T22:13:10.4346367Z fi 2024-08-20T22:13:10.4346741Z  2024-08-20T22:13:10.4347278Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:10.4348404Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:10.4349320Z  exit 1 2024-08-20T22:13:10.4349722Z fi 2024-08-20T22:13:10.4350176Z  2024-08-20T22:13:10.4350732Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:10.4351829Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:10.4352864Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:10.4353954Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:10.4355256Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:10.4356166Z fi 2024-08-20T22:13:10.4356577Z  2024-08-20T22:13:10.4357094Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:10.4365738Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:10.4366371Z env: 2024-08-20T22:13:10.4366757Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:10.4367248Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:10.4367838Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:10.4369250Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4370693Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:10.4371637Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.4372289Z ##[endgroup] 2024-08-20T22:13:10.4402852Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.4403633Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.4405661Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:10.4407254Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:10.9735320Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:10.9736577Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:10.9737674Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:10.9738291Z 2024-08-20T22:13:10.9738453Z Login Succeeded 2024-08-20T22:13:10.9760374Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.1707008Z { 2024-08-20T22:13:11.1707770Z "schemaVersion": 2, 2024-08-20T22:13:11.1709352Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:11.1710611Z "config": { 2024-08-20T22:13:11.1711444Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:11.1714479Z "size": 48711, 2024-08-20T22:13:11.1715180Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:11.1716362Z }, 2024-08-20T22:13:11.1716692Z "layers": [ 2024-08-20T22:13:11.1717319Z { 2024-08-20T22:13:11.1718227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1718862Z "size": 28580681, 2024-08-20T22:13:11.1719618Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:11.1721248Z }, 2024-08-20T22:13:11.1721790Z { 2024-08-20T22:13:11.1722448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1723264Z "size": 7943451, 2024-08-20T22:13:11.1723997Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:11.1725431Z }, 2024-08-20T22:13:11.1726198Z { 2024-08-20T22:13:11.1726855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1727714Z "size": 55728572, 2024-08-20T22:13:11.1729373Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:11.1730743Z }, 2024-08-20T22:13:11.1732114Z { 2024-08-20T22:13:11.1733006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1734057Z "size": 186, 2024-08-20T22:13:11.1734675Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:11.1735415Z }, 2024-08-20T22:13:11.1735814Z { 2024-08-20T22:13:11.1736933Z + exit 0 2024-08-20T22:13:11.1737925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1739533Z "size": 6886, 2024-08-20T22:13:11.1740703Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:11.1742107Z }, 2024-08-20T22:13:11.1742610Z { 2024-08-20T22:13:11.1743282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1743945Z "size": 1329001756, 2024-08-20T22:13:11.1744633Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:11.1745452Z }, 2024-08-20T22:13:11.1745766Z { 2024-08-20T22:13:11.1746490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1747273Z "size": 62501, 2024-08-20T22:13:11.1747932Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:11.1748614Z }, 2024-08-20T22:13:11.1749060Z { 2024-08-20T22:13:11.1749787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1750519Z "size": 1684, 2024-08-20T22:13:11.1751207Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:11.1751922Z }, 2024-08-20T22:13:11.1752307Z { 2024-08-20T22:13:11.1752915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1753548Z "size": 1523, 2024-08-20T22:13:11.1754271Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:11.1755072Z }, 2024-08-20T22:13:11.1755392Z { 2024-08-20T22:13:11.1755964Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1756696Z "size": 2528295403, 2024-08-20T22:13:11.1757397Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:11.1758085Z }, 2024-08-20T22:13:11.1758532Z { 2024-08-20T22:13:11.1759103Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1759831Z "size": 86016, 2024-08-20T22:13:11.1760463Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:11.1761245Z }, 2024-08-20T22:13:11.1761658Z { 2024-08-20T22:13:11.1762172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1762857Z "size": 1822, 2024-08-20T22:13:11.1763561Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:11.1764312Z }, 2024-08-20T22:13:11.1764628Z { 2024-08-20T22:13:11.1765227Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1765912Z "size": 246761563, 2024-08-20T22:13:11.1766590Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:11.1767338Z }, 2024-08-20T22:13:11.1767719Z { 2024-08-20T22:13:11.1768298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1769052Z "size": 544, 2024-08-20T22:13:11.1769650Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:11.1770357Z }, 2024-08-20T22:13:11.1770794Z { 2024-08-20T22:13:11.1771311Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1771977Z "size": 1258, 2024-08-20T22:13:11.1772695Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:11.1773408Z }, 2024-08-20T22:13:11.1773734Z { 2024-08-20T22:13:11.1774350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1775043Z "size": 484, 2024-08-20T22:13:11.1775695Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:11.1776406Z }, 2024-08-20T22:13:11.1776760Z { 2024-08-20T22:13:11.1777327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1778012Z "size": 91711766, 2024-08-20T22:13:11.1778681Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:11.1779418Z }, 2024-08-20T22:13:11.1779839Z { 2024-08-20T22:13:11.1780354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1781199Z "size": 3390, 2024-08-20T22:13:11.1781943Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:11.1782679Z }, 2024-08-20T22:13:11.1783000Z { 2024-08-20T22:13:11.1783603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1784288Z "size": 1909, 2024-08-20T22:13:11.1784974Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:11.1785648Z }, 2024-08-20T22:13:11.1786117Z { 2024-08-20T22:13:11.1786739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1787368Z "size": 701, 2024-08-20T22:13:11.1788042Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:11.1788928Z }, 2024-08-20T22:13:11.1789336Z { 2024-08-20T22:13:11.1789848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1790575Z "size": 2864254879, 2024-08-20T22:13:11.1791291Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:11.1792030Z }, 2024-08-20T22:13:11.1792426Z { 2024-08-20T22:13:11.1792986Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1793664Z "size": 380, 2024-08-20T22:13:11.1794347Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:11.1795078Z }, 2024-08-20T22:13:11.1795446Z { 2024-08-20T22:13:11.1796109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1796747Z "size": 12900, 2024-08-20T22:13:11.1797429Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:11.1798281Z }, 2024-08-20T22:13:11.1798637Z { 2024-08-20T22:13:11.1799208Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1799946Z "size": 803, 2024-08-20T22:13:11.1800607Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:11.1801340Z }, 2024-08-20T22:13:11.1801761Z { 2024-08-20T22:13:11.1802326Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1802953Z "size": 106, 2024-08-20T22:13:11.1803684Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:11.1804420Z }, 2024-08-20T22:13:11.1804732Z { 2024-08-20T22:13:11.1805327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1806009Z "size": 503, 2024-08-20T22:13:11.1806698Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:11.1807377Z }, 2024-08-20T22:13:11.1807753Z { 2024-08-20T22:13:11.1808347Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1809049Z "size": 121477267, 2024-08-20T22:13:11.1809675Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:11.1810474Z }, 2024-08-20T22:13:11.1810857Z { 2024-08-20T22:13:11.1811361Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1812080Z "size": 109, 2024-08-20T22:13:11.1812758Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:11.1813476Z }, 2024-08-20T22:13:11.1813840Z { 2024-08-20T22:13:11.1814400Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1815061Z "size": 490, 2024-08-20T22:13:11.1815842Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:11.1816547Z }, 2024-08-20T22:13:11.1816926Z { 2024-08-20T22:13:11.1817505Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1818162Z "size": 297, 2024-08-20T22:13:11.1818789Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:11.1819676Z }, 2024-08-20T22:13:11.1819999Z { 2024-08-20T22:13:11.1820603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1821339Z "size": 103, 2024-08-20T22:13:11.1821944Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:11.1822686Z }, 2024-08-20T22:13:11.1823096Z { 2024-08-20T22:13:11.1823673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1824324Z "size": 1473, 2024-08-20T22:13:11.1825042Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:11.1825778Z }, 2024-08-20T22:13:11.1826300Z { 2024-08-20T22:13:11.1826924Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1827624Z "size": 594729516, 2024-08-20T22:13:11.1828345Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:11.1829315Z }, 2024-08-20T22:13:11.1829642Z { 2024-08-20T22:13:11.1830250Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1830985Z "size": 164, 2024-08-20T22:13:11.1831617Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:11.1832384Z }, 2024-08-20T22:13:11.1832757Z { 2024-08-20T22:13:11.1834771Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1835442Z "size": 564, 2024-08-20T22:13:11.1836105Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:11.1836822Z }, 2024-08-20T22:13:11.1837253Z { 2024-08-20T22:13:11.1837751Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1838422Z "size": 35874203, 2024-08-20T22:13:11.1839167Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:11.1839875Z }, 2024-08-20T22:13:11.1840235Z { 2024-08-20T22:13:11.1840872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1841533Z "size": 104, 2024-08-20T22:13:11.1842129Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:11.1842932Z }, 2024-08-20T22:13:11.1843282Z { 2024-08-20T22:13:11.1843852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1844532Z "size": 425, 2024-08-20T22:13:11.1845191Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:11.1845957Z }, 2024-08-20T22:13:11.1846327Z { 2024-08-20T22:13:11.1846909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1847606Z "size": 20262183, 2024-08-20T22:13:11.1848313Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:11.1848985Z }, 2024-08-20T22:13:11.1849355Z { 2024-08-20T22:13:11.1849942Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1850673Z "size": 438, 2024-08-20T22:13:11.1851278Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:11.1852048Z }, 2024-08-20T22:13:11.1852431Z { 2024-08-20T22:13:11.1853098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1853757Z "size": 701, 2024-08-20T22:13:11.1854420Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:11.1855195Z }, 2024-08-20T22:13:11.1855506Z { 2024-08-20T22:13:11.1856061Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1856794Z "size": 141, 2024-08-20T22:13:11.1857469Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:11.1858163Z }, 2024-08-20T22:13:11.1858568Z { 2024-08-20T22:13:11.1859140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1859961Z "size": 135, 2024-08-20T22:13:11.1860685Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:11.1861397Z }, 2024-08-20T22:13:11.1861751Z { 2024-08-20T22:13:11.1862329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1862996Z "size": 4838068375, 2024-08-20T22:13:11.1863657Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:11.1864441Z }, 2024-08-20T22:13:11.1864748Z { 2024-08-20T22:13:11.1865286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1866109Z "size": 194, 2024-08-20T22:13:11.1866748Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:11.1867617Z }, 2024-08-20T22:13:11.1868030Z { 2024-08-20T22:13:11.1868573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1869263Z "size": 564, 2024-08-20T22:13:11.1869940Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:11.1870687Z }, 2024-08-20T22:13:11.1871042Z { 2024-08-20T22:13:11.1871599Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1872291Z "size": 43162921, 2024-08-20T22:13:11.1872948Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:11.1873705Z }, 2024-08-20T22:13:11.1874043Z { 2024-08-20T22:13:11.1874616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1875315Z "size": 106, 2024-08-20T22:13:11.1875967Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:11.1876648Z }, 2024-08-20T22:13:11.1877066Z { 2024-08-20T22:13:11.1877608Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1878320Z "size": 1380, 2024-08-20T22:13:11.1878935Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:11.1879633Z }, 2024-08-20T22:13:11.1880041Z { 2024-08-20T22:13:11.1880623Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1881314Z "size": 701, 2024-08-20T22:13:11.1882006Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:11.1882755Z }, 2024-08-20T22:13:11.1883071Z { 2024-08-20T22:13:11.1883688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1884357Z "size": 139, 2024-08-20T22:13:11.1884987Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:11.1885747Z }, 2024-08-20T22:13:11.1886097Z { 2024-08-20T22:13:11.1886628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1887329Z "size": 118, 2024-08-20T22:13:11.1887985Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:11.1888726Z }, 2024-08-20T22:13:11.1889155Z { 2024-08-20T22:13:11.1889658Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1890339Z "size": 4324672728, 2024-08-20T22:13:11.1891064Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:11.1891781Z }, 2024-08-20T22:13:11.1892112Z { 2024-08-20T22:13:11.1892699Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1893369Z "size": 174, 2024-08-20T22:13:11.1894028Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:11.1894759Z }, 2024-08-20T22:13:11.1895139Z { 2024-08-20T22:13:11.1895698Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1896373Z "size": 907, 2024-08-20T22:13:11.1897046Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:11.1897882Z }, 2024-08-20T22:13:11.1898289Z { 2024-08-20T22:13:11.1898819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1899483Z "size": 701, 2024-08-20T22:13:11.1900199Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:11.1900915Z }, 2024-08-20T22:13:11.1901224Z { 2024-08-20T22:13:11.1901828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1902509Z "size": 134, 2024-08-20T22:13:11.1903112Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:11.1903898Z }, 2024-08-20T22:13:11.1904238Z { 2024-08-20T22:13:11.1904830Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1905562Z "size": 32, 2024-08-20T22:13:11.1906284Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:11.1907078Z }, 2024-08-20T22:13:11.1907426Z { 2024-08-20T22:13:11.1907922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1908646Z "size": 158, 2024-08-20T22:13:11.1909327Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:11.1910065Z }, 2024-08-20T22:13:11.1910450Z { 2024-08-20T22:13:11.1911015Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1911689Z "size": 1841, 2024-08-20T22:13:11.1912354Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:11.1913069Z }, 2024-08-20T22:13:11.1913439Z { 2024-08-20T22:13:11.1914044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1914674Z "size": 7529776, 2024-08-20T22:13:11.1915351Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:11.1916152Z }, 2024-08-20T22:13:11.1916513Z { 2024-08-20T22:13:11.1917015Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1917733Z "size": 163, 2024-08-20T22:13:11.1918390Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:11.1919048Z }, 2024-08-20T22:13:11.1919458Z { 2024-08-20T22:13:11.1920015Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1920674Z "size": 7943, 2024-08-20T22:13:11.1921331Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:11.1922066Z }, 2024-08-20T22:13:11.1922424Z { 2024-08-20T22:13:11.1923083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1923719Z "size": 8068, 2024-08-20T22:13:11.1924374Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:11.1925164Z }, 2024-08-20T22:13:11.1925511Z { 2024-08-20T22:13:11.1926018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1926745Z "size": 302, 2024-08-20T22:13:11.1927384Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:11.1928053Z }, 2024-08-20T22:13:11.1928474Z { 2024-08-20T22:13:11.1929344Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1930125Z "size": 7629638, 2024-08-20T22:13:11.1930732Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:11.1931439Z }, 2024-08-20T22:13:11.1931864Z { 2024-08-20T22:13:11.1932411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1933041Z "size": 108, 2024-08-20T22:13:11.1933765Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:11.1934485Z }, 2024-08-20T22:13:11.1934796Z { 2024-08-20T22:13:11.1935415Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1936228Z "size": 54145661, 2024-08-20T22:13:11.1936965Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:11.1937721Z }, 2024-08-20T22:13:11.1938072Z { 2024-08-20T22:13:11.1938644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1939368Z "size": 473, 2024-08-20T22:13:11.1939987Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:11.1940791Z }, 2024-08-20T22:13:11.1941200Z { 2024-08-20T22:13:11.1941707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1942405Z "size": 1374858766, 2024-08-20T22:13:11.1943149Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:11.1944029Z }, 2024-08-20T22:13:11.1944388Z { 2024-08-20T22:13:11.1944992Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1945678Z "size": 106, 2024-08-20T22:13:11.1946446Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:11.1947218Z }, 2024-08-20T22:13:11.1947568Z { 2024-08-20T22:13:11.1948099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1948817Z "size": 559, 2024-08-20T22:13:11.1949407Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:11.1950111Z }, 2024-08-20T22:13:11.1950529Z { 2024-08-20T22:13:11.1951072Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1951727Z "size": 46248488, 2024-08-20T22:13:11.1952453Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:11.1953172Z }, 2024-08-20T22:13:11.1953504Z { 2024-08-20T22:13:11.1954097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1954754Z "size": 111, 2024-08-20T22:13:11.1955472Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:11.1956152Z }, 2024-08-20T22:13:11.1956602Z { 2024-08-20T22:13:11.1957178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1957896Z "size": 32, 2024-08-20T22:13:11.1958532Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:11.1959308Z }, 2024-08-20T22:13:11.1959666Z { 2024-08-20T22:13:11.1960191Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1960909Z "size": 32, 2024-08-20T22:13:11.1961551Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:11.1962301Z }, 2024-08-20T22:13:11.1962675Z { 2024-08-20T22:13:11.1963238Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1963901Z "size": 32, 2024-08-20T22:13:11.1964613Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:11.1965360Z }, 2024-08-20T22:13:11.1965716Z { 2024-08-20T22:13:11.1966311Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:11.1967001Z "size": 32, 2024-08-20T22:13:11.1967612Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:11.1968385Z } 2024-08-20T22:13:11.1968759Z ] 2024-08-20T22:13:11.1969064Z } 2024-08-20T22:13:11.2281988Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:11.2282494Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:11.2283054Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:11.2292401Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:11.2292930Z env: 2024-08-20T22:13:11.2293216Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.2294398Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.2295607Z ##[endgroup] 2024-08-20T22:13:11.2329882Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.2384763Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:11.2385356Z with: 2024-08-20T22:13:11.2386562Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.2387901Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:11.2388424Z env: 2024-08-20T22:13:11.2388899Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.2389251Z ##[endgroup] 2024-08-20T22:13:11.2407049Z ##[group]Run set -x 2024-08-20T22:13:11.2407409Z set -x 2024-08-20T22:13:11.2407736Z set +e 2024-08-20T22:13:11.2408052Z  2024-08-20T22:13:11.2408351Z login() { 2024-08-20T22:13:11.2409047Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:11.2409796Z } 2024-08-20T22:13:11.2410098Z  2024-08-20T22:13:11.2410425Z retry () { 2024-08-20T22:13:11.2410842Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:11.2411309Z } 2024-08-20T22:13:11.2411604Z  2024-08-20T22:13:11.2411942Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:11.2412377Z  2024-08-20T22:13:11.2412672Z set -e 2024-08-20T22:13:11.2413179Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:11.2413918Z # only pull docker image if it's not available locally 2024-08-20T22:13:11.2414708Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:11.2415431Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:11.2415898Z fi 2024-08-20T22:13:11.2424846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:11.2425355Z env: 2024-08-20T22:13:11.2425646Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:11.2426939Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.2428276Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:11.2429160Z ##[endgroup] 2024-08-20T22:13:11.2458972Z + set +e 2024-08-20T22:13:11.2459531Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:11.2460215Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:11.2462555Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:11.2464449Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:11.7801961Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:11.7803096Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:11.7804248Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:11.7804867Z 2024-08-20T22:13:11.7804996Z Login Succeeded 2024-08-20T22:13:11.7824419Z + set -e 2024-08-20T22:13:11.7825762Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.7989499Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:11.7991696Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:12.0555637Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:12.0557055Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:12.0557570Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:12.0558450Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:12.0558970Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:12.0559490Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:12.0559996Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:12.0560504Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:12.0561442Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:12.0561942Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:12.0562484Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:12.0562971Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:12.0563755Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:12.0564269Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:12.0564775Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:12.0565268Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:12.0565737Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:12.0566123Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:12.0566492Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:12.0566873Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:12.0567266Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:12.0567698Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:12.0568100Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:12.0568512Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:12.0568931Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:12.0569327Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:12.0569754Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:12.0570133Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:12.0570576Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:12.0570961Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:12.0571410Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:12.0571783Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:12.0572158Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:12.0572556Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:12.0572981Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:12.0573375Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:12.0573816Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:12.0574201Z 019620471c25: Pulling fs layer 2024-08-20T22:13:12.0574623Z a41a8d1c11c8: Waiting 2024-08-20T22:13:12.0574968Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:12.0575396Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:12.0575775Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:12.0576135Z 7ed32bc8e469: Waiting 2024-08-20T22:13:12.0576462Z ca426ffc9d2b: Waiting 2024-08-20T22:13:12.0576781Z d8d1234baab3: Waiting 2024-08-20T22:13:12.0577119Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:12.0577510Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:12.0577874Z 4373aa743caf: Waiting 2024-08-20T22:13:12.0578219Z 2df849f72ba4: Waiting 2024-08-20T22:13:12.0578589Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:12.0578985Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:12.0579395Z 385d0421d67c: Waiting 2024-08-20T22:13:12.0579717Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:12.0580117Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:12.0580495Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:12.0580915Z 0bf61114d37b: Waiting 2024-08-20T22:13:12.0581241Z ec1e7978c1fe: Waiting 2024-08-20T22:13:12.0581587Z b119c4fc6c58: Waiting 2024-08-20T22:13:12.0581953Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:12.0582339Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:12.0582762Z b57676e46aee: Waiting 2024-08-20T22:13:12.0583096Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:12.0583525Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:12.0583885Z 0c1227890755: Waiting 2024-08-20T22:13:12.0584272Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:12.0584648Z 33b639226eee: Waiting 2024-08-20T22:13:12.0584993Z f957c1624481: Pulling fs layer 2024-08-20T22:13:12.0585392Z d3547bd43146: Waiting 2024-08-20T22:13:12.0585725Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:12.0586270Z 7cbfce999d76: Waiting 2024-08-20T22:13:12.0586591Z 34c47d63c5de: Waiting 2024-08-20T22:13:12.0586976Z 560cd6122a38: Waiting 2024-08-20T22:13:12.0587313Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:12.0587735Z 6efb96fc87ea: Waiting 2024-08-20T22:13:12.0588214Z 1fef4bd19792: Waiting 2024-08-20T22:13:12.0588605Z 829ed7d91e38: Waiting 2024-08-20T22:13:12.0588925Z 5a4eb19d7219: Waiting 2024-08-20T22:13:12.0589272Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:12.0589703Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:12.0590082Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:12.0590514Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:12.0590892Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:12.0591316Z da6a838a66c7: Waiting 2024-08-20T22:13:12.0591769Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:12.0592199Z cbb79a705353: Waiting 2024-08-20T22:13:12.0592538Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:12.0592988Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:12.0593365Z ee1273153675: Pulling fs layer 2024-08-20T22:13:12.0593802Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:12.0594188Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:12.0594614Z 302c8c177944: Waiting 2024-08-20T22:13:12.0594944Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:12.0595366Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:12.0595758Z 02970347dff6: Waiting 2024-08-20T22:13:12.0596094Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:12.0596528Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:12.0596928Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:12.0597365Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:12.0597739Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:12.0598111Z 4f4fb700ef54: Waiting 2024-08-20T22:13:12.0598433Z 78ee77eddef2: Waiting 2024-08-20T22:13:12.0598804Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:12.0599149Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:12.0599457Z 37513e9b43d7: Waiting 2024-08-20T22:13:12.0599832Z 0479fbac3361: Waiting 2024-08-20T22:13:12.0600152Z 75b2abb16c68: Waiting 2024-08-20T22:13:12.0600501Z 1811ff4d3556: Waiting 2024-08-20T22:13:12.0600836Z 33444a5a7e01: Waiting 2024-08-20T22:13:12.0601148Z 0f085d4fa279: Waiting 2024-08-20T22:13:12.0601521Z 4a0412373305: Waiting 2024-08-20T22:13:12.0601825Z 7754ee068b33: Waiting 2024-08-20T22:13:12.0602178Z f957c1624481: Waiting 2024-08-20T22:13:12.0602523Z 916c5e858016: Waiting 2024-08-20T22:13:12.0602829Z c42b3602a1fb: Waiting 2024-08-20T22:13:12.0603208Z 11fad972ef6c: Waiting 2024-08-20T22:13:12.0603529Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:12.0603878Z a8c1e85b5e14: Waiting 2024-08-20T22:13:12.0604222Z 019620471c25: Waiting 2024-08-20T22:13:12.0604534Z b10de02255c0: Waiting 2024-08-20T22:13:12.0604910Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:12.0605230Z a7a5392e2bf6: Waiting 2024-08-20T22:13:12.0605573Z d79fd7d52698: Waiting 2024-08-20T22:13:12.0605921Z 768c31dca016: Waiting 2024-08-20T22:13:12.0606239Z 9752d6fbbcab: Waiting 2024-08-20T22:13:12.0606614Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:12.0606932Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:12.0607273Z 83808b2eff4d: Waiting 2024-08-20T22:13:12.0607626Z eb09cbf3e416: Waiting 2024-08-20T22:13:12.0607941Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:12.0608318Z ee1273153675: Waiting 2024-08-20T22:13:12.0608632Z 5d115b29b5d5: Waiting 2024-08-20T22:13:12.0608972Z 829c0e19f4b7: Waiting 2024-08-20T22:13:12.0609324Z f1d6874dee9c: Waiting 2024-08-20T22:13:12.1911317Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:12.1911926Z 224fe954d725: Download complete 2024-08-20T22:13:12.2769044Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:12.2769650Z d527cbbb87e3: Download complete 2024-08-20T22:13:12.3508950Z b57676e46aee: Download complete 2024-08-20T22:13:12.3889210Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:12.3889767Z 7a2c55901189: Download complete 2024-08-20T22:13:12.4669452Z a41a8d1c11c8: Download complete 2024-08-20T22:13:12.5386286Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:12.5386897Z 0c1227890755: Download complete 2024-08-20T22:13:12.5997690Z d8d1234baab3: Download complete 2024-08-20T22:13:12.6580395Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:12.6580995Z 75722010b82e: Download complete 2024-08-20T22:13:12.7315686Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:12.7316313Z ec1e7978c1fe: Download complete 2024-08-20T22:13:12.7933890Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:12.7934403Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:13.5498862Z 7a2c55901189: Pull complete 2024-08-20T22:13:13.8682365Z 224fe954d725: Pull complete 2024-08-20T22:13:14.7864725Z 75722010b82e: Pull complete 2024-08-20T22:13:14.8054926Z d527cbbb87e3: Pull complete 2024-08-20T22:13:14.8233032Z b57676e46aee: Pull complete 2024-08-20T22:13:15.3073853Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:15.3074498Z ca426ffc9d2b: Download complete 2024-08-20T22:13:15.3740984Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:15.3741434Z 4373aa743caf: Download complete 2024-08-20T22:13:15.4498180Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:15.4498633Z 75b2abb16c68: Download complete 2024-08-20T22:13:15.5211650Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:16.4912342Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:16.4912822Z 0bf61114d37b: Download complete 2024-08-20T22:13:16.5532563Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:16.5533217Z 2df849f72ba4: Download complete 2024-08-20T22:13:16.6293474Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:16.6958796Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:13:16.6959268Z 1fef4bd19792: Download complete 2024-08-20T22:13:25.6930248Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:25.6930745Z a8c1e85b5e14: Download complete 2024-08-20T22:13:25.7670813Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:25.7671305Z b119c4fc6c58: Download complete 2024-08-20T22:13:25.8345110Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:25.8345569Z 33b639226eee: Download complete 2024-08-20T22:13:25.9103870Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:25.9105041Z d3547bd43146: Download complete 2024-08-20T22:13:25.9889620Z 5645520256fa: Download complete 2024-08-20T22:13:26.0574267Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:13:26.0574762Z 829ed7d91e38: Download complete 2024-08-20T22:13:27.3275370Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:27.3275894Z 385d0421d67c: Download complete 2024-08-20T22:13:27.3933292Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:27.3933730Z 7cbfce999d76: Download complete 2024-08-20T22:13:27.4559436Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:27.4559885Z 6658e558ebe4: Download complete 2024-08-20T22:13:27.5410686Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:27.5411222Z 33444a5a7e01: Download complete 2024-08-20T22:13:27.6476415Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:27.6476844Z 7c47122f4e09: Download complete 2024-08-20T22:13:27.7150884Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:27.7151454Z 1811ff4d3556: Download complete 2024-08-20T22:13:33.7146511Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:33.7147130Z 7754ee068b33: Download complete 2024-08-20T22:13:33.7938615Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:33.7939186Z 0479fbac3361: Download complete 2024-08-20T22:13:33.8590073Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:33.8590686Z f1d6874dee9c: Download complete 2024-08-20T22:13:34.2762609Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:34.2763281Z afc429e87b59: Download complete 2024-08-20T22:13:34.3426627Z 019620471c25: Verifying Checksum 2024-08-20T22:13:34.3427203Z 019620471c25: Download complete 2024-08-20T22:13:34.4215141Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:34.6815204Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:34.6815956Z 829c0e19f4b7: Download complete 2024-08-20T22:13:34.7427046Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:34.7427716Z 560cd6122a38: Download complete 2024-08-20T22:13:34.8101514Z 5a4eb19d7219: Download complete 2024-08-20T22:13:34.8765988Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:34.8766526Z da6a838a66c7: Download complete 2024-08-20T22:13:37.9253003Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:37.9253522Z 7ed32bc8e469: Download complete 2024-08-20T22:13:38.0132526Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:38.1133095Z 0f085d4fa279: Download complete 2024-08-20T22:13:38.6970248Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:38.6971251Z 4a0412373305: Download complete 2024-08-20T22:13:38.7697366Z 302c8c177944: Download complete 2024-08-20T22:13:38.8315644Z 02970347dff6: Download complete 2024-08-20T22:13:38.8763720Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:38.9013779Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:38.9014373Z 916c5e858016: Download complete 2024-08-20T22:13:38.9812728Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:38.9813311Z c42b3602a1fb: Download complete 2024-08-20T22:13:39.0252433Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:39.1750920Z 0c1227890755: Pull complete 2024-08-20T22:13:39.3496795Z d8d1234baab3: Pull complete 2024-08-20T22:13:45.3997783Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:45.4714110Z 6efb96fc87ea: Download complete 2024-08-20T22:13:45.4714768Z f957c1624481: Download complete 2024-08-20T22:13:45.5395904Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:45.5396445Z 768c31dca016: Download complete 2024-08-20T22:13:45.5991539Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:45.6074572Z 4f4fb700ef54: Download complete 2024-08-20T22:13:45.6662534Z 78ee77eddef2: Download complete 2024-08-20T22:13:45.7329412Z 37513e9b43d7: Download complete 2024-08-20T22:13:45.8515053Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:45.8515922Z a7a5392e2bf6: Download complete 2024-08-20T22:13:45.9212822Z b10de02255c0: Download complete 2024-08-20T22:13:45.9959532Z e56eeff2db93: Download complete 2024-08-20T22:13:46.0625161Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:46.1462525Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:46.1463204Z eb09cbf3e416: Download complete 2024-08-20T22:13:46.2721957Z ee1273153675: Verifying Checksum 2024-08-20T22:13:46.2722585Z ee1273153675: Download complete 2024-08-20T22:13:46.3726328Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:13:46.3726814Z e22979f9e0ee: Download complete 2024-08-20T22:13:46.9599146Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:46.9599765Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:47.0290184Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:47.0290875Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:14.4133775Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:14.4134412Z 5d115b29b5d5: Download complete 2024-08-20T22:14:14.4813465Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:14.5523777Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:14.5524245Z d79fd7d52698: Download complete 2024-08-20T22:14:15.7240404Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:15.7241060Z 9752d6fbbcab: Download complete 2024-08-20T22:14:15.7897178Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:15.7897624Z 83808b2eff4d: Download complete 2024-08-20T22:14:52.8408556Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:14:54.6380645Z cbb79a705353: Verifying Checksum 2024-08-20T22:14:54.6381124Z cbb79a705353: Download complete 2024-08-20T22:15:09.2441210Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:09.4689049Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:09.6995972Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:18.3490967Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:18.5749540Z 4373aa743caf: Pull complete 2024-08-20T22:15:18.8094532Z 75b2abb16c68: Pull complete 2024-08-20T22:15:19.0346495Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:21.6172170Z 0bf61114d37b: Pull complete 2024-08-20T22:15:21.7918224Z 2df849f72ba4: Pull complete 2024-08-20T22:15:21.9781007Z 34c47d63c5de: Pull complete 2024-08-20T22:15:22.1394250Z 1fef4bd19792: Pull complete 2024-08-20T22:16:21.0794924Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:21.3031631Z b119c4fc6c58: Pull complete 2024-08-20T22:16:21.4821063Z 33b639226eee: Pull complete 2024-08-20T22:16:21.7134729Z d3547bd43146: Pull complete 2024-08-20T22:16:21.9367829Z 5645520256fa: Pull complete 2024-08-20T22:16:22.0881154Z 829ed7d91e38: Pull complete 2024-08-20T22:16:25.6671399Z 385d0421d67c: Pull complete 2024-08-20T22:16:25.8826918Z 7cbfce999d76: Pull complete 2024-08-20T22:16:26.1093802Z 6658e558ebe4: Pull complete 2024-08-20T22:16:26.3357939Z 33444a5a7e01: Pull complete 2024-08-20T22:16:26.5580146Z 7c47122f4e09: Pull complete 2024-08-20T22:16:26.7936934Z 1811ff4d3556: Pull complete 2024-08-20T22:16:37.4721402Z 7754ee068b33: Pull complete 2024-08-20T22:16:37.6820210Z 0479fbac3361: Pull complete 2024-08-20T22:16:37.7956206Z f1d6874dee9c: Pull complete 2024-08-20T22:16:38.7027149Z afc429e87b59: Pull complete 2024-08-20T22:16:38.9314307Z 019620471c25: Pull complete 2024-08-20T22:16:39.1294005Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:39.6062361Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:39.7777582Z 560cd6122a38: Pull complete 2024-08-20T22:16:40.1510375Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:40.3036702Z da6a838a66c7: Pull complete 2024-08-20T22:17:18.9972267Z cbb79a705353: Pull complete 2024-08-20T22:17:19.2177742Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:19.4433987Z 0f085d4fa279: Pull complete 2024-08-20T22:17:21.8628221Z 4a0412373305: Pull complete 2024-08-20T22:17:21.8776578Z 302c8c177944: Pull complete 2024-08-20T22:17:21.8923890Z 02970347dff6: Pull complete 2024-08-20T22:17:21.9217002Z 916c5e858016: Pull complete 2024-08-20T22:17:21.9373738Z c42b3602a1fb: Pull complete 2024-08-20T22:18:43.7292611Z 11fad972ef6c: Pull complete 2024-08-20T22:18:43.9385887Z f957c1624481: Pull complete 2024-08-20T22:18:44.1567635Z 768c31dca016: Pull complete 2024-08-20T22:18:44.5780804Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:44.8067721Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:45.0460064Z 78ee77eddef2: Pull complete 2024-08-20T22:18:45.2765282Z 37513e9b43d7: Pull complete 2024-08-20T22:18:45.6508063Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:45.8688802Z b10de02255c0: Pull complete 2024-08-20T22:18:46.0373087Z e56eeff2db93: Pull complete 2024-08-20T22:18:46.2667514Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:46.4973695Z eb09cbf3e416: Pull complete 2024-08-20T22:18:47.9086763Z ee1273153675: Pull complete 2024-08-20T22:18:48.1336359Z e22979f9e0ee: Pull complete 2024-08-20T22:18:50.3318476Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:50.5548782Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:03.9751602Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:04.2143055Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:04.4418782Z d79fd7d52698: Pull complete 2024-08-20T22:19:05.3323461Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:05.5603010Z 83808b2eff4d: Pull complete 2024-08-20T22:19:06.5282498Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:06.5708859Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:06.5925241Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:06.5985829Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:06.5986944Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:06.5997939Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:06.5998447Z env: 2024-08-20T22:19:06.5998741Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:06.5999105Z ##[endgroup] 2024-08-20T22:19:06.6203727Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:06.6204291Z with: 2024-08-20T22:19:06.6204584Z driver-version: 550.54.15 2024-08-20T22:19:06.6204922Z env: 2024-08-20T22:19:06.6205202Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:06.6205570Z ##[endgroup] 2024-08-20T22:19:06.6303655Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:06.6304222Z with: 2024-08-20T22:19:06.6304517Z timeout_minutes: 10 2024-08-20T22:19:06.6304864Z max_attempts: 3 2024-08-20T22:19:06.6339500Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:06.6373533Z retry_wait_seconds: 10 2024-08-20T22:19:06.6373922Z polling_interval_seconds: 1 2024-08-20T22:19:06.6374304Z warning_on_retry: true 2024-08-20T22:19:06.6374663Z continue_on_error: false 2024-08-20T22:19:06.6375003Z env: 2024-08-20T22:19:06.6375284Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:06.6375647Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:06.6376008Z ##[endgroup] 2024-08-20T22:19:06.7270592Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:06.7271218Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:06.7271759Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:07.0404748Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:07.0405304Z No packages marked for removal. 2024-08-20T22:19:07.0469083Z Dependencies resolved. 2024-08-20T22:19:07.0479033Z Nothing to do. 2024-08-20T22:19:07.0479363Z Complete! 2024-08-20T22:19:07.1423471Z + install_nvidia_driver_common 2024-08-20T22:19:07.1426868Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:07.1427567Z + lspci 2024-08-20T22:19:07.1427994Z Before installing NVIDIA driver 2024-08-20T22:19:07.1537785Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:07.1538704Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:07.1539830Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:07.1540937Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:07.1541805Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:07.1542618Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:07.1543384Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:07.1544226Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:07.1544846Z + lsmod 2024-08-20T22:19:07.1588180Z Module Size Used by 2024-08-20T22:19:07.1588792Z veth 36864 0 2024-08-20T22:19:07.1589180Z nvidia_modeset 1351680 0 2024-08-20T22:19:07.1589607Z video 65536 1 nvidia_modeset 2024-08-20T22:19:07.1590044Z wmi 36864 1 video 2024-08-20T22:19:07.1590450Z nvidia_uvm 4706304 0 2024-08-20T22:19:07.1590907Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:07.1591392Z drm 602112 1 nvidia 2024-08-20T22:19:07.1591886Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:07.1592616Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:07.1593237Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:07.1593729Z xt_conntrack 16384 1 2024-08-20T22:19:07.1594247Z nft_chain_nat 16384 3 2024-08-20T22:19:07.1594758Z xt_MASQUERADE 20480 1 2024-08-20T22:19:07.1595318Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:07.1595930Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:07.1596708Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:07.1597571Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:07.1598196Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:07.1599013Z xfrm_user 57344 1 2024-08-20T22:19:07.1599552Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:07.1600118Z xt_addrtype 16384 2 2024-08-20T22:19:07.1602006Z nft_compat 20480 4 2024-08-20T22:19:07.1602468Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:07.1603091Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:07.1603657Z br_netfilter 36864 0 2024-08-20T22:19:07.1604069Z bridge 307200 1 br_netfilter 2024-08-20T22:19:07.1604502Z stp 16384 1 bridge 2024-08-20T22:19:07.1605014Z llc 16384 2 bridge,stp 2024-08-20T22:19:07.1605639Z overlay 167936 0 2024-08-20T22:19:07.1606190Z tls 114688 0 2024-08-20T22:19:07.1606728Z nls_ascii 16384 1 2024-08-20T22:19:07.1607252Z nls_cp437 20480 1 2024-08-20T22:19:07.1607806Z vfat 24576 1 2024-08-20T22:19:07.1608371Z fat 86016 1 vfat 2024-08-20T22:19:07.1608908Z sunrpc 692224 1 2024-08-20T22:19:07.1609409Z ena 167936 0 2024-08-20T22:19:07.1609907Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:07.1610424Z ptp 36864 1 ena 2024-08-20T22:19:07.1610957Z pps_core 24576 1 ptp 2024-08-20T22:19:07.1611538Z aesni_intel 393216 0 2024-08-20T22:19:07.1612070Z i8042 45056 0 2024-08-20T22:19:07.1612587Z serio 28672 3 i8042 2024-08-20T22:19:07.1613162Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:07.1613725Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:07.1614421Z button 24576 0 2024-08-20T22:19:07.1614979Z sch_fq_codel 20480 17 2024-08-20T22:19:07.1615370Z dm_mod 188416 0 2024-08-20T22:19:07.1615739Z fuse 163840 1 2024-08-20T22:19:07.1616111Z configfs 57344 1 2024-08-20T22:19:07.1616481Z loop 36864 0 2024-08-20T22:19:07.1616859Z dax 45056 1 dm_mod 2024-08-20T22:19:07.1617368Z dmi_sysfs 20480 0 2024-08-20T22:19:07.1617905Z crc32_pclmul 16384 0 2024-08-20T22:19:07.1618300Z crc32c_intel 24576 0 2024-08-20T22:19:07.1618674Z efivarfs 24576 1 2024-08-20T22:19:07.1619040Z + modinfo nvidia 2024-08-20T22:19:07.1619757Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:07.1620461Z alias: char-major-195-* 2024-08-20T22:19:07.1620853Z version: 550.54.15 2024-08-20T22:19:07.1621358Z supported: external 2024-08-20T22:19:07.1621704Z license: NVIDIA 2024-08-20T22:19:07.1622091Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:07.1622579Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:07.1623043Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:07.1623507Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:07.1624020Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:07.1624516Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:07.1625020Z depends: i2c-core,drm 2024-08-20T22:19:07.1625381Z retpoline: Y 2024-08-20T22:19:07.1625691Z name: nvidia 2024-08-20T22:19:07.1626458Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:07.1627176Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:07.1627829Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:07.1628439Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:07.1629188Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:07.1629636Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:07.1630096Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:07.1630543Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:07.1631170Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:07.1631702Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:07.1632282Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:07.1632770Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:07.1633209Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:07.1633627Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:07.1634090Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:07.1634629Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:07.1635223Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:07.1635791Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:07.1636413Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:07.1637025Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:07.1637654Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:07.1638258Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:07.1638761Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:07.1639306Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:07.1639864Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:07.1640359Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:07.1640829Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:07.1641320Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:07.1641800Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:07.1642249Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:07.1642767Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:07.1643304Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:07.1643793Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:07.1644281Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:07.1644794Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:07.1645298Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:07.1645804Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:07.1646292Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:07.1646710Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:07.1647187Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:07.1647709Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:07.1648173Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:07.1648663Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:07.1649147Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:07.1649742Z parm: rm_firmware_active:charp 2024-08-20T22:19:07.1650165Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:07.1650573Z ++ command -v nvidia-smi 2024-08-20T22:19:07.1650989Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:07.1651357Z + set +e 2024-08-20T22:19:07.1651885Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:07.1911704Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:07.1912371Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:07.1912848Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:07.1913230Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:07.1913621Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:07.1914368Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:07.1915258Z + set -e 2024-08-20T22:19:07.1915748Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:07.1916369Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:07.1917067Z + post_install_nvidia_driver_common 2024-08-20T22:19:07.1919300Z + sudo modprobe nvidia 2024-08-20T22:19:07.2834923Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:07.2835391Z + lspci 2024-08-20T22:19:07.2835687Z After installing NVIDIA driver 2024-08-20T22:19:07.2941604Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:07.2942755Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:07.2943911Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:07.2944871Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:07.2945712Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:07.2946653Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:07.2947411Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:07.2948251Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:07.2948855Z + lsmod 2024-08-20T22:19:07.2974685Z Module Size Used by 2024-08-20T22:19:07.2975137Z veth 36864 0 2024-08-20T22:19:07.2975647Z nvidia_modeset 1351680 0 2024-08-20T22:19:07.2976192Z video 65536 1 nvidia_modeset 2024-08-20T22:19:07.2976872Z wmi 36864 1 video 2024-08-20T22:19:07.2977418Z nvidia_uvm 4706304 0 2024-08-20T22:19:07.2978019Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:07.2978655Z drm 602112 1 nvidia 2024-08-20T22:19:07.2979252Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:07.2979887Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:07.2980403Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:07.2980827Z xt_conntrack 16384 1 2024-08-20T22:19:07.2982678Z nft_chain_nat 16384 3 2024-08-20T22:19:07.2983061Z xt_MASQUERADE 20480 1 2024-08-20T22:19:07.2983523Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:07.2984017Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:07.2984611Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:07.2985265Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:07.2985744Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:07.2986177Z xfrm_user 57344 1 2024-08-20T22:19:07.2986689Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:07.2987114Z xt_addrtype 16384 2 2024-08-20T22:19:07.2987489Z nft_compat 20480 4 2024-08-20T22:19:07.2987953Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:07.2988577Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:07.2989139Z br_netfilter 36864 0 2024-08-20T22:19:07.2989557Z bridge 307200 1 br_netfilter 2024-08-20T22:19:07.2990219Z stp 16384 1 bridge 2024-08-20T22:19:07.2990657Z llc 16384 2 bridge,stp 2024-08-20T22:19:07.2991089Z overlay 167936 0 2024-08-20T22:19:07.2991462Z tls 114688 0 2024-08-20T22:19:07.2991842Z nls_ascii 16384 1 2024-08-20T22:19:07.2992299Z nls_cp437 20480 1 2024-08-20T22:19:07.2992830Z vfat 24576 1 2024-08-20T22:19:07.2993353Z fat 86016 1 vfat 2024-08-20T22:19:07.2993900Z sunrpc 692224 1 2024-08-20T22:19:07.2994389Z ena 167936 0 2024-08-20T22:19:07.2994819Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:07.2995212Z ptp 36864 1 ena 2024-08-20T22:19:07.2995618Z pps_core 24576 1 ptp 2024-08-20T22:19:07.2996017Z aesni_intel 393216 0 2024-08-20T22:19:07.2996382Z i8042 45056 0 2024-08-20T22:19:07.2996756Z serio 28672 3 i8042 2024-08-20T22:19:07.2997191Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:07.2997775Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:07.2998272Z button 24576 0 2024-08-20T22:19:07.2998652Z sch_fq_codel 20480 17 2024-08-20T22:19:07.2999036Z dm_mod 188416 0 2024-08-20T22:19:07.2999561Z fuse 163840 1 2024-08-20T22:19:07.2999934Z configfs 57344 1 2024-08-20T22:19:07.3000333Z loop 36864 0 2024-08-20T22:19:07.3000864Z dax 45056 1 dm_mod 2024-08-20T22:19:07.3001472Z dmi_sysfs 20480 0 2024-08-20T22:19:07.3001929Z crc32_pclmul 16384 0 2024-08-20T22:19:07.3002308Z crc32c_intel 24576 0 2024-08-20T22:19:07.3002690Z efivarfs 24576 1 2024-08-20T22:19:07.3003086Z + modinfo nvidia 2024-08-20T22:19:07.3003772Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:07.3004492Z alias: char-major-195-* 2024-08-20T22:19:07.3004893Z version: 550.54.15 2024-08-20T22:19:07.3005244Z supported: external 2024-08-20T22:19:07.3005598Z license: NVIDIA 2024-08-20T22:19:07.3005992Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:07.3006492Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:07.3006967Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:07.3007448Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:07.3007961Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:07.3008472Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:07.3008980Z depends: i2c-core,drm 2024-08-20T22:19:07.3009362Z retpoline: Y 2024-08-20T22:19:07.3009692Z name: nvidia 2024-08-20T22:19:07.3010332Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:07.3011057Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:07.3011741Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:07.3012369Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:07.3012841Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:07.3013296Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:07.3013784Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:07.3014253Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:07.3014715Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:07.3015274Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:07.3015870Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:07.3016377Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:07.3016829Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:07.3017276Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:07.3017762Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:07.3018322Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:07.3019063Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:07.3019660Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:07.3020302Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:07.3020937Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:07.3021579Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:07.3022205Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:07.3022730Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:07.3023292Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:07.3023861Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:07.3024383Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:07.3024875Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:07.3025388Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:07.3025879Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:07.3026450Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:07.3026987Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:07.3027553Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:07.3028099Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:07.3032798Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:07.3033369Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:07.3033865Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:07.3034387Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:07.3034880Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:07.3035391Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:07.3035875Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:07.3036356Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:07.3036827Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:07.3037316Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:07.3037806Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:07.3038277Z parm: rm_firmware_active:charp 2024-08-20T22:19:07.3038692Z + set +e 2024-08-20T22:19:07.3039043Z + nvidia-smi 2024-08-20T22:19:07.3208356Z Tue Aug 20 22:19:07 2024 2024-08-20T22:19:07.3209235Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:07.3210316Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:07.3211145Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:07.3211968Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:07.3212897Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:07.3213614Z | | | MIG M. | 2024-08-20T22:19:07.3214187Z |=========================================+========================+======================| 2024-08-20T22:19:07.3421629Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:07.3422401Z | 0% 26C P8 16W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:19:07.3423057Z | | | N/A | 2024-08-20T22:19:07.3423768Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:07.3425757Z 2024-08-20T22:19:07.3426646Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:07.3427278Z | Processes: | 2024-08-20T22:19:07.3428235Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:07.3429261Z | ID ID Usage | 2024-08-20T22:19:07.3429836Z |=========================================================================================| 2024-08-20T22:19:07.3431776Z | No running processes found | 2024-08-20T22:19:07.3432574Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:07.6369741Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:07.6555472Z NVIDIA A10G 2024-08-20T22:19:07.6616842Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:07.6617455Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:07.6617983Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:07.6618546Z + set -e 2024-08-20T22:19:07.6618953Z INFO: Ignoring allowed status 0 2024-08-20T22:19:07.6629294Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:07.6633211Z + sudo yum install -y yum-utils 2024-08-20T22:19:08.1080337Z Last metadata expiration check: 0:34:58 ago on Tue Aug 20 21:44:10 2024. 2024-08-20T22:19:08.1411570Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:08.1714631Z Dependencies resolved. 2024-08-20T22:19:08.1841294Z Nothing to do. 2024-08-20T22:19:08.1842238Z Complete! 2024-08-20T22:19:08.2894854Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:08.2896012Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.2897359Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.5815166Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:08.6518385Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:09.1369070Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:19:09.1594878Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:09.1888616Z Dependencies resolved. 2024-08-20T22:19:09.2011950Z Nothing to do. 2024-08-20T22:19:09.2012280Z Complete! 2024-08-20T22:19:09.3023094Z + sudo systemctl restart docker 2024-08-20T22:20:07.4903100Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:20:07.5138727Z Tue Aug 20 22:20:07 2024 2024-08-20T22:20:07.5139515Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:07.5140377Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:07.5141200Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:07.5142033Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:07.5143051Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:07.5143984Z | | | MIG M. | 2024-08-20T22:20:07.5144610Z |=========================================+========================+======================| 2024-08-20T22:20:07.5339327Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:07.5340117Z | 0% 26C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:20:07.5340782Z | | | N/A | 2024-08-20T22:20:07.5341655Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:07.5343750Z 2024-08-20T22:20:07.5344608Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:07.5346804Z | Processes: | 2024-08-20T22:20:07.5347564Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:07.5348428Z | ID ID Usage | 2024-08-20T22:20:07.5349185Z |=========================================================================================| 2024-08-20T22:20:07.5349866Z | No running processes found | 2024-08-20T22:20:07.5350677Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:08.7568976Z Command completed after 1 attempt(s). 2024-08-20T22:20:08.7655723Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:08.7656519Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:08.7657234Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:08.7657910Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:08.7672578Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:08.7673095Z env: 2024-08-20T22:20:08.7673397Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:08.7674085Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:08.7674598Z ##[endgroup] 2024-08-20T22:20:09.0102699Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:09.0272188Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T22:20:09.0277429Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T22:20:09.1523050Z Prepare all required actions 2024-08-20T22:20:09.1523740Z Getting action download info 2024-08-20T22:20:09.2512627Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:09.4486526Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:09.5888260Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:09.5888882Z with: 2024-08-20T22:20:09.5889288Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:09.5889818Z s3-bucket: gha-artifacts 2024-08-20T22:20:09.5890213Z env: 2024-08-20T22:20:09.5890538Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:09.5891048Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:09.5891601Z ##[endgroup] 2024-08-20T22:20:09.5928507Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:09.5929267Z with: 2024-08-20T22:20:09.5929667Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:09.5930141Z s3-bucket: gha-artifacts 2024-08-20T22:20:09.5930518Z region: us-east-1 2024-08-20T22:20:09.5930831Z env: 2024-08-20T22:20:09.5931122Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:09.5931579Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:09.5932077Z ##[endgroup] 2024-08-20T22:20:10.0820139Z (node:183536) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:10.0821578Z 2024-08-20T22:20:10.0822109Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:10.0823596Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:10.0825025Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:10.1460367Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:10.1461491Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0080588Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:25.0087019Z Artifact download has finished successfully 2024-08-20T22:20:25.0445766Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:25.0446219Z unzip -o artifacts.zip 2024-08-20T22:20:25.0455417Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:25.0455940Z env: 2024-08-20T22:20:25.0456243Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.0456719Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.0457226Z ##[endgroup] 2024-08-20T22:20:25.0511700Z Archive: artifacts.zip 2024-08-20T22:20:25.0513055Z creating: dist/ 2024-08-20T22:20:27.1843144Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.1843989Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.1844713Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.1845493Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.1846372Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.1852746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.1853921Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.1854976Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.1856364Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.1857528Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.1858846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.1860433Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.1861520Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.1862747Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.1864080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.1865391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.1867039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.1868294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.1869755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.1871235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.1872380Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.1873435Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.1912460Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.1952633Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.1954165Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2000987Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2002869Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2004658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2006472Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2008155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2009764Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2011515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2014629Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2016233Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2017716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2019009Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2020272Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2021737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2022978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2024235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2078461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2138891Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2140312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2141585Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2142644Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2143799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2144972Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.2146196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2147781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2149079Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.2150239Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.2151339Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2152470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.2153602Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2154716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.2155823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.2170922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.2314184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.2315466Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.2316856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.2318327Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.2319761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.2321050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.2322397Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.2323589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.2324751Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.2325931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.2327105Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.2343863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.2426942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.2428460Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.2429978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.2431333Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.2432483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.2433422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.2434366Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.2435404Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.2436385Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.2437354Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.2557245Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.2620781Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.2622445Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.2623890Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.2625561Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.2630057Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.2631230Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.2632353Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.2633385Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.2634526Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.2635686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.2637097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.2638179Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.2639254Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.2640557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.2642876Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.2644152Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.2645330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.2646533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.2647733Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.2648818Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.2649843Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.2689480Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.2728857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.2730619Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.2776688Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.2779282Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.2780942Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.2782454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.2783908Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.2785310Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.2786843Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.2788257Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.2789639Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.2790965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.2792220Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.2793453Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.2794704Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.2795935Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.2797178Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2851094Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2911423Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2912643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2913654Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2914542Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2915454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2916429Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.2917533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.2918750Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.2919925Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.2921023Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.2922172Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.2923323Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.2924463Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.2925762Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.2926889Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.2943953Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.3008044Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.3009794Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3012006Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3014825Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3016682Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3018510Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3019676Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.3020545Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.3021352Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.3022190Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.3070198Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.3070979Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.3071766Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.3072669Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.3079078Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.3080097Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.3081120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.3082210Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.3083267Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.3084460Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.3086111Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.3087327Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.3088494Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.3089934Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.3091363Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.3093033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.3094380Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.3095876Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.3097414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.3098621Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.3099933Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.3140379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.3180784Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.3184310Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.3226738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.3228837Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.3230717Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.3232694Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.3234453Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.3236230Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.3237965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.3239625Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.3241270Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.3242655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.3243998Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.3245316Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.3246640Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.3247929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.3249248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.3301473Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.3362077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.3363598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.3364948Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.3366112Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.3367312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.3368587Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.3369951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.3371424Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.3373047Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.3374245Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.3375461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.3376985Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.3378259Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.3379486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.3380703Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.3382002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.3503598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.3505105Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.3506708Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.3508267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.3509880Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.3511356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.3512664Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.3513983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.3515283Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.3516583Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.3517863Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.3533417Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.3588715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.3591815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.3594597Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.3596872Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.3599117Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.3600127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.3601121Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.3602057Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.3602929Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.3603817Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.3702588Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.3745918Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.3746670Z creating: build/lib/ 2024-08-20T22:20:27.3835092Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.4291983Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.4301928Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.4310136Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.4318236Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.4319048Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.4337695Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.4340422Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.4405156Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.4480338Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.4544199Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.4551957Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.4579704Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.4707501Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.4727532Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.4822506Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.4824529Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.4825492Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.4845659Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.5349164Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.5368788Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.5369524Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.5370074Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.5759850Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.5942687Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.6516261Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.6573326Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.6573957Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.7834131Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.8091688Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.8597460Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.8782265Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.8823925Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:28.9517719Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.4006196Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.4010927Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.4015209Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.4702184Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.4703240Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.4705989Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.2910395Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.4903888Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.4978412Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.4999133Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.5025107Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.5051319Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.5085657Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.5086146Z creating: build/bin/ 2024-08-20T22:20:34.5137280Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.5188578Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.5240181Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.5299226Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.5353358Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.5402137Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.5454569Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.5508818Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.5564802Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.5621339Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.5692785Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.5746107Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.5795269Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.5845489Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.5896376Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.5954250Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.6009660Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.6061630Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.6117697Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.6169341Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.6220720Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.6270532Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.6321912Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.6377650Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.6428349Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.6485798Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.6541979Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.6594768Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.6645867Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.6696517Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.6861502Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.6912396Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.6966472Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.7025235Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.7088968Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.7164963Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.7316366Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.7370370Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.7422042Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.7474550Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.7524971Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.7584230Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.7633484Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.7689527Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.8136704Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.8584642Z inflating: build/bin/protoc 2024-08-20T22:20:34.8638212Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.8691381Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.8744627Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.8796852Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.8850085Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.8902947Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.8952644Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.9005418Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:34.9338389Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:34.9683543Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:35.0043648Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:35.0096246Z inflating: build/bin/BackoffTest 2024-08-20T22:20:35.0150055Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:35.0205741Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:35.0260122Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:35.0273496Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:35.0327454Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:35.0332174Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:35.0335091Z inflating: build/bin/example_allreduce 2024-08-20T22:20:35.0390545Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:35.0459479Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:35.0462169Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:35.0529573Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:35.0539236Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:35.0885651Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.2070312Z inflating: build/bin/test_api 2024-08-20T22:20:35.2144138Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.2196679Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.2261754Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.2319445Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.2379569Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.2438470Z inflating: build/bin/atest 2024-08-20T22:20:35.2502038Z inflating: build/bin/basic 2024-08-20T22:20:35.2557497Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.2608725Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.2667444Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.2721030Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.2814583Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.2865550Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.2917264Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.2975856Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.3031742Z inflating: build/bin/half_test 2024-08-20T22:20:35.3127600Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.3178617Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.3233671Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.3287832Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.3341526Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.3394776Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.3451261Z inflating: build/bin/native_test 2024-08-20T22:20:35.3502131Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.3554679Z inflating: build/bin/operators_test 2024-08-20T22:20:35.3606211Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.3673809Z inflating: build/bin/pow_test 2024-08-20T22:20:35.3731818Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.3782026Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.3833273Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.3890068Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.3950432Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.4002528Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.4055482Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.4135880Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.4189989Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.4193070Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.4250133Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.4309976Z inflating: build/bin/type_test 2024-08-20T22:20:35.4362791Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.4364360Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.4435749Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.4487699Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.4539602Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.4591607Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.4652267Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.4758987Z inflating: build/bin/List_test 2024-08-20T22:20:35.4825717Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.4948461Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.5045705Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.5174753Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.5278821Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.5340440Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.5438021Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.5489586Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.5545852Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.5594999Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.5898573Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.5962809Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.6016562Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.6068442Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.6127786Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.6183000Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.6254053Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.6312703Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.6362636Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.6421094Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.6471706Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.6536851Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.6594644Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.6643909Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.6696731Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.6746396Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.6798457Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.6851553Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.6902365Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.6962833Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.7013009Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.7065684Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.7079632Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.7145851Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.7204576Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.7268675Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.7331941Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.8162336Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.8744759Z inflating: build/bin/test_jit 2024-08-20T22:20:35.8745227Z creating: .additional_ci_files/ 2024-08-20T22:20:35.8807638Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.9048141Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.9085378Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.9085784Z rm artifacts.zip 2024-08-20T22:20:35.9094735Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9095245Z env: 2024-08-20T22:20:35.9095530Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9095995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9096484Z ##[endgroup] 2024-08-20T22:20:36.0249719Z ##[group]Run df -H 2024-08-20T22:20:36.0250056Z df -H 2024-08-20T22:20:36.0259050Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.0259562Z env: 2024-08-20T22:20:36.0259845Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.0260511Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.0261011Z ##[endgroup] 2024-08-20T22:20:36.0310355Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:36.0310969Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:36.0311454Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:36.0311927Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:36.0312375Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:36.0313040Z tmpfs 34G 17k 34G 1% /tmp 2024-08-20T22:20:36.0313545Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:36.0314847Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:36.0354729Z Prepare all required actions 2024-08-20T22:20:36.0355177Z Getting action download info 2024-08-20T22:20:36.1527830Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.1528294Z with: 2024-08-20T22:20:36.1528553Z env: 2024-08-20T22:20:36.1529237Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.1529724Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.1530227Z ##[endgroup] 2024-08-20T22:20:36.1563592Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.1564037Z with: 2024-08-20T22:20:36.1564305Z name: td_results 2024-08-20T22:20:36.1564625Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.1564977Z region: us-east-1 2024-08-20T22:20:36.1565278Z env: 2024-08-20T22:20:36.1565547Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.1566005Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.1566496Z ##[endgroup] 2024-08-20T22:20:36.6348973Z (node:183561) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.6349719Z 2024-08-20T22:20:36.6349984Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.6350730Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.6351643Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.7015616Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.7022260Z Artifact download has finished successfully 2024-08-20T22:20:36.7351576Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.7352075Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.7352659Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.7362428Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.7362939Z env: 2024-08-20T22:20:36.7363224Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7363701Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7364187Z ##[endgroup] 2024-08-20T22:20:36.7421794Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.7436403Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.7470855Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.7471357Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.7479544Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.7479899Z env: 2024-08-20T22:20:36.7480190Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7480654Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7481146Z ##[endgroup] 2024-08-20T22:20:36.7768502Z Prepare all required actions 2024-08-20T22:20:36.7808615Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.7809080Z with: 2024-08-20T22:20:36.7809650Z github-token: *** 2024-08-20T22:20:36.7809972Z env: 2024-08-20T22:20:36.7810254Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7810718Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7811223Z ##[endgroup] 2024-08-20T22:20:36.7829084Z ##[group]Run set -eux 2024-08-20T22:20:36.7829439Z set -eux 2024-08-20T22:20:36.7830033Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.7839171Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.7839672Z env: 2024-08-20T22:20:36.7839951Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.7840410Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.7841064Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.7841414Z ##[endgroup] 2024-08-20T22:20:36.7869482Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-07cd86c6b6ea02560 2024-08-20T22:20:38.2732683Z setting job-id=29026528971 2024-08-20T22:20:38.2733609Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.2955417Z Prepare all required actions 2024-08-20T22:20:38.2955873Z Getting action download info 2024-08-20T22:20:38.3993346Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.3993808Z with: 2024-08-20T22:20:38.3994272Z github-token: *** 2024-08-20T22:20:38.4003424Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.4013290Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.4014072Z env: 2024-08-20T22:20:38.4014373Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.4014851Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.4015348Z ##[endgroup] 2024-08-20T22:20:38.4058424Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.4058983Z with: 2024-08-20T22:20:38.4059262Z shell: bash 2024-08-20T22:20:38.4059577Z timeout_minutes: 10 2024-08-20T22:20:38.4059911Z max_attempts: 5 2024-08-20T22:20:38.4060239Z retry_wait_seconds: 30 2024-08-20T22:20:38.4061380Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.4062739Z polling_interval_seconds: 1 2024-08-20T22:20:38.4063126Z warning_on_retry: true 2024-08-20T22:20:38.4063484Z continue_on_error: false 2024-08-20T22:20:38.4063830Z env: 2024-08-20T22:20:38.4064114Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.4064569Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.4065228Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.4065551Z ##[endgroup] 2024-08-20T22:20:38.4832350Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.7202677Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:38.7383406Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T22:20:38.7386173Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T22:20:38.7500240Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:20:38.7509262Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T22:20:38.7517619Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:38.7522977Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.4609490Z Command completed after 1 attempt(s). 2024-08-20T22:20:39.4669386Z ##[group]Run set -x 2024-08-20T22:20:39.4669727Z set -x 2024-08-20T22:20:39.4670042Z  2024-08-20T22:20:39.4670607Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:39.4671295Z # in runner workspace 2024-08-20T22:20:39.4671835Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:39.4680803Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:39.4681307Z env: 2024-08-20T22:20:39.4681596Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:39.4682071Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:39.4682574Z ##[endgroup] 2024-08-20T22:20:39.4713030Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:39.4949620Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:39.4950182Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:39.4950656Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:39.4951068Z  2024-08-20T22:20:39.4951630Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:39.4952318Z # in runner workspace 2024-08-20T22:20:39.4952908Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:39.4953567Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:39.4954029Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:39.4963483Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:39.4973248Z  --selected-test-configs "" \ 2024-08-20T22:20:39.4973720Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:39.4974154Z  --tag "${TAG}" \ 2024-08-20T22:20:39.4974560Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:39.4975010Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:39.4975457Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:39.4985792Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:39.4986302Z env: 2024-08-20T22:20:39.4986682Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:39.4987151Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:39.4987852Z GITHUB_TOKEN: *** 2024-08-20T22:20:39.4988575Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:39.4989355Z PR_NUMBER: 2024-08-20T22:20:39.4989676Z TAG: ciflow/inductor/133712 2024-08-20T22:20:39.4990059Z EVENT_NAME: push 2024-08-20T22:20:39.4990379Z SCHEDULE: 2024-08-20T22:20:39.4990672Z HEAD_BRANCH: 2024-08-20T22:20:39.4990981Z ##[endgroup] 2024-08-20T22:20:39.5019345Z Workflow: inductor 2024-08-20T22:20:39.5020356Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:39.7200732Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:39.8675662Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:39.8676128Z echo "Filtered matrix:" 2024-08-20T22:20:39.8685461Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:39.8695042Z  2024-08-20T22:20:39.8695323Z echo 2024-08-20T22:20:39.8695715Z echo "Is the current job unstable? False" 2024-08-20T22:20:39.8696176Z  2024-08-20T22:20:39.8696460Z echo 2024-08-20T22:20:39.8696831Z echo "Is keep-going label set? False" 2024-08-20T22:20:39.8697282Z  2024-08-20T22:20:39.8697557Z echo 2024-08-20T22:20:39.8697884Z echo "Renabled issues? " 2024-08-20T22:20:39.8706938Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:39.8707445Z env: 2024-08-20T22:20:39.8707749Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:39.8708220Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:39.8708717Z ##[endgroup] 2024-08-20T22:20:39.8741626Z Filtered matrix: 2024-08-20T22:20:39.8753101Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:39.8762351Z 2024-08-20T22:20:39.8762506Z Is the current job unstable? False 2024-08-20T22:20:39.8762798Z 2024-08-20T22:20:39.8763162Z Is keep-going label set? False 2024-08-20T22:20:39.8763425Z 2024-08-20T22:20:39.8764214Z Renabled issues? 2024-08-20T22:20:39.8816299Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:39.8817021Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:39.8825274Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:39.8825774Z env: 2024-08-20T22:20:39.8826062Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:39.8826658Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:39.8827168Z JOB_TIMEOUT: 240 2024-08-20T22:20:39.8827481Z ##[endgroup] 2024-08-20T22:20:39.8910043Z ##[group]Run set -x 2024-08-20T22:20:39.8910442Z set -x 2024-08-20T22:20:39.8910745Z  2024-08-20T22:20:39.8911105Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:39.8911657Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:39.8912229Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:39.8912750Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:39.8913184Z else 2024-08-20T22:20:39.8913529Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:39.8913955Z fi 2024-08-20T22:20:39.8914235Z  2024-08-20T22:20:39.8914715Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:39.8915481Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:39.8916175Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:39.8916775Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:39.8917247Z container_name=$(docker run \ 2024-08-20T22:20:39.8917675Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:39.8918053Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:39.8918460Z  -e PR_NUMBER \ 2024-08-20T22:20:39.8918826Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:39.8919226Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:39.8919638Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:39.8920039Z  -e GITHUB_JOB \ 2024-08-20T22:20:39.8920399Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:39.8920790Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:39.8921203Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:39.8921611Z  -e JOB_ID \ 2024-08-20T22:20:39.8921953Z  -e JOB_NAME \ 2024-08-20T22:20:39.8922332Z  -e BASE_SHA \ 2024-08-20T22:20:39.8922704Z  -e BRANCH \ 2024-08-20T22:20:39.8923039Z  -e SHA1 \ 2024-08-20T22:20:39.8923391Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:39.8923806Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:39.8924193Z  -e SHARD_NUMBER \ 2024-08-20T22:20:39.8924566Z  -e TEST_CONFIG \ 2024-08-20T22:20:39.8924943Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:39.8925345Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:39.8925768Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:39.8926199Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:39.8926613Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:39.8927014Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:39.8927396Z  -e NO_TD \ 2024-08-20T22:20:39.8927739Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:39.8928126Z  -e PR_LABELS \ 2024-08-20T22:20:39.8928538Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:39.8929377Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:39.8929790Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:39.8930204Z  -e XLA_CUDA \ 2024-08-20T22:20:39.8930617Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:39.8931130Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:39.8931662Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:39.8932188Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:39.8932665Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:39.8933121Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:39.8933738Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:39.8934220Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:39.8934787Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:39.8935256Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:39.8935653Z  --ipc=host \ 2024-08-20T22:20:39.8936019Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:39.8936420Z  --tty \ 2024-08-20T22:20:39.8936733Z  --detach \ 2024-08-20T22:20:39.8937241Z  --name="${container_name}" \ 2024-08-20T22:20:39.8937672Z  --user jenkins \ 2024-08-20T22:20:39.8938170Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:39.8938735Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:39.8939180Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:39.8939535Z ) 2024-08-20T22:20:39.8939948Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:39.8940879Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:39.8941868Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:39.8942794Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:39.8951391Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:39.8951740Z env: 2024-08-20T22:20:39.8952022Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:39.8952488Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:39.8953103Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:39.8953596Z PR_NUMBER: 2024-08-20T22:20:39.8953922Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:39.8954342Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:39.8954703Z GITHUB_JOB: test 2024-08-20T22:20:39.8955013Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:39.8955383Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:39.8955741Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:39.8956074Z JOB_ID: 29026528971 2024-08-20T22:20:39.8956769Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:39.8957526Z BRANCH: 2024-08-20T22:20:39.8957864Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:39.8958388Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:39.8958876Z TEST_CONFIG: dynamic_inductor_timm 2024-08-20T22:20:39.8959288Z SHARD_NUMBER: 1 2024-08-20T22:20:39.8959605Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:39.8959937Z REENABLED_ISSUES: 2024-08-20T22:20:39.8960283Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:39.8960682Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:39.8961049Z TEST_SHOWLOCALS: False 2024-08-20T22:20:39.8961400Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:39.8961745Z NO_TD: False 2024-08-20T22:20:39.8962062Z TD_DISTRIBUTED: False 2024-08-20T22:20:39.8962497Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:39.8962992Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:39.8963383Z SHM_SIZE: 2g 2024-08-20T22:20:39.8964496Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:39.8965691Z XLA_CUDA: 2024-08-20T22:20:39.8966183Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:39.8966819Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:39.8967271Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:39.8967694Z DASHBOARD_TAG: 2024-08-20T22:20:39.8968212Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:39.8968604Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:39.8968987Z ##[endgroup] 2024-08-20T22:20:39.8996704Z + [[ dynamic_inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:39.8997501Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:39.8998155Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:39.9005528Z +++ nproc --ignore=2 2024-08-20T22:20:39.9024223Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:51.2434131Z + container_name=289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T22:20:51.2436836Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:51.2438244Z + docker exec -i 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:51.3997683Z + echo DOCKER_CONTAINER_ID=289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T22:20:51.4002475Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:51.4005434Z + docker exec -t 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:51.8020590Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:52.1295456Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:52.1298016Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:52.1300542Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:52.1304202Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:52.1306968Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:52.1312829Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:52.1329511Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:52.1343431Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:52.1361465Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:52.1754487Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:52.7665761Z Installing collected packages: torch 2024-08-20T22:21:02.6364844Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:02.6366288Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:02.6368337Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:02.7222676Z + export TERM=vt100 2024-08-20T22:21:02.7223070Z + TERM=vt100 2024-08-20T22:21:02.7226140Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:02.7237490Z + source .ci/pytorch/common.sh 2024-08-20T22:21:02.7241499Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:02.7250803Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:02.7252538Z +++ declare -f -t trap_add 2024-08-20T22:21:02.7258229Z ++ set -ex 2024-08-20T22:21:02.7258805Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:02.7259318Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:02.7260205Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:02.7262984Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:02.7283062Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:02.7283630Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:02.7284047Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:02.7284437Z + shift 2024-08-20T22:21:02.7284720Z + for trap_add_name in "$@" 2024-08-20T22:21:02.7290743Z +++ trap -p EXIT 2024-08-20T22:21:02.7294274Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:02.7294661Z +++ extract_trap_cmd 2024-08-20T22:21:02.7295032Z +++ printf '%s\n' '' 2024-08-20T22:21:02.7295443Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:02.7297796Z + trap -- ' 2024-08-20T22:21:02.7298230Z cleanup_workspace' EXIT 2024-08-20T22:21:02.7298747Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:03.4337244Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:03.4359542Z + echo 'Environment variables:' 2024-08-20T22:21:03.4360072Z Environment variables: 2024-08-20T22:21:03.4360408Z + env 2024-08-20T22:21:03.4369932Z INSTALLED_DB=yes 2024-08-20T22:21:03.4370479Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:03.4370937Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:03.4371355Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:03.4372010Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:03.4372734Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:03.4375159Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:03.4375904Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:03.4376639Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:03.4377168Z HOSTNAME=289234f66ec0 2024-08-20T22:21:03.4378158Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4379099Z GITHUB_ACTION=__self 2024-08-20T22:21:03.4379567Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:03.4384282Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:03.4388954Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:03.4389485Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:03.4389844Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:03.4390254Z TEST_CONFIG=dynamic_inductor_timm 2024-08-20T22:21:03.4390679Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:03.4391179Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:03.4391669Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:03.4392117Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:03.4392538Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:03.4393124Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:03.4393595Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:03.4394110Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:03.4394587Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:03.4394921Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:03.4395304Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:03.4395709Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4396166Z XLA_CUDA= 2024-08-20T22:21:03.4396678Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:03.4398788Z *** 2024-08-20T22:21:03.4399092Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:03.4399507Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:03.4399884Z GITHUB_ACTIONS=true 2024-08-20T22:21:03.4400223Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:03.4400765Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:03.4401323Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:03.4401817Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4402334Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:03.4402825Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4403639Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:03.4404369Z UCC_HOME=/usr 2024-08-20T22:21:03.4404711Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:03.4405118Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:03.4405473Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:03.4405949Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:03.4406459Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:03.4406940Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:03.4407325Z SHARD_NUMBER=1 2024-08-20T22:21:03.4407639Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:03.4408000Z HOME=/var/lib/jenkins 2024-08-20T22:21:03.4408378Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:03.4408831Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:03.4409324Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:03.4409821Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:03.4410201Z CUDA_VERSION=12.1.1 2024-08-20T22:21:03.4410644Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:03.4411090Z NUM_TEST_SHARDS=2 2024-08-20T22:21:03.4411399Z UCX_HOME=/usr 2024-08-20T22:21:03.4411937Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:03.4413060Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4414380Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:03.4415676Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4416828Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:03.4417510Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:03.4417845Z DASHBOARD_TAG= 2024-08-20T22:21:03.4418160Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:03.4418639Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:03.4419196Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:03.4420236Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4421378Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:03.4421811Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:03.4422203Z PR_NUMBER= 2024-08-20T22:21:03.4422498Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:03.4422847Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:03.4423303Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:03.4423771Z TERM=vt100 2024-08-20T22:21:03.4424124Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:03.4424554Z INSTALLED_VISION=yes 2024-08-20T22:21:03.4424876Z BRANCH= 2024-08-20T22:21:03.4425265Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:03.4425694Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:03.4426129Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:03.4427070Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:03.4427843Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:03.4428341Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:03.4431997Z REENABLED_ISSUES= 2024-08-20T22:21:03.4432311Z SHLVL=1 2024-08-20T22:21:03.4432586Z MAX_JOBS=14 2024-08-20T22:21:03.4432932Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:03.4433295Z NVARCH=x86_64 2024-08-20T22:21:03.4433594Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:03.4434066Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4434619Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:03.4435126Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:03.4435815Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:03.4436416Z GITHUB_JOB=test 2024-08-20T22:21:03.4436833Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:03.4437399Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:03.4437926Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:03.4438266Z TD_DISTRIBUTED=False 2024-08-20T22:21:03.4438664Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:03.4439110Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:03.4439560Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:03.4439934Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:03.4440293Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:03.4440647Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:03.4441727Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:03.4442820Z GITHUB_BASE_REF= 2024-08-20T22:21:03.4443155Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:03.4443526Z CI=true 2024-08-20T22:21:03.4443857Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:03.4444289Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:03.4444669Z JOB_ID=29026528971 2024-08-20T22:21:03.4444990Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:03.4445330Z GITHUB_HEAD_REF= 2024-08-20T22:21:03.4445642Z GITHUB_ACTION_REF= 2024-08-20T22:21:03.4446091Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:03.4446570Z TEST_SHOWLOCALS=False 2024-08-20T22:21:03.4446924Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:03.4447288Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:03.4448234Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4449065Z NO_TD=False 2024-08-20T22:21:03.4449380Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:03.4449747Z _=/usr/bin/env 2024-08-20T22:21:03.4450252Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:03.4603180Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:03.4604267Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:03.4605268Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:03.4606468Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:03.4607099Z + BUILD_DIR=build 2024-08-20T22:21:03.4607432Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:03.4608080Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:03.4608437Z + SHARD_NUMBER=1 2024-08-20T22:21:03.4608748Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:03.4609068Z + export VALGRIND=ON 2024-08-20T22:21:03.4609390Z + VALGRIND=ON 2024-08-20T22:21:03.4609963Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:03.4610462Z + [[ 0 == \1 ]] 2024-08-20T22:21:03.4610826Z + [[ False == \1 ]] 2024-08-20T22:21:03.4611320Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:03.4611856Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:03.4623345Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:03.4624104Z + [[ -n '' ]] 2024-08-20T22:21:03.4624467Z + echo 'Environment variables' 2024-08-20T22:21:03.4624852Z Environment variables 2024-08-20T22:21:03.4625177Z + env 2024-08-20T22:21:03.4632152Z INSTALLED_DB=yes 2024-08-20T22:21:03.4632670Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:03.4633192Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:03.4633748Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:03.4634562Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:03.4635348Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:03.4636037Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:03.4636752Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:03.4637549Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:03.4638243Z HOSTNAME=289234f66ec0 2024-08-20T22:21:03.4639335Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4640263Z GITHUB_ACTION=__self 2024-08-20T22:21:03.4640629Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:03.4645045Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:03.4649517Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:03.4650046Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:03.4650409Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:03.4650765Z TEST_CONFIG=dynamic_inductor_timm 2024-08-20T22:21:03.4651180Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:03.4651669Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:03.4652136Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:03.4652595Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:03.4653011Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:03.4653424Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:03.4653893Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:03.4654406Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:03.4654887Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:03.4655235Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:03.4655622Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:03.4656026Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4656484Z XLA_CUDA= 2024-08-20T22:21:03.4656951Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:03.4657348Z *** 2024-08-20T22:21:03.4657647Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:03.4658071Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:03.4658490Z GITHUB_ACTIONS=true 2024-08-20T22:21:03.4658831Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:03.4659357Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:03.4660118Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:03.4660597Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4661135Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:03.4661635Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4662454Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:03.4663190Z UCC_HOME=/usr 2024-08-20T22:21:03.4663553Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:03.4663969Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:03.4664469Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:03.4664954Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:03.4665471Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:03.4665959Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:03.4666494Z SHARD_NUMBER=1 2024-08-20T22:21:03.4666812Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:03.4667182Z HOME=/var/lib/jenkins 2024-08-20T22:21:03.4667561Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:03.4668022Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:03.4668518Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:03.4669018Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:03.4669399Z CUDA_VERSION=12.1.1 2024-08-20T22:21:03.4669839Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:03.4670297Z NUM_TEST_SHARDS=2 2024-08-20T22:21:03.4670610Z UCX_HOME=/usr 2024-08-20T22:21:03.4671146Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:03.4672261Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4673566Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:03.4674853Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4676008Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:03.4676688Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:03.4677035Z DASHBOARD_TAG= 2024-08-20T22:21:03.4677359Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:03.4677847Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:03.4678408Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:03.4679439Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4680385Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:03.4680817Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:03.4681206Z PR_NUMBER= 2024-08-20T22:21:03.4681508Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:03.4681844Z VALGRIND=ON 2024-08-20T22:21:03.4682152Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:03.4682617Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:03.4683095Z TERM=vt100 2024-08-20T22:21:03.4683453Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:03.4683887Z INSTALLED_VISION=yes 2024-08-20T22:21:03.4684216Z BRANCH= 2024-08-20T22:21:03.4684514Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:03.4684934Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:03.4685375Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:03.4686160Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:03.4686940Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:03.4687445Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:03.4687932Z REENABLED_ISSUES= 2024-08-20T22:21:03.4688244Z SHLVL=1 2024-08-20T22:21:03.4688523Z MAX_JOBS=14 2024-08-20T22:21:03.4688848Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:03.4689219Z NVARCH=x86_64 2024-08-20T22:21:03.4689532Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:03.4690005Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:03.4690555Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:03.4691169Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:03.4691863Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:03.4692461Z GITHUB_JOB=test 2024-08-20T22:21:03.4692880Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:03.4693459Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:03.4693989Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:03.4694329Z TD_DISTRIBUTED=False 2024-08-20T22:21:03.4694730Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:03.4695275Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:03.4695720Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:03.4696095Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:03.4696459Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:03.4696829Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:03.4697902Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:03.4699043Z GITHUB_BASE_REF= 2024-08-20T22:21:03.4699385Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:03.4699769Z CI=true 2024-08-20T22:21:03.4700105Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:03.4700530Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:03.4700918Z JOB_ID=29026528971 2024-08-20T22:21:03.4701247Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:03.4701588Z GITHUB_HEAD_REF= 2024-08-20T22:21:03.4701908Z GITHUB_ACTION_REF= 2024-08-20T22:21:03.4702367Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:03.4702860Z TEST_SHOWLOCALS=False 2024-08-20T22:21:03.4703206Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:03.4703583Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:03.4704515Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_83135e73-8a33-49b1-89ec-23ce430c9623 2024-08-20T22:21:03.4705353Z NO_TD=False 2024-08-20T22:21:03.4705666Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:03.4706048Z _=/usr/bin/env 2024-08-20T22:21:03.4706565Z + echo 'Testing pytorch' 2024-08-20T22:21:03.4706916Z Testing pytorch 2024-08-20T22:21:03.4707260Z + export LANG=C.UTF-8 2024-08-20T22:21:03.4707618Z + LANG=C.UTF-8 2024-08-20T22:21:03.4707924Z + PR_NUMBER= 2024-08-20T22:21:03.4708287Z + [[ dynamic_inductor_timm == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:03.4708827Z + [[ dynamic_inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:03.4709349Z + [[ dynamic_inductor_timm == \s\l\o\w ]] 2024-08-20T22:21:03.4710010Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:03.4710729Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:03.4711296Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:03.4711787Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:03.4712256Z + [[ dynamic_inductor_timm == *crossref* ]] 2024-08-20T22:21:03.4712859Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:03.4713528Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:03.4714208Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:03.4714791Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:03.4715352Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:04.6567833Z Collecting ninja==1.10.2 2024-08-20T22:21:04.6751253Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:04.7299173Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:05.3478642Z Installing collected packages: ninja 2024-08-20T22:21:05.3559089Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:05.3560397Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:05.4029416Z Successfully installed ninja-1.10.2 2024-08-20T22:21:05.4866094Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:05.4868813Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:05.4870446Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:05.4870974Z + install_tlparse 2024-08-20T22:21:05.4871382Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:05.4872125Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:05.9195959Z Collecting tlparse==0.3.25 2024-08-20T22:21:05.9344341Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:05.9732040Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:06.6176591Z Installing collected packages: tlparse 2024-08-20T22:21:06.6956965Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:06.7739549Z ++ python -m site --user-base 2024-08-20T22:21:06.7986386Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.7988266Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:06.7988949Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:06.7989705Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:06.7990679Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:06.7992050Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:06.7992763Z + cd test 2024-08-20T22:21:06.7993301Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:08.4433134Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:08.4433778Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:08.4437336Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:08.4437933Z + [[ dynamic_inductor_timm == *pr_time_benchmarks* ]] 2024-08-20T22:21:08.4438580Z + [[ dynamic_inductor_timm == *dynamo_eager* ]] 2024-08-20T22:21:08.4439087Z + [[ dynamic_inductor_timm == *aot_eager* ]] 2024-08-20T22:21:08.4439578Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:21:08.4440081Z + [[ dynamic_inductor_timm == *inductor* ]] 2024-08-20T22:21:08.4440558Z + [[ dynamic_inductor_timm != *perf* ]] 2024-08-20T22:21:08.4441243Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:08.4441698Z + [[ dynamic_inductor_timm == *dynamic* ]] 2024-08-20T22:21:08.4442340Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:21:08.4442916Z + [[ dynamic_inductor_timm == *cpu* ]] 2024-08-20T22:21:08.4443426Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:08.4471441Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:08.4472357Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:08.4475148Z + cd test 2024-08-20T22:21:08.4475880Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:09.9166063Z PyTorch built with: 2024-08-20T22:21:09.9166781Z - GCC 9.4 2024-08-20T22:21:09.9167261Z - C++ Version: 201703 2024-08-20T22:21:09.9168419Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:09.9169559Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:09.9170296Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:09.9170823Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:09.9171324Z - NNPACK is enabled 2024-08-20T22:21:09.9171719Z - CPU capability usage: AVX2 2024-08-20T22:21:09.9172520Z - CUDA Runtime 12.1 2024-08-20T22:21:09.9173050Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:09.9173675Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:09.9174130Z - Magma 2.6.1 2024-08-20T22:21:09.9181278Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:09.9187981Z 2024-08-20T22:21:10.2422279Z + cd test 2024-08-20T22:21:10.2423126Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:11.5900341Z ATen/Parallel: 2024-08-20T22:21:11.5900911Z at::get_num_threads() : 8 2024-08-20T22:21:11.5901436Z at::get_num_interop_threads() : 16 2024-08-20T22:21:11.5901990Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:11.5902488Z omp_get_max_threads() : 8 2024-08-20T22:21:11.5903628Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:11.5904492Z mkl_get_max_threads() : 8 2024-08-20T22:21:11.5905141Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:11.5905780Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:11.5906364Z Environment variables: 2024-08-20T22:21:11.5906723Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:11.5907104Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:11.5907485Z ATen parallel backend: OpenMP 2024-08-20T22:21:11.5907751Z 2024-08-20T22:21:11.8702485Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:11.8703276Z + [[ dynamic_inductor_timm == *backward* ]] 2024-08-20T22:21:11.8703820Z + [[ dynamic_inductor_timm == *xla* ]] 2024-08-20T22:21:11.8704291Z + [[ dynamic_inductor_timm == *executorch* ]] 2024-08-20T22:21:11.8704825Z + [[ dynamic_inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:11.8705511Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:11.8706099Z + [[ dynamic_inductor_timm == distributed ]] 2024-08-20T22:21:11.8706771Z + [[ dynamic_inductor_timm == *inductor_distributed* ]] 2024-08-20T22:21:11.8707395Z + [[ dynamic_inductor_timm == *inductor-halide* ]] 2024-08-20T22:21:11.8708047Z + [[ dynamic_inductor_timm == *inductor-micro-benchmark* ]] 2024-08-20T22:21:11.8708619Z + [[ dynamic_inductor_timm == *huggingface* ]] 2024-08-20T22:21:11.8709097Z + [[ dynamic_inductor_timm == *timm* ]] 2024-08-20T22:21:11.8709528Z + install_torchvision 2024-08-20T22:21:11.8709881Z + local orig_preload 2024-08-20T22:21:11.8710204Z + local commit 2024-08-20T22:21:11.8710536Z ++ get_pinned_commit vision 2024-08-20T22:21:11.8710942Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:11.8725279Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:11.8725733Z + orig_preload= 2024-08-20T22:21:11.8726086Z + '[' -n '' ']' 2024-08-20T22:21:11.8726938Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:11.8729162Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.2118466Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:12.2122649Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-bkrzwxeb 2024-08-20T22:21:12.2153784Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-bkrzwxeb 2024-08-20T22:21:13.8317531Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:13.8347669Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:15.1741211Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:15.4767176Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:17.9656967Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:17.9689744Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:17.9693471Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:17.9697697Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:17.9768128Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:17.9772351Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:17.9775302Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:17.9777676Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:17.9780989Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:17.9785673Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:17.9799790Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:18.0288928Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:18.0352814Z Building wheels for collected packages: torchvision 2024-08-20T22:22:36.9184618Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:22:36.9217768Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116018 sha256=df26c20b643f2b876413f41563f2cc1202319a08bc3ed06c61439ee21e8a742b 2024-08-20T22:22:36.9220548Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:36.9256547Z Successfully built torchvision 2024-08-20T22:22:37.4197805Z Installing collected packages: torchvision 2024-08-20T22:22:37.8375931Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:37.9785147Z + '[' -n '' ']' 2024-08-20T22:22:37.9787639Z + id=0 2024-08-20T22:22:37.9788365Z + test_dynamo_benchmark timm_models 0 2024-08-20T22:22:37.9789797Z ++ pwd 2024-08-20T22:22:37.9794843Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:37.9795462Z + local suite=timm_models 2024-08-20T22:22:37.9798094Z + shift 2024-08-20T22:22:37.9798780Z + local shard_id=0 2024-08-20T22:22:37.9799232Z + shift 2024-08-20T22:22:37.9800065Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-08-20T22:22:37.9800723Z + [[ dynamic_inductor_timm == *perf* ]] 2024-08-20T22:22:37.9802158Z + [[ dynamic_inductor_timm == *cpu* ]] 2024-08-20T22:22:37.9802664Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:37.9803453Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-08-20T22:22:37.9804092Z ++ pwd 2024-08-20T22:22:37.9804571Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:37.9805273Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:37.9830193Z + local name=inference 2024-08-20T22:22:37.9830541Z + shift 2024-08-20T22:22:37.9830889Z + local suite=timm_models 2024-08-20T22:22:37.9831314Z + shift 2024-08-20T22:22:37.9831584Z + local shard_id=0 2024-08-20T22:22:37.9831886Z + shift 2024-08-20T22:22:37.9832182Z + partition_flags=() 2024-08-20T22:22:37.9832526Z + local partition_flags 2024-08-20T22:22:37.9832919Z + [[ -n 2 ]] 2024-08-20T22:22:37.9833241Z + [[ -n 0 ]] 2024-08-20T22:22:37.9833860Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:37.9834618Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-08-20T22:22:37.9835123Z + [[ dynamic_inductor_timm == *perf* ]] 2024-08-20T22:22:37.9835627Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:37.9836138Z + [[ dynamic_inductor_timm == *_avx2* ]] 2024-08-20T22:22:37.9836616Z + [[ dynamic_inductor_timm == *_avx512* ]] 2024-08-20T22:22:37.9838506Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-08-20T22:22:44.5013021Z 2024-08-20T22:22:44.8309274Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:22:44.8309743Z 2024-08-20T22:22:44.9399737Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-08-20T22:46:12.4617927Z pass 2024-08-20T22:46:12.5119395Z TIMING: entire_frame_compile:98.3012 code_gen:29.6989 inductor_compile:52.63345 backend_compile:80.90318 total_wall_time:113.68119 2024-08-20T22:46:12.5123510Z STATS: call_* op count: 989 | FakeTensorMode.__torch_dispatch__:63983 | attempt fast:2461 | fast is_contiguous:2461 | FakeTensor.__torch_dispatch__:15635 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-08-20T22:46:12.5124895Z Dynamo produced 3 graphs covering 989 ops with 7 graph breaks (5 unique) 2024-08-20T22:46:21.0994524Z 2024-08-20T22:46:22.0631790Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:22.0632413Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:22.0632962Z cuda train botnet26t_256 2024-08-20T22:47:38.8749744Z pass 2024-08-20T22:47:38.8856847Z TIMING: entire_frame_compile:70.22111 code_gen:23.00571 inductor_compile:39.85797 backend_compile:60.19182 total_wall_time:90.16465 2024-08-20T22:47:38.8861103Z STATS: call_* op count: 265 | FakeTensorMode.__torch_dispatch__:32683 | attempt fast:1260 | fast is_contiguous:1234 | FakeTensor.__torch_dispatch__:5563 | slow no contiguity match:26 | ProxyTorchDispatchMode.__torch_dispatch__:9029 2024-08-20T22:47:38.8862638Z Dynamo produced 2 graphs covering 265 ops with 6 graph breaks (5 unique) 2024-08-20T22:47:47.8968917Z 2024-08-20T22:47:51.8783772Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:51.8784259Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:47:51.8784693Z cuda train cait_m36_384 2024-08-20T22:47:51.8789160Z Traceback (most recent call last): 2024-08-20T22:47:51.8789946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:47:51.8790844Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:47:51.8791801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in forward_and_backward_pass 2024-08-20T22:47:51.8793202Z pred = mod(*cloned_inputs) 2024-08-20T22:47:51.8794547Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:47:51.8795482Z return self._call_impl(*args, **kwargs) 2024-08-20T22:47:51.8796699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:47:51.8797763Z return forward_call(*args, **kwargs) 2024-08-20T22:47:51.8798934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 438, in forward 2024-08-20T22:47:51.8799792Z x = self.forward_features(x) 2024-08-20T22:47:51.8800924Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 423, in forward_features 2024-08-20T22:47:51.8801960Z x = self.blocks(x) 2024-08-20T22:47:51.8803468Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:47:51.8804579Z return self._call_impl(*args, **kwargs) 2024-08-20T22:47:51.8805714Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:47:51.8806571Z return forward_call(*args, **kwargs) 2024-08-20T22:47:51.8809255Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward 2024-08-20T22:47:51.8810106Z input = module(input) 2024-08-20T22:47:51.8811049Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:47:51.8812059Z return self._call_impl(*args, **kwargs) 2024-08-20T22:47:51.8813157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:47:51.8814144Z return forward_call(*args, **kwargs) 2024-08-20T22:47:51.8815116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 199, in forward 2024-08-20T22:47:51.8816073Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-08-20T22:47:51.8817299Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:47:51.8818295Z return self._call_impl(*args, **kwargs) 2024-08-20T22:47:51.8819348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:47:51.8820256Z return forward_call(*args, **kwargs) 2024-08-20T22:47:51.8821206Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 153, in forward 2024-08-20T22:47:51.8822088Z x = (attn @ v).transpose(1, 2).reshape(B, N, C) 2024-08-20T22:47:51.8825684Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 198051 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:47:51.8828949Z 2024-08-20T22:47:51.8829273Z The above exception was the direct cause of the following exception: 2024-08-20T22:47:51.8829843Z 2024-08-20T22:47:51.8830011Z Traceback (most recent call last): 2024-08-20T22:47:51.8830846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:47:51.8831520Z ) = runner.load_model( 2024-08-20T22:47:51.8832191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 328, in load_model 2024-08-20T22:47:51.8832991Z self.validate_model(model, example_inputs) 2024-08-20T22:47:51.8833849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:47:51.8834653Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:47:51.8835142Z RuntimeError: Eager run failed 2024-08-20T22:47:51.8835416Z 2024-08-20T22:47:51.8835542Z eager_fail_to_run 2024-08-20T22:47:55.8258192Z 2024-08-20T22:47:56.7153865Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:56.7154420Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:47:56.7154897Z cuda train coat_lite_mini 2024-08-20T22:49:58.6717098Z pass 2024-08-20T22:49:58.6834348Z TIMING: entire_frame_compile:109.88802 code_gen:27.20001 inductor_compile:55.33653 backend_compile:92.64245 total_wall_time:145.3754 2024-08-20T22:49:58.6837507Z STATS: call_* op count: 456 | FakeTensorMode.__torch_dispatch__:53444 | attempt fast:2202 | fast is_contiguous:2021 | slow no contiguity match:181 | ProxyTorchDispatchMode.__torch_dispatch__:14386 | FakeTensor.__torch_dispatch__:5942 2024-08-20T22:49:58.6839363Z Dynamo produced 2 graphs covering 456 ops with 6 graph breaks (5 unique) 2024-08-20T22:50:15.5582288Z 2024-08-20T22:50:17.2197715Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:50:17.2198626Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:50:17.2199521Z cuda train convit_base 2024-08-20T22:50:17.2522319Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-08-20T22:53:21.5348170Z pass 2024-08-20T22:53:21.5593682Z TIMING: entire_frame_compile:181.16887 code_gen:35.50445 inductor_compile:82.43748 backend_compile:153.40937 total_wall_time:232.68469 2024-08-20T22:53:21.5597456Z STATS: call_* op count: 1142 | FakeTensorMode.__torch_dispatch__:130821 | attempt fast:6927 | slow no contiguity match:303 | fast is_contiguous:6624 | ProxyTorchDispatchMode.__torch_dispatch__:40209 | FakeTensor.__torch_dispatch__:15583 2024-08-20T22:53:21.5599026Z Dynamo produced 3 graphs covering 1142 ops with 7 graph breaks (5 unique) 2024-08-20T22:53:35.2064720Z 2024-08-20T22:53:36.1110646Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:36.1111282Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:36.1114336Z cuda train convmixer_768_32 2024-08-20T22:54:57.7726509Z pass 2024-08-20T22:54:57.7911825Z TIMING: entire_frame_compile:71.58507 code_gen:13.79563 inductor_compile:33.53303 backend_compile:56.55332 total_wall_time:87.30721 2024-08-20T22:54:57.7917884Z STATS: call_* op count: 300 | FakeTensorMode.__torch_dispatch__:51303 | attempt fast:2379 | fast is_contiguous:2379 | FakeTensor.__torch_dispatch__:10796 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-08-20T22:54:57.7919365Z Dynamo produced 2 graphs covering 300 ops with 5 graph breaks (4 unique) 2024-08-20T22:55:05.2691698Z 2024-08-20T22:55:07.0623884Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:07.0624500Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:55:07.0625059Z cuda train convnext_base 2024-08-20T22:56:53.8605691Z W0820 22:56:53.859000 15107 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:58:19.1866835Z pass 2024-08-20T22:58:19.2375917Z TIMING: entire_frame_compile:176.29276 code_gen:48.51791 inductor_compile:92.49695 backend_compile:143.01762 total_wall_time:205.17932 2024-08-20T22:58:19.2382573Z STATS: call_* op count: 1508 | FakeTensorMode.__torch_dispatch__:103745 | attempt fast:4593 | fast is_contiguous:4023 | fast channels_last:284 | slow no contiguity match:286 | ProxyTorchDispatchMode.__torch_dispatch__:35464 | FakeTensor.__torch_dispatch__:27856 2024-08-20T22:58:19.2384312Z Dynamo produced 3 graphs covering 1508 ops with 7 graph breaks (5 unique) 2024-08-20T22:58:32.0450710Z 2024-08-20T22:58:32.8484156Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:32.8484798Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:58:32.8485320Z cuda train crossvit_9_240 2024-08-20T23:00:46.7946807Z W0820 23:00:46.793000 15817 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:01:58.6848626Z pass 2024-08-20T23:01:58.6983280Z TIMING: entire_frame_compile:183.90699 code_gen:42.45357 inductor_compile:89.94555 backend_compile:151.30549 total_wall_time:216.51718 2024-08-20T23:01:58.6986317Z STATS: call_* op count: 1312 | FakeTensorMode.__torch_dispatch__:104447 | attempt fast:4653 | fast is_contiguous:4611 | FakeTensor.__torch_dispatch__:23608 | slow no contiguity match:42 | ProxyTorchDispatchMode.__torch_dispatch__:34361 2024-08-20T23:01:58.6987867Z Dynamo produced 3 graphs covering 1312 ops with 7 graph breaks (5 unique) 2024-08-20T23:02:21.5405242Z 2024-08-20T23:02:22.7483984Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:02:22.7486239Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:02:22.7487209Z cuda train cspdarknet53 2024-08-20T23:04:02.4702112Z W0820 23:04:02.469000 16543 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:04:58.1339983Z pass 2024-08-20T23:04:58.1496248Z TIMING: entire_frame_compile:144.41003 code_gen:38.46386 inductor_compile:76.06798 backend_compile:115.87383 total_wall_time:169.64932 2024-08-20T23:04:58.1499486Z STATS: call_* op count: 950 | FakeTensorMode.__torch_dispatch__:72090 | attempt fast:3256 | fast is_contiguous:3231 | FakeTensor.__torch_dispatch__:22488 | slow no contiguity match:25 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-08-20T23:04:58.1501080Z Dynamo produced 3 graphs covering 950 ops with 7 graph breaks (5 unique) 2024-08-20T23:05:08.6638025Z 2024-08-20T23:05:10.2421198Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:05:10.2422385Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:05:10.2423407Z cuda train deit_base_distilled_patch16_224 2024-08-20T23:06:04.9422363Z W0820 23:06:04.941000 17380 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:06:43.5343538Z pass 2024-08-20T23:06:43.5855800Z TIMING: entire_frame_compile:83.33493 code_gen:22.88692 inductor_compile:44.19226 backend_compile:68.57183 total_wall_time:96.66289 2024-08-20T23:06:43.5857464Z STATS: call_* op count: 753 | FakeTensorMode.__torch_dispatch__:54426 | attempt fast:2348 | fast is_contiguous:2348 | FakeTensor.__torch_dispatch__:13132 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-08-20T23:06:43.5858842Z Dynamo produced 3 graphs covering 753 ops with 7 graph breaks (5 unique) 2024-08-20T23:06:51.4502387Z 2024-08-20T23:06:52.7804819Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:06:52.7805302Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:06:52.7807638Z cuda train dla102 2024-08-20T23:09:03.0446576Z W0820 23:09:03.043000 17673 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:10:28.7588512Z pass 2024-08-20T23:10:28.7816553Z TIMING: entire_frame_compile:200.6648 code_gen:57.49202 inductor_compile:108.66518 backend_compile:165.18369 total_wall_time:232.77072 2024-08-20T23:10:28.7818594Z STATS: call_* op count: 1457 | FakeTensorMode.__torch_dispatch__:107283 | attempt fast:3911 | fast is_contiguous:3833 | FakeTensor.__torch_dispatch__:32523 | ProxyTorchDispatchMode.__torch_dispatch__:36994 | slow no contiguity match:78 2024-08-20T23:10:28.7821275Z Dynamo produced 3 graphs covering 1457 ops with 7 graph breaks (5 unique) 2024-08-20T23:10:42.3409039Z 2024-08-20T23:10:44.1018955Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:10:44.1019518Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:10:44.1019957Z cuda train dm_nfnet_f0 2024-08-20T23:12:08.8891506Z pass 2024-08-20T23:12:08.9405739Z TIMING: entire_frame_compile:77.28067 code_gen:19.4536 inductor_compile:41.01824 backend_compile:63.68874 total_wall_time:102.9313 2024-08-20T23:12:08.9408593Z STATS: call_* op count: 594 | FakeTensorMode.__torch_dispatch__:48588 | attempt fast:2906 | fast is_contiguous:2906 | ProxyTorchDispatchMode.__torch_dispatch__:16005 | FakeTensor.__torch_dispatch__:11566 2024-08-20T23:12:08.9411511Z Dynamo produced 2 graphs covering 594 ops with 6 graph breaks (5 unique) 2024-08-20T23:12:16.4929567Z 2024-08-20T23:12:18.0849210Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:12:18.0849819Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:12:18.0850373Z cuda train dpn107 2024-08-20T23:15:34.5375278Z pass 2024-08-20T23:15:34.6001687Z TIMING: entire_frame_compile:175.97766 code_gen:40.42976 inductor_compile:86.12034 backend_compile:146.94315 total_wall_time:218.14576 2024-08-20T23:15:34.6004408Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:93328 | attempt fast:4068 | fast is_contiguous:3769 | FakeTensor.__torch_dispatch__:18198 | slow no contiguity match:299 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-08-20T23:15:34.6006312Z Dynamo produced 2 graphs covering 634 ops with 6 graph breaks (5 unique) 2024-08-20T23:15:54.5597857Z 2024-08-20T23:15:55.5430781Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:55.5431371Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:55.5431830Z cuda train eca_botnext26ts_256 2024-08-20T23:17:21.8062441Z W0820 23:17:21.805000 21084 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:17:53.1642487Z pass 2024-08-20T23:17:53.1761302Z TIMING: entire_frame_compile:107.66551 code_gen:32.01117 inductor_compile:58.75769 backend_compile:89.60569 total_wall_time:131.02122 2024-08-20T23:17:53.1763563Z STATS: call_* op count: 635 | FakeTensorMode.__torch_dispatch__:50898 | attempt fast:1817 | fast is_contiguous:1756 | FakeTensor.__torch_dispatch__:12627 | slow no contiguity match:61 | ProxyTorchDispatchMode.__torch_dispatch__:15134 2024-08-20T23:17:53.1765100Z Dynamo produced 3 graphs covering 635 ops with 7 graph breaks (5 unique) 2024-08-20T23:18:04.4005295Z 2024-08-20T23:18:05.3978815Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:18:05.3979420Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:18:05.3979920Z cuda train eca_halonext26ts 2024-08-20T23:19:43.9820074Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:19:43.9821281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:19:43.9825240Z pred = mod(*cloned_inputs) 2024-08-20T23:19:43.9826686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1433, in forward 2024-08-20T23:19:43.9827598Z x = self.forward_features(x) 2024-08-20T23:19:43.9829013Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1425, in forward_features 2024-08-20T23:19:43.9829992Z x = self.stages(x) 2024-08-20T23:19:43.9830897Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 894, in forward 2024-08-20T23:19:43.9831720Z x = self.self_attn(x) 2024-08-20T23:19:43.9832545Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-08-20T23:19:43.9833650Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-08-20T23:19:43.9834247Z 2024-08-20T23:19:50.9784523Z W0820 23:19:50.977000 21773 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:20:22.9657233Z pass 2024-08-20T23:20:22.9733503Z TIMING: entire_frame_compile:126.6531 code_gen:39.01742 inductor_compile:70.94826 backend_compile:106.23676 total_wall_time:158.09724 2024-08-20T23:20:22.9735309Z STATS: call_* op count: 668 | FakeTensorMode.__torch_dispatch__:52552 | attempt fast:1807 | fast is_contiguous:1746 | FakeTensor.__torch_dispatch__:12393 | slow no contiguity match:61 | ProxyTorchDispatchMode.__torch_dispatch__:15754 2024-08-20T23:20:22.9737891Z Dynamo produced 3 graphs covering 668 ops with 7 graph breaks (5 unique) 2024-08-20T23:20:38.0915786Z 2024-08-20T23:20:38.9449072Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:38.9449684Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:38.9450152Z cuda train ese_vovnet19b_dw 2024-08-20T23:21:28.9123554Z W0820 23:21:28.911000 22250 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:21:56.7845236Z pass 2024-08-20T23:21:56.7925283Z TIMING: entire_frame_compile:71.11493 code_gen:27.62002 inductor_compile:42.92903 backend_compile:59.16808 total_wall_time:88.46538 2024-08-20T23:21:56.7928401Z STATS: call_* op count: 434 | FakeTensorMode.__torch_dispatch__:32141 | attempt fast:1009 | fast is_contiguous:985 | FakeTensor.__torch_dispatch__:9254 | ProxyTorchDispatchMode.__torch_dispatch__:10392 | slow no contiguity match:24 2024-08-20T23:21:56.7931407Z Dynamo produced 4 graphs covering 434 ops with 7 graph breaks (5 unique) 2024-08-20T23:22:03.7082295Z 2024-08-20T23:22:04.5899343Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:04.5899966Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:04.5900527Z cuda train fbnetc_100 2024-08-20T23:23:20.6366742Z W0820 23:23:20.635000 22866 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:24:16.5119957Z pass 2024-08-20T23:24:16.5167441Z TIMING: entire_frame_compile:120.63999 code_gen:36.1624 inductor_compile:66.37544 backend_compile:97.6971 total_wall_time:138.14959 2024-08-20T23:24:16.5169174Z STATS: call_* op count: 870 | FakeTensorMode.__torch_dispatch__:63257 | attempt fast:2264 | fast is_contiguous:2264 | FakeTensor.__torch_dispatch__:20059 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-08-20T23:24:16.5170534Z Dynamo produced 4 graphs covering 870 ops with 7 graph breaks (5 unique) 2024-08-20T23:24:26.2582059Z 2024-08-20T23:24:27.3310934Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:24:27.3311563Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:24:27.3312096Z cuda train fbnetv3_b 2024-08-20T23:27:20.4775260Z pass 2024-08-20T23:27:20.4861023Z TIMING: entire_frame_compile:163.50307 code_gen:33.51288 inductor_compile:77.35806 backend_compile:131.71073 total_wall_time:204.67892 2024-08-20T23:27:20.4862699Z STATS: call_* op count: 459 | FakeTensorMode.__torch_dispatch__:89032 | attempt fast:5511 | fast is_contiguous:5511 | FakeTensor.__torch_dispatch__:19826 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-08-20T23:27:20.4864057Z Dynamo produced 2 graphs covering 459 ops with 6 graph breaks (5 unique) 2024-08-20T23:27:33.1325512Z 2024-08-20T23:27:34.3591232Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:27:34.3591756Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:27:34.3592195Z cuda train gernet_l 2024-08-20T23:28:42.0571578Z pass 2024-08-20T23:28:42.0796556Z TIMING: entire_frame_compile:62.08109 code_gen:13.94978 inductor_compile:30.3333 backend_compile:50.02786 total_wall_time:76.11234 2024-08-20T23:28:42.0799452Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:38822 | attempt fast:2021 | fast is_contiguous:2021 | FakeTensor.__torch_dispatch__:8864 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-08-20T23:28:42.0800973Z Dynamo produced 2 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-08-20T23:28:49.1981903Z 2024-08-20T23:28:50.1655933Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:50.1656409Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:28:50.1660813Z cuda train ghostnet_100 2024-08-20T23:31:30.6964906Z pass 2024-08-20T23:31:30.7015203Z TIMING: entire_frame_compile:142.61002 code_gen:32.13124 inductor_compile:69.05591 backend_compile:116.9712 total_wall_time:177.59069 2024-08-20T23:31:30.7018139Z STATS: call_* op count: 405 | FakeTensorMode.__torch_dispatch__:65679 | attempt fast:3045 | fast is_contiguous:2981 | FakeTensor.__torch_dispatch__:13766 | ProxyTorchDispatchMode.__torch_dispatch__:19864 | slow no contiguity match:64 2024-08-20T23:31:30.7019678Z Dynamo produced 2 graphs covering 405 ops with 6 graph breaks (5 unique) 2024-08-20T23:31:55.2013009Z 2024-08-20T23:31:56.4671597Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:31:56.4673797Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:31:56.4674287Z cuda train gluon_inception_v3 2024-08-20T23:33:01.5502107Z W0820 23:33:01.549000 26961 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:34:15.4258270Z pass 2024-08-20T23:34:15.4491644Z TIMING: entire_frame_compile:128.32841 inductor_compile:44.76228 backend_compile:95.43545 code_gen:23.51549 total_wall_time:132.31536 2024-08-20T23:34:15.4494583Z STATS: call_* op count: 1281 | FakeTensorMode.__torch_dispatch__:78217 | attempt fast:2261 | fast is_contiguous:2261 | FakeTensor.__torch_dispatch__:14446 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-08-20T23:34:15.4496574Z Dynamo produced 3 graphs covering 1281 ops with 7 graph breaks (5 unique) 2024-08-20T23:34:25.4759973Z 2024-08-20T23:34:26.4238056Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:26.4238530Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:34:26.4238969Z cuda train gmixer_24_224 2024-08-20T23:36:31.6217252Z pass 2024-08-20T23:36:31.6470653Z TIMING: entire_frame_compile:117.4898 code_gen:22.63771 inductor_compile:53.67533 backend_compile:96.89948 total_wall_time:149.15504 2024-08-20T23:36:31.6473196Z STATS: call_* op count: 587 | FakeTensorMode.__torch_dispatch__:88773 | attempt fast:4737 | fast is_contiguous:4017 | slow no contiguity match:720 | ProxyTorchDispatchMode.__torch_dispatch__:23826 | FakeTensor.__torch_dispatch__:11028 2024-08-20T23:36:31.6474775Z Dynamo produced 2 graphs covering 587 ops with 6 graph breaks (5 unique) 2024-08-20T23:36:41.0224092Z 2024-08-20T23:36:41.8781080Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:36:41.8782230Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:36:41.8782772Z cuda train gmlp_s16_224 2024-08-20T23:38:32.3068852Z W0820 23:38:32.305000 27488 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:39:45.3001042Z pass 2024-08-20T23:39:45.3211043Z TIMING: entire_frame_compile:171.3833 code_gen:44.92696 inductor_compile:89.81817 backend_compile:141.46094 total_wall_time:202.89684 2024-08-20T23:39:45.3212862Z STATS: call_* op count: 1396 | FakeTensorMode.__torch_dispatch__:108077 | attempt fast:5367 | fast is_contiguous:4917 | slow no contiguity match:450 | ProxyTorchDispatchMode.__torch_dispatch__:35364 | FakeTensor.__torch_dispatch__:26710 2024-08-20T23:39:45.3217281Z Dynamo produced 3 graphs covering 1396 ops with 7 graph breaks (5 unique) 2024-08-20T23:39:57.0866580Z 2024-08-20T23:39:58.7806082Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:58.7806583Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:39:58.7807032Z cuda train hrnet_w18 2024-08-20T23:45:46.2038860Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:45:46.2040175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:45:46.2044547Z pred = mod(*cloned_inputs) 2024-08-20T23:45:46.2045762Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-08-20T23:45:46.2046568Z y = self.forward_features(x) 2024-08-20T23:45:46.2047476Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-08-20T23:45:46.2048608Z yl = self.stages(x) 2024-08-20T23:45:46.2049567Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-08-20T23:45:46.2050352Z yl = self.stage2(xl) 2024-08-20T23:45:46.2051157Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-08-20T23:45:46.2051916Z x = module(x) 2024-08-20T23:45:46.2052696Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-08-20T23:45:46.2053442Z y = y + f(x[j]) 2024-08-20T23:45:46.2053646Z 2024-08-20T23:46:09.7377538Z pass 2024-08-20T23:46:09.7631042Z TIMING: entire_frame_compile:354.31302 code_gen:78.7191 inductor_compile:170.65217 backend_compile:288.52491 total_wall_time:433.50173 2024-08-20T23:46:09.7632709Z STATS: call_* op count: 1475 | FakeTensorMode.__torch_dispatch__:219858 | attempt fast:11855 | fast is_contiguous:11855 | FakeTensor.__torch_dispatch__:46888 | ProxyTorchDispatchMode.__torch_dispatch__:71305 2024-08-20T23:46:09.7636455Z Dynamo produced 2 graphs covering 1475 ops with 5 graph breaks (4 unique) 2024-08-20T23:46:33.6019018Z 2024-08-20T23:46:34.8559879Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:34.8562158Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:34.8562878Z cuda train inception_v3 2024-08-20T23:47:41.6735955Z pass 2024-08-20T23:47:41.6990830Z TIMING: entire_frame_compile:60.49339 inductor_compile:7.14346 backend_compile:41.09141 total_wall_time:64.43412 2024-08-20T23:47:41.6994277Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:48817 | attempt fast:2261 | fast is_contiguous:2261 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-08-20T23:47:41.6995680Z Dynamo produced 2 graphs covering 412 ops with 6 graph breaks (5 unique) 2024-08-20T23:47:48.6357133Z 2024-08-20T23:47:50.1845440Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:50.1845927Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:47:50.1846375Z cuda train jx_nest_base 2024-08-20T23:50:49.2878902Z W0820 23:50:49.286000 29048 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:52:09.2847810Z pass 2024-08-20T23:52:09.3254137Z TIMING: entire_frame_compile:236.25276 code_gen:60.33532 inductor_compile:119.16581 backend_compile:196.54153 total_wall_time:277.01006 2024-08-20T23:52:09.3256272Z STATS: call_* op count: 1671 | FakeTensorMode.__torch_dispatch__:145499 | attempt fast:5914 | fast is_contiguous:5858 | ProxyTorchDispatchMode.__torch_dispatch__:43281 | fast channels_last:24 | slow no contiguity match:32 | FakeTensor.__torch_dispatch__:27886 2024-08-20T23:52:09.3257935Z Dynamo produced 3 graphs covering 1671 ops with 7 graph breaks (5 unique) 2024-08-20T23:52:27.7443405Z 2024-08-20T23:52:28.5310421Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:28.5311330Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:28.5312204Z cuda train lcnet_050 2024-08-20T23:53:22.8611801Z pass 2024-08-20T23:53:22.8637015Z TIMING: entire_frame_compile:50.8207 code_gen:11.514 inductor_compile:24.68811 backend_compile:41.43601 total_wall_time:63.39375 2024-08-20T23:53:22.8640908Z STATS: call_* op count: 129 | FakeTensorMode.__torch_dispatch__:26833 | attempt fast:1707 | fast is_contiguous:1707 | FakeTensor.__torch_dispatch__:6048 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-08-20T23:53:22.8642282Z Dynamo produced 2 graphs covering 129 ops with 6 graph breaks (5 unique) 2024-08-20T23:53:29.3308982Z 2024-08-20T23:53:30.4220230Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:30.4221159Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:53:30.4222012Z cuda train levit_128 2024-08-20T23:55:53.1888309Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:55:53.1889490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:55:53.1894313Z pred = mod(*cloned_inputs) 2024-08-20T23:55:53.1895667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 718, in forward 2024-08-20T23:55:53.1896508Z x = self.forward_features(x) 2024-08-20T23:55:53.1897420Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 709, in forward_features 2024-08-20T23:55:53.1899680Z x = self.stages(x) 2024-08-20T23:55:53.1900839Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 520, in forward 2024-08-20T23:55:53.1901631Z x = self.blocks(x) 2024-08-20T23:55:53.1902458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 458, in forward 2024-08-20T23:55:53.1903261Z x = x + self.drop_path1(self.attn(x)) 2024-08-20T23:55:53.1904151Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 237, in forward 2024-08-20T23:55:53.1905053Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-08-20T23:55:53.1906609Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 212, in get_attention_biases 2024-08-20T23:55:53.1907590Z return self.attention_biases[:, self.attention_bias_idxs] 2024-08-20T23:55:53.1908001Z 2024-08-20T23:56:01.1101508Z W0820 23:56:01.109000 30326 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:56:53.6100731Z pass 2024-08-20T23:56:53.6138855Z TIMING: entire_frame_compile:189.53057 code_gen:51.17762 inductor_compile:96.68115 backend_compile:157.88744 total_wall_time:227.38804 2024-08-20T23:56:53.6140761Z STATS: call_* op count: 1261 | FakeTensorMode.__torch_dispatch__:96925 | attempt fast:3507 | fast is_contiguous:3459 | FakeTensor.__torch_dispatch__:22761 | slow no contiguity match:46 | ProxyTorchDispatchMode.__torch_dispatch__:32073 | fast channels_last:2 2024-08-20T23:56:53.6144284Z Dynamo produced 3 graphs covering 1261 ops with 7 graph breaks (5 unique) 2024-08-20T23:57:12.8767766Z 2024-08-20T23:57:14.1028404Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:57:14.1029116Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:57:14.1031372Z cuda train mixer_b16_224 2024-08-20T23:58:13.2149323Z W0820 23:58:13.214000 31358 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:58:50.0374452Z pass 2024-08-20T23:58:50.0803254Z TIMING: entire_frame_compile:87.17214 code_gen:23.64602 inductor_compile:45.62928 backend_compile:72.06978 total_wall_time:102.17976 2024-08-20T23:58:50.0805046Z STATS: call_* op count: 670 | FakeTensorMode.__torch_dispatch__:53162 | attempt fast:2769 | fast is_contiguous:2625 | slow no contiguity match:144 | ProxyTorchDispatchMode.__torch_dispatch__:17868 | FakeTensor.__torch_dispatch__:13293 2024-08-20T23:58:50.0806578Z Dynamo produced 3 graphs covering 670 ops with 7 graph breaks (5 unique) 2024-08-20T23:58:54.7623595Z accuracy pass_rate=96.67% 2024-08-20T23:58:54.7624504Z calls_captured gmean=0.00x mean=779.167x 2024-08-20T23:58:54.7627662Z unique_graphs gmean=0.00x mean=2.533x 2024-08-20T23:58:54.7633175Z graph_breaks gmean=0.00x mean=6.267x 2024-08-20T23:58:54.7636814Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-08-20T23:58:54.7640458Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:58:54.7644009Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:58:54.7647740Z cudagraph_skips gmean=0.00x mean=0.133x 2024-08-20T23:58:57.0027066Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv 2024-08-20T23:58:57.2672469Z adv_inception_v3 PASS 2024-08-20T23:58:57.2678307Z beit_base_patch16_224 PASS 2024-08-20T23:58:57.2683810Z botnet26t_256 PASS 2024-08-20T23:58:57.2688732Z cait_m36_384 XFAIL 2024-08-20T23:58:57.2694268Z coat_lite_mini PASS 2024-08-20T23:58:57.2699519Z convit_base PASS 2024-08-20T23:58:57.2704645Z convmixer_768_32 PASS 2024-08-20T23:58:57.2710094Z convnext_base PASS 2024-08-20T23:58:57.2715417Z crossvit_9_240 PASS 2024-08-20T23:58:57.2720444Z cspdarknet53 PASS 2024-08-20T23:58:57.2725617Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:58:57.2731634Z dla102 PASS 2024-08-20T23:58:57.2737543Z dm_nfnet_f0 PASS 2024-08-20T23:58:57.2742654Z dpn107 PASS 2024-08-20T23:58:57.2748052Z eca_botnext26ts_256 PASS 2024-08-20T23:58:57.2753462Z eca_halonext26ts PASS 2024-08-20T23:58:57.2758511Z ese_vovnet19b_dw PASS 2024-08-20T23:58:57.2763982Z fbnetc_100 PASS 2024-08-20T23:58:57.2769055Z fbnetv3_b PASS 2024-08-20T23:58:57.2774158Z gernet_l PASS 2024-08-20T23:58:57.2779464Z ghostnet_100 PASS 2024-08-20T23:58:57.2784478Z gluon_inception_v3 PASS 2024-08-20T23:58:57.2790009Z gmixer_24_224 PASS 2024-08-20T23:58:57.2794905Z gmlp_s16_224 PASS 2024-08-20T23:58:57.2800070Z hrnet_w18 PASS 2024-08-20T23:58:57.2805357Z inception_v3 PASS 2024-08-20T23:58:57.2810654Z jx_nest_base PASS 2024-08-20T23:58:57.2815633Z lcnet_050 PASS 2024-08-20T23:58:57.2820644Z levit_128 PASS 2024-08-20T23:58:57.2825851Z mixer_b16_224 PASS 2024-08-20T23:58:57.3251248Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_timm_training.csv 2024-08-20T23:58:57.5917458Z adv_inception_v3 PASS 2024-08-20T23:58:57.5920523Z beit_base_patch16_224 PASS 2024-08-20T23:58:57.5923305Z botnet26t_256 PASS 2024-08-20T23:58:57.5927790Z cait_m36_384 PASS 2024-08-20T23:58:57.5934779Z coat_lite_mini PASS 2024-08-20T23:58:57.5938839Z convit_base PASS 2024-08-20T23:58:57.5943231Z convmixer_768_32 PASS 2024-08-20T23:58:57.5948084Z convnext_base PASS 2024-08-20T23:58:57.5952640Z crossvit_9_240 PASS 2024-08-20T23:58:57.5957189Z cspdarknet53 PASS 2024-08-20T23:58:57.5961640Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:58:57.5966284Z dla102 PASS 2024-08-20T23:58:57.5970752Z dm_nfnet_f0 PASS 2024-08-20T23:58:57.5975285Z dpn107 PASS 2024-08-20T23:58:57.5979850Z eca_botnext26ts_256 PASS 2024-08-20T23:58:57.5984405Z eca_halonext26ts PASS 2024-08-20T23:58:57.5989090Z ese_vovnet19b_dw PASS 2024-08-20T23:58:57.5993633Z fbnetc_100 PASS 2024-08-20T23:58:57.5998123Z fbnetv3_b PASS 2024-08-20T23:58:57.6002653Z gernet_l PASS 2024-08-20T23:58:57.6007126Z ghostnet_100 PASS 2024-08-20T23:58:57.6011684Z gluon_inception_v3 PASS 2024-08-20T23:58:57.6016342Z gmixer_24_224 PASS 2024-08-20T23:58:57.6020883Z gmlp_s16_224 PASS 2024-08-20T23:58:57.6025381Z hrnet_w18 PASS 2024-08-20T23:58:57.6030546Z inception_v3 PASS 2024-08-20T23:58:57.6034915Z jx_nest_base PASS 2024-08-20T23:58:57.6039404Z lcnet_050 PASS 2024-08-20T23:58:57.6043821Z levit_128 PASS 2024-08-20T23:58:57.6048412Z mixer_b16_224 PASS 2024-08-20T23:58:57.6493042Z + cleanup_workspace 2024-08-20T23:58:57.6494159Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:58:57.6495265Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:58:57.6496236Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:58:57.6497025Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:58:57.6497900Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:58:57.6498813Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:58:57.6499530Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:58:58.4027115Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:58:58.4027657Z cat test/**/*_toprint.log || true 2024-08-20T23:58:58.4042654Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.4043164Z env: 2024-08-20T23:58:58.4043457Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.4043997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.4044730Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.4045369Z ##[endgroup] 2024-08-20T23:58:58.4139627Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:58:58.4174340Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:58:58.4174828Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:58:58.4183710Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.4184216Z env: 2024-08-20T23:58:58.4184509Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.4184983Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.4185726Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.4186480Z MONITOR_SCRIPT_PID: 183516 2024-08-20T23:58:58.4186857Z ##[endgroup] 2024-08-20T23:58:58.4392495Z Prepare all required actions 2024-08-20T23:58:58.4392963Z Getting action download info 2024-08-20T23:58:58.6682087Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:58:58.8319756Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:58:58.8320234Z with: 2024-08-20T23:58:58.8320826Z file-suffix: test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971 2024-08-20T23:58:58.8321549Z s3-bucket: gha-artifacts 2024-08-20T23:58:58.8321894Z env: 2024-08-20T23:58:58.8322171Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.8322639Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.8323386Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.8324034Z ##[endgroup] 2024-08-20T23:58:58.8351534Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:58:58.8352217Z # Remove any previous test jsons if they exist 2024-08-20T23:58:58.8352735Z rm -f test-jsons-*.zip 2024-08-20T23:58:58.8353271Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:58:58.8362216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.8362736Z env: 2024-08-20T23:58:58.8363017Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.8363484Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.8364226Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.8365165Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971 2024-08-20T23:58:58.8365850Z ##[endgroup] 2024-08-20T23:58:58.8644932Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:58:58.8674401Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:58:58.8675141Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:58:58.8681587Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:58:58.8687301Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:58:58.8722342Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:58:58.8723010Z # Remove any previous test reports if they exist 2024-08-20T23:58:58.8724844Z rm -f test-reports-*.zip 2024-08-20T23:58:58.8725454Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:58:58.8734767Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.8735275Z env: 2024-08-20T23:58:58.8735559Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.8736029Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.8736764Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.8737890Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971 2024-08-20T23:58:58.8738602Z ##[endgroup] 2024-08-20T23:58:58.8998284Z adding: test/test-reports/inference_timm_models.csv (deflated 68%) 2024-08-20T23:58:58.8999407Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-08-20T23:58:58.9003522Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-08-20T23:58:58.9004846Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-08-20T23:58:58.9038531Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:58:58.9039174Z # Remove any previous usage logs if they exist 2024-08-20T23:58:58.9039686Z rm -f logs-*.zip 2024-08-20T23:58:58.9040366Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:58:58.9041172Z # so check to see if the file exists first 2024-08-20T23:58:58.9041699Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:58:58.9042446Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:58:58.9042963Z fi 2024-08-20T23:58:58.9043346Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:58:58.9043934Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:58:58.9044432Z fi 2024-08-20T23:58:58.9053091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.9053611Z env: 2024-08-20T23:58:58.9053905Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.9054374Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.9055121Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.9056064Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971 2024-08-20T23:58:58.9056769Z ##[endgroup] 2024-08-20T23:58:58.9156903Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:58:58.9245901Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:58:58.9246656Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:58:58.9247232Z rm -f debug-*.zip 2024-08-20T23:58:58.9247635Z if [ -d 'test/debug' ]; then 2024-08-20T23:58:58.9248161Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:58:58.9248647Z fi 2024-08-20T23:58:58.9256748Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:58:58.9257265Z env: 2024-08-20T23:58:58.9257552Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.9258014Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.9258765Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.9259713Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971 2024-08-20T23:58:58.9260417Z ##[endgroup] 2024-08-20T23:58:58.9354678Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:58:58.9355120Z with: 2024-08-20T23:58:58.9355408Z s3-bucket: gha-artifacts 2024-08-20T23:58:58.9355843Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:58:58.9356325Z retention-days: 14 2024-08-20T23:58:58.9356667Z if-no-files-found: warn 2024-08-20T23:58:58.9357020Z path: test-jsons-*.zip 2024-08-20T23:58:58.9357370Z name: artifact 2024-08-20T23:58:58.9357769Z region: us-east-1 2024-08-20T23:58:58.9358105Z env: 2024-08-20T23:58:58.9358375Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:58.9358837Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:58.9359573Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:58.9360210Z ##[endgroup] 2024-08-20T23:58:59.2985312Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:58:59.2986056Z With the provided path, there will be 1 file uploaded 2024-08-20T23:58:59.2986895Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:58:59.3042269Z Starting upload of test-jsons-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:58:59.4016481Z Finished upload of test-jsons-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:58:59.4297869Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:58:59.4298319Z with: 2024-08-20T23:58:59.4298608Z s3-bucket: gha-artifacts 2024-08-20T23:58:59.4299058Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:58:59.4299544Z retention-days: 14 2024-08-20T23:58:59.4299896Z if-no-files-found: error 2024-08-20T23:58:59.4300304Z path: test-reports-*.zip 2024-08-20T23:58:59.4300679Z name: artifact 2024-08-20T23:58:59.4300995Z region: us-east-1 2024-08-20T23:58:59.4301304Z env: 2024-08-20T23:58:59.4301578Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:59.4302044Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:59.4302800Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:59.4303625Z ##[endgroup] 2024-08-20T23:58:59.7926889Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:58:59.7927504Z With the provided path, there will be 1 file uploaded 2024-08-20T23:58:59.7928140Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:58:59.7983922Z Starting upload of test-reports-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:58:59.8836847Z Finished upload of test-reports-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:58:59.9122198Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:58:59.9122647Z with: 2024-08-20T23:58:59.9122938Z s3-bucket: gha-artifacts 2024-08-20T23:58:59.9123384Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:58:59.9123863Z retention-days: 14 2024-08-20T23:58:59.9124212Z if-no-files-found: ignore 2024-08-20T23:58:59.9124591Z path: logs-*.zip 2024-08-20T23:58:59.9124923Z name: artifact 2024-08-20T23:58:59.9125228Z region: us-east-1 2024-08-20T23:58:59.9125537Z env: 2024-08-20T23:58:59.9125822Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:58:59.9126292Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:58:59.9127028Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:58:59.9127666Z ##[endgroup] 2024-08-20T23:59:00.2703205Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:59:00.2704040Z With the provided path, there will be 1 file uploaded 2024-08-20T23:59:00.2704807Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:59:00.2760881Z Starting upload of logs-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:59:00.4142014Z Finished upload of logs-test-dynamic_inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528971.zip 2024-08-20T23:59:00.4419242Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:59:00.4419714Z with: 2024-08-20T23:59:00.4419998Z s3-bucket: gha-artifacts 2024-08-20T23:59:00.4420445Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:59:00.4420931Z retention-days: 14 2024-08-20T23:59:00.4421276Z if-no-files-found: ignore 2024-08-20T23:59:00.4421635Z path: debug-*.zip 2024-08-20T23:59:00.4421957Z name: artifact 2024-08-20T23:59:00.4422272Z region: us-east-1 2024-08-20T23:59:00.4422572Z env: 2024-08-20T23:59:00.4422851Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:59:00.4423316Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:59:00.4424049Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:00.4424673Z ##[endgroup] 2024-08-20T23:59:00.7914811Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:59:00.8195955Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:59:00.8196636Z # shellcheck disable=SC2156 2024-08-20T23:59:00.8197447Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:59:00.8206636Z shell: /usr/bin/bash -e {0} 2024-08-20T23:59:00.8206996Z env: 2024-08-20T23:59:00.8207288Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:59:00.8207761Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:59:00.8208497Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:00.8209143Z ##[endgroup] 2024-08-20T23:59:01.0815212Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:59:01.0815784Z with: 2024-08-20T23:59:01.0816045Z env: 2024-08-20T23:59:01.0816335Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:59:01.0816806Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:59:01.0817539Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:01.0818192Z ##[endgroup] 2024-08-20T23:59:01.0838330Z ##[group]Run set -eou pipefail 2024-08-20T23:59:01.0838746Z set -eou pipefail 2024-08-20T23:59:01.0839103Z  2024-08-20T23:59:01.0839628Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:59:01.0840281Z for _ in $(seq 1440); do 2024-08-20T23:59:01.0840756Z  # Break if no ssh session exists anymore 2024-08-20T23:59:01.0841260Z  if [ "$(who)" = "" ]; then 2024-08-20T23:59:01.0841674Z  break 2024-08-20T23:59:01.0842021Z  fi 2024-08-20T23:59:01.0842323Z  echo "." 2024-08-20T23:59:01.0842657Z  sleep 5 2024-08-20T23:59:01.0842981Z done 2024-08-20T23:59:01.0851964Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:59:01.0852466Z env: 2024-08-20T23:59:01.0852760Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:59:01.0853228Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:59:01.0853976Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:01.0854611Z ##[endgroup] 2024-08-20T23:59:01.0887174Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:59:01.0945981Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:59:01.0946926Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:59:01.0947550Z # shellcheck disable=SC2046 2024-08-20T23:59:01.0948053Z docker stop $(docker ps -q) || true 2024-08-20T23:59:01.0948563Z # Prune all of the docker images 2024-08-20T23:59:01.0949029Z docker system prune -af 2024-08-20T23:59:01.0957914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:59:01.0958433Z env: 2024-08-20T23:59:01.0958725Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:59:01.0959204Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:59:01.0959968Z DOCKER_CONTAINER_ID: 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:01.0960620Z ##[endgroup] 2024-08-20T23:59:01.6680883Z 289234f66ec0 2024-08-20T23:59:03.1259944Z Deleted Containers: 2024-08-20T23:59:03.1260636Z 289234f66ec075a9e73627f04fd8dc9142f731822981eef9111f8c71f041f509 2024-08-20T23:59:03.1261185Z 2024-08-20T23:59:14.2116133Z Deleted Images: 2024-08-20T23:59:14.2117817Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:59:14.2120609Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:59:14.2122207Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:59:14.2123279Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:59:14.2124763Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:59:14.2125603Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:59:14.2126408Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:59:14.2127281Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:59:14.2128113Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:59:14.2129506Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:59:14.2130347Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:59:14.2131198Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:59:14.2132025Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:59:14.2132859Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:59:14.2133692Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:59:14.2134533Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:59:14.2135356Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:59:14.2136200Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:59:14.2137045Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:59:14.2137881Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:59:14.2138712Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:59:14.2139536Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:59:14.2140379Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:59:14.2141225Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:59:14.2142062Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:59:14.2142888Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:59:14.2143720Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:59:14.2144622Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:59:14.2145466Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:59:14.2146457Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:59:14.2147375Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:59:14.2148212Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:59:14.2149063Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:59:14.2149885Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:59:14.2150717Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:59:14.2151554Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:59:14.2152392Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:59:14.2153214Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:59:14.2154052Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:59:14.2154892Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:59:14.2155730Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:59:14.2156706Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:59:14.2157552Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:59:14.2158407Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:59:14.2159251Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:59:14.2160065Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:59:14.2160998Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:59:14.2161855Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:59:14.2162698Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:59:14.2163508Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:59:14.2164359Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:59:14.2165227Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:59:14.2166066Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:59:14.2166884Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:59:14.2167703Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:59:14.2168546Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:59:14.2169393Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:59:14.2170235Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:59:14.2171065Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:59:14.2171899Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:59:14.2172732Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:59:14.2173568Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:59:14.2174411Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:59:14.2175261Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:59:14.2176108Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:59:14.2176936Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:59:14.2177768Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:59:14.2178595Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:59:14.2179426Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:59:14.2180274Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:59:14.2181123Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:59:14.2181944Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:59:14.2182768Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:59:14.2183601Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:59:14.2184485Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:59:14.2185314Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:59:14.2186160Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:59:14.2187126Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:59:14.2188078Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:59:14.2188921Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:59:14.2189757Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:59:14.2190263Z 2024-08-20T23:59:14.2190411Z Total reclaimed space: 44.92GB 2024-08-20T23:59:14.2259155Z Post job cleanup. 2024-08-20T23:59:14.2310636Z Post job cleanup. 2024-08-20T23:59:14.3205370Z [command]/usr/bin/git version 2024-08-20T23:59:14.3258583Z git version 2.40.1 2024-08-20T23:59:14.3297885Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c8e5ffaa-f4ee-49bf-b8db-445e3f3884ff' before making global git config changes 2024-08-20T23:59:14.3299191Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:59:14.3302917Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:59:14.3352776Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:59:14.3398112Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:59:14.3769685Z Entering 'android/libs/fbjni' 2024-08-20T23:59:14.3840686Z Entering 'third_party/FP16' 2024-08-20T23:59:14.3908991Z Entering 'third_party/FXdiv' 2024-08-20T23:59:14.3978351Z Entering 'third_party/NNPACK' 2024-08-20T23:59:14.4047767Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:59:14.4116650Z Entering 'third_party/XNNPACK' 2024-08-20T23:59:14.4199961Z Entering 'third_party/benchmark' 2024-08-20T23:59:14.4269143Z Entering 'third_party/cpp-httplib' 2024-08-20T23:59:14.4336227Z Entering 'third_party/cpuinfo' 2024-08-20T23:59:14.4406384Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:59:14.4474546Z Entering 'third_party/cutlass' 2024-08-20T23:59:14.4550296Z Entering 'third_party/eigen' 2024-08-20T23:59:14.4623332Z Entering 'third_party/fbgemm' 2024-08-20T23:59:14.4691834Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:59:14.4757979Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:59:14.4827900Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:59:14.4899339Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:59:14.4967345Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:59:14.5037407Z Entering 'third_party/flatbuffers' 2024-08-20T23:59:14.5109226Z Entering 'third_party/fmt' 2024-08-20T23:59:14.5176943Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:59:14.5245651Z Entering 'third_party/gloo' 2024-08-20T23:59:14.5315537Z Entering 'third_party/googletest' 2024-08-20T23:59:14.5383849Z Entering 'third_party/ideep' 2024-08-20T23:59:14.5451346Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:59:14.5526753Z Entering 'third_party/ittapi' 2024-08-20T23:59:14.5597224Z Entering 'third_party/kineto' 2024-08-20T23:59:14.5664869Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:59:14.5731321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:59:14.5799454Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:59:14.5871540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:59:14.5938726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:59:14.6008779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:59:14.6080093Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:59:14.6148409Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:59:14.6216059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:59:14.6286065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:59:14.6356521Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:59:14.6423013Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:59:14.6501338Z Entering 'third_party/mimalloc' 2024-08-20T23:59:14.6571850Z Entering 'third_party/nccl/nccl' 2024-08-20T23:59:14.6641241Z Entering 'third_party/nlohmann' 2024-08-20T23:59:14.6710447Z Entering 'third_party/onnx' 2024-08-20T23:59:14.6792222Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:59:14.6859742Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:59:14.6934085Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:59:14.7005392Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:59:14.7071314Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:59:14.7138713Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:59:14.7204402Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:59:14.7275478Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:59:14.7341390Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:59:14.7407957Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:59:14.7472571Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:59:14.7542241Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:59:14.7612677Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:59:14.7705569Z Entering 'third_party/pocketfft' 2024-08-20T23:59:14.7774184Z Entering 'third_party/protobuf' 2024-08-20T23:59:14.7845453Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:59:14.7911199Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:59:14.7983569Z Entering 'third_party/psimd' 2024-08-20T23:59:14.8052536Z Entering 'third_party/pthreadpool' 2024-08-20T23:59:14.8121336Z Entering 'third_party/pybind11' 2024-08-20T23:59:14.8191710Z Entering 'third_party/python-peachpy' 2024-08-20T23:59:14.8260980Z Entering 'third_party/sleef' 2024-08-20T23:59:14.8330223Z Entering 'third_party/tensorpipe' 2024-08-20T23:59:14.8397532Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:59:14.8466184Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:59:14.8532101Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:59:14.8598212Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:59:14.8662520Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:59:14.8758792Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:59:14.8796613Z http.https://github.com/.extraheader 2024-08-20T23:59:14.8807502Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:59:14.8856981Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:59:14.9224361Z Entering 'android/libs/fbjni' 2024-08-20T23:59:14.9268153Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9312162Z Entering 'third_party/FP16' 2024-08-20T23:59:14.9355926Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9399183Z Entering 'third_party/FXdiv' 2024-08-20T23:59:14.9444451Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9488314Z Entering 'third_party/NNPACK' 2024-08-20T23:59:14.9531845Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9575777Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:59:14.9622620Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9668607Z Entering 'third_party/XNNPACK' 2024-08-20T23:59:14.9711851Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9771532Z Entering 'third_party/benchmark' 2024-08-20T23:59:14.9814557Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9857732Z Entering 'third_party/cpp-httplib' 2024-08-20T23:59:14.9901267Z http.https://github.com/.extraheader 2024-08-20T23:59:14.9944847Z Entering 'third_party/cpuinfo' 2024-08-20T23:59:14.9992939Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0037953Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:59:15.0080879Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0124039Z Entering 'third_party/cutlass' 2024-08-20T23:59:15.0169433Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0220244Z Entering 'third_party/eigen' 2024-08-20T23:59:15.0264828Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0310117Z Entering 'third_party/fbgemm' 2024-08-20T23:59:15.0354777Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0397359Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:59:15.0440870Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0483798Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:59:15.0526206Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0570626Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:59:15.0612912Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0665480Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:59:15.0707248Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0750853Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:59:15.0793181Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0839234Z Entering 'third_party/flatbuffers' 2024-08-20T23:59:15.0882514Z http.https://github.com/.extraheader 2024-08-20T23:59:15.0937653Z Entering 'third_party/fmt' 2024-08-20T23:59:15.0981398Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1024439Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:59:15.1073182Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1116620Z Entering 'third_party/gloo' 2024-08-20T23:59:15.1160209Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1203122Z Entering 'third_party/googletest' 2024-08-20T23:59:15.1252293Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1295809Z Entering 'third_party/ideep' 2024-08-20T23:59:15.1343820Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1387052Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:59:15.1430347Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1481975Z Entering 'third_party/ittapi' 2024-08-20T23:59:15.1525256Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1570987Z Entering 'third_party/kineto' 2024-08-20T23:59:15.1614615Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1657949Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:59:15.1701115Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1744944Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:59:15.1788349Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1834716Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:59:15.1877855Z http.https://github.com/.extraheader 2024-08-20T23:59:15.1921246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:59:15.1965522Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2010659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:59:15.2053606Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2095428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:59:15.2138499Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2185831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:59:15.2228591Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2272978Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:59:15.2316019Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2360509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:59:15.2405590Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2453219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:59:15.2496522Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2543564Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:59:15.2586689Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2633685Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:59:15.2677001Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2722890Z Entering 'third_party/mimalloc' 2024-08-20T23:59:15.2767890Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2811299Z Entering 'third_party/nccl/nccl' 2024-08-20T23:59:15.2855721Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2899435Z Entering 'third_party/nlohmann' 2024-08-20T23:59:15.2944911Z http.https://github.com/.extraheader 2024-08-20T23:59:15.2989991Z Entering 'third_party/onnx' 2024-08-20T23:59:15.3036683Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3093565Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:59:15.3137968Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3182334Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:59:15.3225992Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3274912Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:59:15.3318870Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3363111Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:59:15.3405675Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3448706Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:59:15.3490945Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3534654Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:59:15.3576443Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3619904Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:59:15.3662685Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3709693Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:59:15.3751692Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3796025Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:59:15.3839052Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3881991Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:59:15.3924671Z http.https://github.com/.extraheader 2024-08-20T23:59:15.3967879Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:59:15.4010441Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4057257Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:59:15.4100122Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4147086Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:59:15.4189639Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4255137Z Entering 'third_party/pocketfft' 2024-08-20T23:59:15.4298930Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4344931Z Entering 'third_party/protobuf' 2024-08-20T23:59:15.4393089Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4438266Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:59:15.4481814Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4525391Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:59:15.4568880Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4615221Z Entering 'third_party/psimd' 2024-08-20T23:59:15.4659556Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4703021Z Entering 'third_party/pthreadpool' 2024-08-20T23:59:15.4747387Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4792870Z Entering 'third_party/pybind11' 2024-08-20T23:59:15.4841488Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4886769Z Entering 'third_party/python-peachpy' 2024-08-20T23:59:15.4929589Z http.https://github.com/.extraheader 2024-08-20T23:59:15.4972955Z Entering 'third_party/sleef' 2024-08-20T23:59:15.5016741Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5060386Z Entering 'third_party/tensorpipe' 2024-08-20T23:59:15.5103645Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5150322Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:59:15.5192244Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5235202Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:59:15.5277238Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5319226Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:59:15.5361464Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5404303Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:59:15.5446587Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5487697Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:59:15.5532410Z http.https://github.com/.extraheader 2024-08-20T23:59:15.5691200Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:59:15.5713722Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:59:15.5722209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:59:15.5722735Z ##[endgroup] 2024-08-20T23:59:22.4220599Z Cleaning up orphan processes