2024-12-18T00:16:23.7888267Z Current runner version: '2.321.0' 2024-12-18T00:16:23.7895094Z Runner name: 'i-079e9e3a7e7495371' 2024-12-18T00:16:23.7896089Z Runner group name: 'Default' 2024-12-18T00:16:23.7897276Z Machine name: 'ip-10-0-71-197' 2024-12-18T00:16:23.7902426Z ##[group]GITHUB_TOKEN Permissions 2024-12-18T00:16:23.7905063Z Actions: read 2024-12-18T00:16:23.7905845Z Attestations: read 2024-12-18T00:16:23.7906633Z Checks: read 2024-12-18T00:16:23.7907339Z Contents: read 2024-12-18T00:16:23.7908098Z Deployments: read 2024-12-18T00:16:23.7908899Z Discussions: read 2024-12-18T00:16:23.7909569Z Issues: read 2024-12-18T00:16:23.7910240Z Metadata: read 2024-12-18T00:16:23.7911021Z Packages: read 2024-12-18T00:16:23.7911716Z Pages: read 2024-12-18T00:16:23.7912352Z PullRequests: read 2024-12-18T00:16:23.7913187Z RepositoryProjects: read 2024-12-18T00:16:23.7913968Z SecurityEvents: read 2024-12-18T00:16:23.7914754Z Statuses: read 2024-12-18T00:16:23.7915512Z ##[endgroup] 2024-12-18T00:16:23.7918765Z Secret source: Actions 2024-12-18T00:16:23.7919851Z Prepare workflow directory 2024-12-18T00:16:23.8884889Z Prepare all required actions 2024-12-18T00:16:23.8926365Z Getting action download info 2024-12-18T00:16:24.1204805Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-18T00:16:25.7374478Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T00:16:38.4150119Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-12-18T00:16:38.6321485Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-18T00:16:38.9260698Z Getting action download info 2024-12-18T00:16:39.0665649Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-18T00:16:39.3403831Z Getting action download info 2024-12-18T00:16:39.5207152Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-18T00:16:39.7597909Z Getting action download info 2024-12-18T00:16:39.8635486Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-12-18T00:16:40.0160849Z Getting action download info 2024-12-18T00:16:40.2039927Z Download action repository 'pytorch/test-infra@main' (SHA:a07505a74641a4ff5123d635defac481ef28ef1e) 2024-12-18T00:16:41.5805698Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T00:16:41.5807700Z ##[group] Inputs 2024-12-18T00:16:41.5808100Z build-environment: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:16:41.5809655Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-12-18T00:16:41.5811618Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:16:41.5812461Z sync-tag: 2024-12-18T00:16:41.5813178Z timeout-minutes: 240 2024-12-18T00:16:41.5813447Z use-gha: 2024-12-18T00:16:41.5813665Z dashboard-tag: 2024-12-18T00:16:41.5813915Z s3-bucket: gha-artifacts 2024-12-18T00:16:41.5814189Z aws-role-to-assume: 2024-12-18T00:16:41.5814720Z disable-monitor: false 2024-12-18T00:16:41.5815000Z ##[endgroup] 2024-12-18T00:16:41.5815534Z Complete job name: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:16:41.6548920Z A job started hook has been configured by the self-hosted runner administrator 2024-12-18T00:16:41.6654241Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-12-18T00:16:41.6665566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:16:41.6666195Z ##[endgroup] 2024-12-18T00:16:43.5651318Z Runner Type: linux.g5.12xlarge.nvidia.gpu 2024-12-18T00:16:43.5651822Z Instance Type: g5.12xlarge 2024-12-18T00:16:43.5652101Z AMI Name: unknown 2024-12-18T00:16:43.5701071Z AMI ID: ami-0fff1b9a61dec8a5f 2024-12-18T00:16:49.5165424Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@release/2.6 2024-12-18T00:16:49.5165917Z with: 2024-12-18T00:16:49.5166539Z github-secret: *** 2024-12-18T00:16:49.5167252Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-12-18T00:16:49.5167995Z activate-with-label: false 2024-12-18T00:16:49.5168288Z label: with-ssh 2024-12-18T00:16:49.5168556Z remove-existing-keys: true 2024-12-18T00:16:49.5168873Z fail-silently: true 2024-12-18T00:16:49.5169126Z env: 2024-12-18T00:16:49.5169360Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:16:49.5169646Z ##[endgroup] 2024-12-18T00:16:49.6390132Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-12-18T00:16:49.6391446Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2024-12-18T00:16:49.6541047Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-18T00:16:49.6541512Z with: 2024-12-18T00:16:49.6541744Z no-sudo: true 2024-12-18T00:16:49.6542000Z submodules: recursive 2024-12-18T00:16:49.6542289Z fetch-depth: 0 2024-12-18T00:16:49.6542536Z env: 2024-12-18T00:16:49.6542765Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:16:49.6543047Z ##[endgroup] 2024-12-18T00:16:49.6618430Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:16:49.6619371Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:16:49.6630189Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:16:49.6630579Z env: 2024-12-18T00:16:49.6630820Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:16:49.6631097Z ##[endgroup] 2024-12-18T00:16:49.6732481Z ##[group]Run retry () { 2024-12-18T00:16:49.6732788Z retry () { 2024-12-18T00:16:49.6733164Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-18T00:16:49.6733594Z } 2024-12-18T00:16:49.6733852Z echo "${GITHUB_WORKSPACE}" 2024-12-18T00:16:49.6734192Z if [ -z "${NO_SUDO}" ]; then 2024-12-18T00:16:49.6734565Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T00:16:49.6734907Z else 2024-12-18T00:16:49.6735189Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T00:16:49.6735523Z fi 2024-12-18T00:16:49.6735789Z mkdir "${GITHUB_WORKSPACE}" 2024-12-18T00:16:49.6744094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:16:49.6744480Z env: 2024-12-18T00:16:49.6744721Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:16:49.6744996Z NO_SUDO: true 2024-12-18T00:16:49.6745249Z ##[endgroup] 2024-12-18T00:16:49.6775622Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:16:54.2604798Z ##[group]Run malfet/checkout@silent-checkout 2024-12-18T00:16:54.2605143Z with: 2024-12-18T00:16:54.2605412Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:16:54.2605760Z fetch-depth: 0 2024-12-18T00:16:54.2606010Z submodules: recursive 2024-12-18T00:16:54.2606295Z quiet-checkout: true 2024-12-18T00:16:54.2606584Z repository: pytorch/pytorch 2024-12-18T00:16:54.2606983Z token: *** 2024-12-18T00:16:54.2607223Z ssh-strict: true 2024-12-18T00:16:54.2607500Z persist-credentials: true 2024-12-18T00:16:54.2607790Z clean: true 2024-12-18T00:16:54.2608303Z sparse-checkout-cone-mode: true 2024-12-18T00:16:54.2608614Z lfs: false 2024-12-18T00:16:54.2608869Z set-safe-directory: true 2024-12-18T00:16:54.2609141Z env: 2024-12-18T00:16:54.2609377Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:16:54.2609658Z ##[endgroup] 2024-12-18T00:16:54.3513232Z Syncing repository: pytorch/pytorch 2024-12-18T00:16:54.3514389Z ##[group]Getting Git version info 2024-12-18T00:16:54.3514874Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-18T00:16:54.3515530Z [command]/usr/bin/git version 2024-12-18T00:16:54.3515853Z git version 2.40.1 2024-12-18T00:16:54.3536895Z ##[endgroup] 2024-12-18T00:16:54.3550231Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d50b1fae-9186-49fd-89ea-acd034c3277f' before making global git config changes 2024-12-18T00:16:54.3551304Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T00:16:54.3554116Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:16:54.3588240Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-12-18T00:16:54.3591378Z ##[group]Initializing the repository 2024-12-18T00:16:54.3593810Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:16:54.3626857Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-18T00:16:54.3628153Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-18T00:16:54.3629261Z hint: of your new repositories, which will suppress this warning, call: 2024-12-18T00:16:54.3630072Z hint: 2024-12-18T00:16:54.3630698Z hint: git config --global init.defaultBranch 2024-12-18T00:16:54.3631403Z hint: 2024-12-18T00:16:54.3632082Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-18T00:16:54.3632774Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-18T00:16:54.3633231Z hint: 2024-12-18T00:16:54.3633467Z hint: git branch -m 2024-12-18T00:16:54.3633988Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-18T00:16:54.3638966Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-18T00:16:54.3673390Z ##[endgroup] 2024-12-18T00:16:54.3673873Z ##[group]Disabling automatic garbage collection 2024-12-18T00:16:54.3676256Z [command]/usr/bin/git config --local gc.auto 0 2024-12-18T00:16:54.3704617Z ##[endgroup] 2024-12-18T00:16:54.3705049Z ##[group]Setting up auth 2024-12-18T00:16:54.3709018Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T00:16:54.3739081Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T00:16:54.4106101Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T00:16:54.4134009Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T00:16:54.4502839Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T00:16:54.4546919Z ##[endgroup] 2024-12-18T00:16:54.4547361Z ##[group]Fetching the repository 2024-12-18T00:16:54.4552436Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-18T00:16:57.8831840Z remote: Enumerating objects: 1056564 2024-12-18T00:16:57.8832253Z remote: Enumerating objects: 1057269, done. 2024-12-18T00:16:57.8832762Z remote: Counting objects: 0% (1/705) 2024-12-18T00:16:57.8833978Z remote: Counting objects: 1% (8/705) 2024-12-18T00:16:57.8834887Z remote: Counting objects: 2% (15/705) 2024-12-18T00:16:57.8835264Z remote: Counting objects: 3% (22/705) 2024-12-18T00:16:57.8835623Z remote: Counting objects: 4% (29/705) 2024-12-18T00:16:57.8835989Z remote: Counting objects: 5% (36/705) 2024-12-18T00:16:57.8836367Z remote: Counting objects: 6% (43/705) 2024-12-18T00:16:57.8836740Z remote: Counting objects: 7% (50/705) 2024-12-18T00:16:57.8837112Z remote: Counting objects: 8% (57/705) 2024-12-18T00:16:57.8837471Z remote: Counting objects: 9% (64/705) 2024-12-18T00:16:57.8837827Z remote: Counting objects: 10% (71/705) 2024-12-18T00:16:57.8838185Z remote: Counting objects: 11% (78/705) 2024-12-18T00:16:57.8838536Z remote: Counting objects: 12% (85/705) 2024-12-18T00:16:57.8838916Z remote: Counting objects: 13% (92/705) 2024-12-18T00:16:57.8839281Z remote: Counting objects: 14% (99/705) 2024-12-18T00:16:57.8839642Z remote: Counting objects: 15% (106/705) 2024-12-18T00:16:57.8840003Z remote: Counting objects: 16% (113/705) 2024-12-18T00:16:57.8840363Z remote: Counting objects: 17% (120/705) 2024-12-18T00:16:57.8840729Z remote: Counting objects: 18% (127/705) 2024-12-18T00:16:57.8841078Z remote: Counting objects: 19% (134/705) 2024-12-18T00:16:57.8841651Z remote: Counting objects: 20% (141/705) 2024-12-18T00:16:57.8842017Z remote: Counting objects: 21% (149/705) 2024-12-18T00:16:57.8842375Z remote: Counting objects: 22% (156/705) 2024-12-18T00:16:57.8842746Z remote: Counting objects: 23% (163/705) 2024-12-18T00:16:57.8843267Z remote: Counting objects: 24% (170/705) 2024-12-18T00:16:57.8843629Z remote: Counting objects: 25% (177/705) 2024-12-18T00:16:57.8843986Z remote: Counting objects: 26% (184/705) 2024-12-18T00:16:57.8844444Z remote: Counting objects: 27% (191/705) 2024-12-18T00:16:57.8844818Z remote: Counting objects: 28% (198/705) 2024-12-18T00:16:57.8845183Z remote: Counting objects: 29% (205/705) 2024-12-18T00:16:57.8845542Z remote: Counting objects: 30% (212/705) 2024-12-18T00:16:57.8845896Z remote: Counting objects: 31% (219/705) 2024-12-18T00:16:57.8846252Z remote: Counting objects: 32% (226/705) 2024-12-18T00:16:57.8846684Z remote: Counting objects: 33% (233/705) 2024-12-18T00:16:57.8847043Z remote: Counting objects: 34% (240/705) 2024-12-18T00:16:57.8847403Z remote: Counting objects: 35% (247/705) 2024-12-18T00:16:57.8847760Z remote: Counting objects: 36% (254/705) 2024-12-18T00:16:57.8848122Z remote: Counting objects: 37% (261/705) 2024-12-18T00:16:57.8848482Z remote: Counting objects: 38% (268/705) 2024-12-18T00:16:57.8848851Z remote: Counting objects: 39% (275/705) 2024-12-18T00:16:57.8849212Z remote: Counting objects: 40% (282/705) 2024-12-18T00:16:57.8849565Z remote: Counting objects: 41% (290/705) 2024-12-18T00:16:57.8849933Z remote: Counting objects: 42% (297/705) 2024-12-18T00:16:57.8850288Z remote: Counting objects: 43% (304/705) 2024-12-18T00:16:57.8850646Z remote: Counting objects: 44% (311/705) 2024-12-18T00:16:57.8851012Z remote: Counting objects: 45% (318/705) 2024-12-18T00:16:57.8851377Z remote: Counting objects: 46% (325/705) 2024-12-18T00:16:57.8851737Z remote: Counting objects: 47% (332/705) 2024-12-18T00:16:57.8852092Z remote: Counting objects: 48% (339/705) 2024-12-18T00:16:57.8852451Z remote: Counting objects: 49% (346/705) 2024-12-18T00:16:57.8852810Z remote: Counting objects: 50% (353/705) 2024-12-18T00:16:57.8853162Z remote: Counting objects: 51% (360/705) 2024-12-18T00:16:57.8853516Z remote: Counting objects: 52% (367/705) 2024-12-18T00:16:57.8853880Z remote: Counting objects: 53% (374/705) 2024-12-18T00:16:57.8854239Z remote: Counting objects: 54% (381/705) 2024-12-18T00:16:57.8854708Z remote: Counting objects: 55% (388/705) 2024-12-18T00:16:57.8855071Z remote: Counting objects: 56% (395/705) 2024-12-18T00:16:57.8855430Z remote: Counting objects: 57% (402/705) 2024-12-18T00:16:57.8855787Z remote: Counting objects: 58% (409/705) 2024-12-18T00:16:57.8856151Z remote: Counting objects: 59% (416/705) 2024-12-18T00:16:57.8856515Z remote: Counting objects: 60% (423/705) 2024-12-18T00:16:57.8856871Z remote: Counting objects: 61% (431/705) 2024-12-18T00:16:57.8857235Z remote: Counting objects: 62% (438/705) 2024-12-18T00:16:57.8857627Z remote: Counting objects: 63% (445/705) 2024-12-18T00:16:57.8858094Z remote: Counting objects: 64% (452/705) 2024-12-18T00:16:57.8858458Z remote: Counting objects: 65% (459/705) 2024-12-18T00:16:57.8858820Z remote: Counting objects: 66% (466/705) 2024-12-18T00:16:57.8859175Z remote: Counting objects: 67% (473/705) 2024-12-18T00:16:57.8859540Z remote: Counting objects: 68% (480/705) 2024-12-18T00:16:57.8859893Z remote: Counting objects: 69% (487/705) 2024-12-18T00:16:57.8860255Z remote: Counting objects: 70% (494/705) 2024-12-18T00:16:57.8860611Z remote: Counting objects: 71% (501/705) 2024-12-18T00:16:57.8860971Z remote: Counting objects: 72% (508/705) 2024-12-18T00:16:57.8861442Z remote: Counting objects: 73% (515/705) 2024-12-18T00:16:57.8861805Z remote: Counting objects: 74% (522/705) 2024-12-18T00:16:57.8862165Z remote: Counting objects: 75% (529/705) 2024-12-18T00:16:57.8862524Z remote: Counting objects: 76% (536/705) 2024-12-18T00:16:57.8862887Z remote: Counting objects: 77% (543/705) 2024-12-18T00:16:57.8863258Z remote: Counting objects: 78% (550/705) 2024-12-18T00:16:57.8864015Z remote: Counting objects: 79% (557/705) 2024-12-18T00:16:57.8864374Z remote: Counting objects: 80% (564/705) 2024-12-18T00:16:57.8864737Z remote: Counting objects: 81% (572/705) 2024-12-18T00:16:57.8865094Z remote: Counting objects: 82% (579/705) 2024-12-18T00:16:57.8865454Z remote: Counting objects: 83% (586/705) 2024-12-18T00:16:57.8865822Z remote: Counting objects: 84% (593/705) 2024-12-18T00:16:57.8866187Z remote: Counting objects: 85% (600/705) 2024-12-18T00:16:57.8866550Z remote: Counting objects: 86% (607/705) 2024-12-18T00:16:57.8866909Z remote: Counting objects: 87% (614/705) 2024-12-18T00:16:57.8867263Z remote: Counting objects: 88% (621/705) 2024-12-18T00:16:57.8867631Z remote: Counting objects: 89% (628/705) 2024-12-18T00:16:57.8867992Z remote: Counting objects: 90% (635/705) 2024-12-18T00:16:57.8868355Z remote: Counting objects: 91% (642/705) 2024-12-18T00:16:57.8868706Z remote: Counting objects: 92% (649/705) 2024-12-18T00:16:57.8869062Z remote: Counting objects: 93% (656/705) 2024-12-18T00:16:57.8869424Z remote: Counting objects: 94% (663/705) 2024-12-18T00:16:57.8869783Z remote: Counting objects: 95% (670/705) 2024-12-18T00:16:57.8870142Z remote: Counting objects: 96% (677/705) 2024-12-18T00:16:57.8870498Z remote: Counting objects: 97% (684/705) 2024-12-18T00:16:57.8870850Z remote: Counting objects: 98% (691/705) 2024-12-18T00:16:57.8871214Z remote: Counting objects: 99% (698/705) 2024-12-18T00:16:57.8871578Z remote: Counting objects: 100% (705/705) 2024-12-18T00:16:57.8871966Z remote: Counting objects: 100% (705/705), done. 2024-12-18T00:16:57.8951579Z remote: Compressing objects: 0% (1/304) 2024-12-18T00:16:57.9042260Z remote: Compressing objects: 1% (4/304) 2024-12-18T00:16:57.9092082Z remote: Compressing objects: 2% (7/304) 2024-12-18T00:16:57.9178959Z remote: Compressing objects: 3% (10/304) 2024-12-18T00:16:57.9350622Z remote: Compressing objects: 4% (13/304) 2024-12-18T00:16:57.9817424Z remote: Compressing objects: 5% (16/304) 2024-12-18T00:16:58.0310426Z remote: Compressing objects: 6% (19/304) 2024-12-18T00:16:58.1157896Z remote: Compressing objects: 7% (22/304) 2024-12-18T00:16:58.1627250Z remote: Compressing objects: 8% (25/304) 2024-12-18T00:16:58.2026170Z remote: Compressing objects: 9% (28/304) 2024-12-18T00:16:58.2228970Z remote: Compressing objects: 10% (31/304) 2024-12-18T00:16:58.2351276Z remote: Compressing objects: 11% (34/304) 2024-12-18T00:16:58.2360121Z remote: Compressing objects: 12% (37/304) 2024-12-18T00:16:58.2362531Z remote: Compressing objects: 13% (40/304) 2024-12-18T00:16:58.2366982Z remote: Compressing objects: 14% (43/304) 2024-12-18T00:16:58.2373216Z remote: Compressing objects: 15% (46/304) 2024-12-18T00:16:58.2377362Z remote: Compressing objects: 16% (49/304) 2024-12-18T00:16:58.2380628Z remote: Compressing objects: 17% (52/304) 2024-12-18T00:16:58.2382552Z remote: Compressing objects: 18% (55/304) 2024-12-18T00:16:58.2428986Z remote: Compressing objects: 19% (58/304) 2024-12-18T00:16:58.2429767Z remote: Compressing objects: 20% (61/304) 2024-12-18T00:16:58.2430524Z remote: Compressing objects: 21% (64/304) 2024-12-18T00:16:58.2431274Z remote: Compressing objects: 22% (67/304) 2024-12-18T00:16:58.2432353Z remote: Compressing objects: 23% (70/304) 2024-12-18T00:16:58.2433124Z remote: Compressing objects: 24% (73/304) 2024-12-18T00:16:58.2433715Z remote: Compressing objects: 25% (76/304) 2024-12-18T00:16:58.2434100Z remote: Compressing objects: 26% (80/304) 2024-12-18T00:16:58.2434469Z remote: Compressing objects: 27% (83/304) 2024-12-18T00:16:58.2434848Z remote: Compressing objects: 28% (86/304) 2024-12-18T00:16:58.2435392Z remote: Compressing objects: 29% (89/304) 2024-12-18T00:16:58.2437265Z remote: Compressing objects: 30% (92/304) 2024-12-18T00:16:58.2439575Z remote: Compressing objects: 31% (95/304) 2024-12-18T00:16:58.2445564Z remote: Compressing objects: 32% (98/304) 2024-12-18T00:16:58.2447825Z remote: Compressing objects: 33% (101/304) 2024-12-18T00:16:58.2453796Z remote: Compressing objects: 34% (104/304) 2024-12-18T00:16:58.2459264Z remote: Compressing objects: 35% (107/304) 2024-12-18T00:16:58.2461298Z remote: Compressing objects: 36% (110/304) 2024-12-18T00:16:58.2466693Z remote: Compressing objects: 37% (113/304) 2024-12-18T00:16:58.2467388Z remote: Compressing objects: 38% (116/304) 2024-12-18T00:16:58.2470493Z remote: Compressing objects: 39% (119/304) 2024-12-18T00:16:58.2473035Z remote: Compressing objects: 40% (122/304) 2024-12-18T00:16:58.2478474Z remote: Compressing objects: 41% (125/304) 2024-12-18T00:16:58.2480352Z remote: Compressing objects: 42% (128/304) 2024-12-18T00:16:58.2485203Z remote: Compressing objects: 43% (131/304) 2024-12-18T00:16:58.2486939Z remote: Compressing objects: 44% (134/304) 2024-12-18T00:16:58.2489586Z remote: Compressing objects: 45% (137/304) 2024-12-18T00:16:58.2495321Z remote: Compressing objects: 46% (140/304) 2024-12-18T00:16:58.2496445Z remote: Compressing objects: 47% (143/304) 2024-12-18T00:16:58.2497811Z remote: Compressing objects: 48% (146/304) 2024-12-18T00:16:58.2501569Z remote: Compressing objects: 49% (149/304) 2024-12-18T00:16:58.2504740Z remote: Compressing objects: 50% (152/304) 2024-12-18T00:16:58.2507029Z remote: Compressing objects: 51% (156/304) 2024-12-18T00:16:58.2509017Z remote: Compressing objects: 52% (159/304) 2024-12-18T00:16:58.2511984Z remote: Compressing objects: 53% (162/304) 2024-12-18T00:16:58.2513561Z remote: Compressing objects: 54% (165/304) 2024-12-18T00:16:58.2515944Z remote: Compressing objects: 55% (168/304) 2024-12-18T00:16:58.2517215Z remote: Compressing objects: 56% (171/304) 2024-12-18T00:16:58.2519999Z remote: Compressing objects: 57% (174/304) 2024-12-18T00:16:58.2521823Z remote: Compressing objects: 58% (177/304) 2024-12-18T00:16:58.2523012Z remote: Compressing objects: 59% (180/304) 2024-12-18T00:16:58.2525791Z remote: Compressing objects: 60% (183/304) 2024-12-18T00:16:58.2527323Z remote: Compressing objects: 61% (186/304) 2024-12-18T00:16:58.2528431Z remote: Compressing objects: 62% (189/304) 2024-12-18T00:16:58.2533019Z remote: Compressing objects: 63% (192/304) 2024-12-18T00:16:58.2533455Z remote: Compressing objects: 64% (195/304) 2024-12-18T00:16:58.2535752Z remote: Compressing objects: 65% (198/304) 2024-12-18T00:16:58.2540042Z remote: Compressing objects: 66% (201/304) 2024-12-18T00:16:58.2544902Z remote: Compressing objects: 67% (204/304) 2024-12-18T00:16:58.2546777Z remote: Compressing objects: 68% (207/304) 2024-12-18T00:16:58.2549263Z remote: Compressing objects: 69% (210/304) 2024-12-18T00:16:58.2550063Z remote: Compressing objects: 70% (213/304) 2024-12-18T00:16:58.2554145Z remote: Compressing objects: 71% (216/304) 2024-12-18T00:16:58.2557598Z remote: Compressing objects: 72% (219/304) 2024-12-18T00:16:58.2559617Z remote: Compressing objects: 73% (222/304) 2024-12-18T00:16:58.2561760Z remote: Compressing objects: 74% (225/304) 2024-12-18T00:16:58.2564714Z remote: Compressing objects: 75% (228/304) 2024-12-18T00:16:58.2565103Z remote: Compressing objects: 76% (232/304) 2024-12-18T00:16:58.2567242Z remote: Compressing objects: 77% (235/304) 2024-12-18T00:16:58.2568017Z remote: Compressing objects: 78% (238/304) 2024-12-18T00:16:58.2568399Z remote: Compressing objects: 79% (241/304) 2024-12-18T00:16:58.2568777Z remote: Compressing objects: 80% (244/304) 2024-12-18T00:16:58.2570889Z remote: Compressing objects: 81% (247/304) 2024-12-18T00:16:58.2573469Z remote: Compressing objects: 82% (250/304) 2024-12-18T00:16:58.2575745Z remote: Compressing objects: 83% (253/304) 2024-12-18T00:16:58.2577275Z remote: Compressing objects: 84% (256/304) 2024-12-18T00:16:58.2579399Z remote: Compressing objects: 85% (259/304) 2024-12-18T00:16:58.2580359Z remote: Compressing objects: 86% (262/304) 2024-12-18T00:16:58.2582125Z remote: Compressing objects: 87% (265/304) 2024-12-18T00:16:58.2583551Z remote: Compressing objects: 88% (268/304) 2024-12-18T00:16:58.2585208Z remote: Compressing objects: 89% (271/304) 2024-12-18T00:16:58.2585674Z remote: Compressing objects: 90% (274/304) 2024-12-18T00:16:58.2587225Z remote: Compressing objects: 91% (277/304) 2024-12-18T00:16:58.2588021Z remote: Compressing objects: 92% (280/304) 2024-12-18T00:16:58.2588596Z remote: Compressing objects: 93% (283/304) 2024-12-18T00:16:58.2590194Z remote: Compressing objects: 94% (286/304) 2024-12-18T00:16:58.2590842Z remote: Compressing objects: 95% (289/304) 2024-12-18T00:16:58.2591670Z remote: Compressing objects: 96% (292/304) 2024-12-18T00:16:58.2592522Z remote: Compressing objects: 97% (295/304) 2024-12-18T00:16:58.2593184Z remote: Compressing objects: 98% (298/304) 2024-12-18T00:16:58.2593997Z remote: Compressing objects: 99% (301/304) 2024-12-18T00:16:58.2594464Z remote: Compressing objects: 100% (304/304) 2024-12-18T00:16:58.2594885Z remote: Compressing objects: 100% (304/304), done. 2024-12-18T00:17:22.6477521Z remote: Total 1057269 (delta 564), reused 404 (delta 401), pack-reused 1056564 (from 3) 2024-12-18T00:17:44.0816971Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-18T00:17:44.0843567Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:17:44.0848797Z ##[endgroup] 2024-12-18T00:17:44.0849231Z ##[group]Determining the checkout info 2024-12-18T00:17:44.0849774Z ##[endgroup] 2024-12-18T00:17:44.0850233Z ##[group]Checking out the ref 2024-12-18T00:17:44.0853485Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:17:45.8643499Z ##[endgroup] 2024-12-18T00:17:45.8644412Z ##[group]Setting up auth for fetching submodules 2024-12-18T00:17:45.8646476Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T00:17:45.8709616Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-18T00:17:45.8739527Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-18T00:17:45.8772620Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-18T00:17:45.8803133Z ##[endgroup] 2024-12-18T00:17:45.8803555Z ##[group]Fetching submodules 2024-12-18T00:17:45.8805463Z [command]/usr/bin/git submodule sync --recursive 2024-12-18T00:17:45.9190737Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-18T00:17:45.9562576Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-18T00:17:45.9565874Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-18T00:17:45.9568831Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-18T00:17:45.9572068Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-18T00:17:45.9575374Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-18T00:17:45.9579119Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-18T00:17:45.9582285Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-18T00:17:45.9585918Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-18T00:17:45.9589647Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-18T00:17:45.9593288Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-18T00:17:45.9596950Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-18T00:17:45.9600908Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-18T00:17:45.9605074Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-18T00:17:45.9608881Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-18T00:17:45.9613016Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-18T00:17:45.9619064Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-18T00:17:45.9626619Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-18T00:17:45.9631205Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:17:45.9635462Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-18T00:17:45.9640192Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-18T00:17:45.9644746Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-18T00:17:45.9649375Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-18T00:17:45.9653961Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-18T00:17:45.9658725Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-18T00:17:45.9663728Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-18T00:17:45.9668966Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-18T00:17:45.9673751Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-18T00:17:45.9678891Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-18T00:17:45.9683960Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-18T00:17:45.9689484Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-18T00:17:45.9694534Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-18T00:17:45.9699670Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-18T00:17:45.9704819Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-18T00:17:45.9713580Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-18T00:17:45.9719117Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-18T00:17:45.9725014Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-18T00:17:45.9759104Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-18T00:17:46.2646444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-18T00:17:46.4867582Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-18T00:17:46.6802450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-18T00:17:46.9979243Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-18T00:17:47.3464045Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-18T00:17:49.3607555Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-18T00:17:59.4432086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-18T00:17:59.9407242Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-18T00:18:02.2266571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-18T00:18:02.7729471Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-18T00:18:03.4647485Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-18T00:18:04.8476541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-18T00:18:07.0567892Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-18T00:18:12.9656321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-18T00:18:14.5525470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-18T00:18:15.8522295Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-18T00:18:17.0514086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-18T00:18:17.5307557Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-18T00:18:17.9224736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-18T00:18:19.0265566Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-18T00:18:19.3879360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-18T00:18:19.6984415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-18T00:18:21.3738883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-18T00:18:22.2427081Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-18T00:18:22.6435730Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-18T00:18:29.4636597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-18T00:18:31.3859635Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-18T00:18:34.8162280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-18T00:18:35.1162175Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T00:18:44.2425185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T00:18:44.4498106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T00:18:44.6650501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T00:18:45.7751362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T00:18:46.1067112Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T00:18:46.7636155Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T00:18:47.2488828Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T00:18:47.2649319Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T00:18:47.2772893Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T00:18:47.3100375Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T00:18:47.3546195Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T00:18:47.4012875Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T00:18:48.3277910Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T00:18:48.3571702Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T00:18:48.6628315Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T00:18:48.7203815Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T00:18:48.8333712Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T00:18:48.8751147Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T00:18:49.5041430Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T00:18:49.7942363Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T00:18:49.9044994Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T00:18:49.9067044Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:18:49.9070698Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:18:49.9073481Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:18:49.9077047Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:18:49.9080895Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:18:49.9120228Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T00:18:50.8885998Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T00:18:51.5317837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T00:18:53.7481568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T00:18:54.7088428Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T00:18:55.1442027Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T00:18:55.2573547Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T00:18:55.7551406Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T00:18:55.8273541Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T00:18:55.8435096Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T00:18:56.0105099Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T00:18:56.0566765Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T00:18:56.1042664Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T00:18:56.1389289Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T00:18:56.1917544Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T00:18:56.2096582Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T00:18:56.2117262Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T00:18:56.2148437Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T00:19:10.4788125Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T00:19:10.5028561Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T00:19:10.6028243Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T00:19:10.6052550Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:10.6055720Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:10.6059322Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:10.6091358Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T00:19:11.3800070Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T00:19:12.5524373Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T00:19:13.6325042Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T00:19:13.6346085Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:13.6350136Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:13.6353953Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:13.6357572Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:13.6361330Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:13.6365543Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:13.6369514Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:13.6373300Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:13.6407923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T00:19:14.5867688Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T00:19:15.0594761Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T00:19:16.2046438Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T00:19:16.5113550Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T00:19:16.9820033Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T00:19:17.9583577Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T00:19:25.0239348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T00:19:25.6644173Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T00:19:25.6887369Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T00:19:25.7330705Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T00:19:25.7509015Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T00:19:25.7529637Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:25.7567043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T00:19:26.0847648Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T00:19:26.1092796Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T00:19:26.1582483Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T00:19:26.2770199Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T00:19:26.2978511Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T00:19:26.3437845Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T00:19:26.4114365Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T00:19:26.4602145Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T00:19:26.4968500Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T00:19:26.6313712Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T00:19:27.1356403Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T00:19:27.1396544Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:27.1476248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T00:19:28.3896187Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T00:19:28.4818727Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T00:19:28.4841491Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:28.4845062Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:28.4848397Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:28.4851830Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:28.4855526Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:28.4859002Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:28.4862747Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:28.4866631Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:28.4902740Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T00:19:28.9533460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T00:19:29.9288959Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T00:19:30.2688673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T00:19:37.2274346Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T00:19:37.5262646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T00:19:37.7540439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T00:19:38.0780587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T00:19:45.1294293Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T00:19:45.1786049Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T00:19:45.1989525Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T00:19:45.3343396Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T00:19:45.3519725Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T00:19:45.3714421Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T00:19:45.3927805Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T00:19:45.3947710Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:45.3951204Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:45.3984116Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T00:19:47.0755252Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T00:19:48.3308656Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T00:19:48.3856111Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T00:19:49.0441705Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T00:19:49.0596293Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T00:19:49.3923657Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T00:19:49.3952047Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:49.3954953Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:49.3986748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T00:19:49.9045734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T00:19:50.9223476Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T00:19:51.0067714Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T00:19:51.0197818Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T00:19:51.0364342Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T00:19:51.0819105Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T00:19:51.1179093Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T00:19:51.1692803Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T00:19:51.2043617Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T00:19:51.2070330Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:51.2073060Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:51.2076239Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:51.2079229Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:51.2111496Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T00:19:52.2007329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T00:19:52.4500189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T00:19:54.9148938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T00:19:56.0866051Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T00:19:56.1069793Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T00:19:56.1808012Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T00:19:56.2168062Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T00:19:56.2188006Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:56.2220134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T00:19:56.4415322Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T00:19:56.4465406Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T00:19:56.4842535Z Entering 'android/libs/fbjni' 2024-12-18T00:19:56.4894598Z Entering 'third_party/FP16' 2024-12-18T00:19:56.4945897Z Entering 'third_party/FXdiv' 2024-12-18T00:19:56.4998398Z Entering 'third_party/NNPACK' 2024-12-18T00:19:56.5061561Z Entering 'third_party/NVTX' 2024-12-18T00:19:56.5115451Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:56.5169026Z Entering 'third_party/XNNPACK' 2024-12-18T00:19:56.5238096Z Entering 'third_party/benchmark' 2024-12-18T00:19:56.5291484Z Entering 'third_party/composable_kernel' 2024-12-18T00:19:56.5351080Z Entering 'third_party/cpp-httplib' 2024-12-18T00:19:56.5402521Z Entering 'third_party/cpuinfo' 2024-12-18T00:19:56.5458127Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:19:56.5515277Z Entering 'third_party/cutlass' 2024-12-18T00:19:56.5575957Z Entering 'third_party/eigen' 2024-12-18T00:19:56.5630616Z Entering 'third_party/fbgemm' 2024-12-18T00:19:56.5683428Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:19:56.5733661Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:19:56.5784995Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:19:56.5842476Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:19:56.5894127Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:19:56.5949185Z Entering 'third_party/flatbuffers' 2024-12-18T00:19:56.6005329Z Entering 'third_party/fmt' 2024-12-18T00:19:56.6057696Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:56.6111015Z Entering 'third_party/gloo' 2024-12-18T00:19:56.6165171Z Entering 'third_party/googletest' 2024-12-18T00:19:56.6218255Z Entering 'third_party/ideep' 2024-12-18T00:19:56.6273009Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:19:56.6332156Z Entering 'third_party/ittapi' 2024-12-18T00:19:56.6384869Z Entering 'third_party/kineto' 2024-12-18T00:19:56.6441207Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:56.6492051Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:56.6547152Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:56.6606460Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:56.6660821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:56.6711419Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:56.6767894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:56.6819992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:56.6873654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:56.6926981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:56.6983283Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:56.7034766Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:56.7089368Z Entering 'third_party/mimalloc' 2024-12-18T00:19:56.7142562Z Entering 'third_party/nccl/nccl' 2024-12-18T00:19:56.7197870Z Entering 'third_party/nlohmann' 2024-12-18T00:19:56.7255291Z Entering 'third_party/onnx' 2024-12-18T00:19:56.7324669Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:56.7385310Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:19:56.7440056Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:56.7490836Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:56.7543349Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:56.7594041Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:56.7649194Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:56.7704594Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:56.7754672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:56.7810675Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:56.7864389Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:56.7919958Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:56.8000195Z Entering 'third_party/pocketfft' 2024-12-18T00:19:56.8053341Z Entering 'third_party/protobuf' 2024-12-18T00:19:56.8110855Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:56.8162721Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:56.8220493Z Entering 'third_party/psimd' 2024-12-18T00:19:56.8276670Z Entering 'third_party/pthreadpool' 2024-12-18T00:19:56.8333647Z Entering 'third_party/pybind11' 2024-12-18T00:19:56.8386954Z Entering 'third_party/python-peachpy' 2024-12-18T00:19:56.8445130Z Entering 'third_party/sleef' 2024-12-18T00:19:56.8498829Z Entering 'third_party/tensorpipe' 2024-12-18T00:19:56.8552852Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:56.8605549Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:56.8656874Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:56.8709717Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:56.8761384Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:56.8840685Z ##[endgroup] 2024-12-18T00:19:56.8841348Z ##[group]Persisting credentials for submodules 2024-12-18T00:19:56.8848755Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T00:19:56.9224183Z Entering 'android/libs/fbjni' 2024-12-18T00:19:56.9294669Z Entering 'third_party/FP16' 2024-12-18T00:19:56.9363897Z Entering 'third_party/FXdiv' 2024-12-18T00:19:56.9443052Z Entering 'third_party/NNPACK' 2024-12-18T00:19:56.9515322Z Entering 'third_party/NVTX' 2024-12-18T00:19:56.9584537Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:56.9664529Z Entering 'third_party/XNNPACK' 2024-12-18T00:19:56.9749150Z Entering 'third_party/benchmark' 2024-12-18T00:19:56.9825320Z Entering 'third_party/composable_kernel' 2024-12-18T00:19:56.9902169Z Entering 'third_party/cpp-httplib' 2024-12-18T00:19:56.9974984Z Entering 'third_party/cpuinfo' 2024-12-18T00:19:57.0047231Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:19:57.0121759Z Entering 'third_party/cutlass' 2024-12-18T00:19:57.0201541Z Entering 'third_party/eigen' 2024-12-18T00:19:57.0280077Z Entering 'third_party/fbgemm' 2024-12-18T00:19:57.0349045Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:19:57.0425408Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:19:57.0493836Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:19:57.0568538Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:19:57.0640502Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:19:57.0711326Z Entering 'third_party/flatbuffers' 2024-12-18T00:19:57.0787027Z Entering 'third_party/fmt' 2024-12-18T00:19:57.0856576Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:57.0926360Z Entering 'third_party/gloo' 2024-12-18T00:19:57.0996633Z Entering 'third_party/googletest' 2024-12-18T00:19:57.1065205Z Entering 'third_party/ideep' 2024-12-18T00:19:57.1132302Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:19:57.1209371Z Entering 'third_party/ittapi' 2024-12-18T00:19:57.1277569Z Entering 'third_party/kineto' 2024-12-18T00:19:57.1350748Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:57.1417857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:57.1494307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:57.1563853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:57.1636485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:57.1703061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:57.1780677Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:57.1849776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:57.1920661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:57.1991909Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:57.2068848Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:57.2142601Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:57.2218559Z Entering 'third_party/mimalloc' 2024-12-18T00:19:57.2288394Z Entering 'third_party/nccl/nccl' 2024-12-18T00:19:57.2357135Z Entering 'third_party/nlohmann' 2024-12-18T00:19:57.2434075Z Entering 'third_party/onnx' 2024-12-18T00:19:57.2517735Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:57.2596493Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:19:57.2667022Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:57.2739913Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:57.2809356Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:57.2883013Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:57.2952869Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:57.3023940Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:57.3093531Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:57.3160595Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:57.3234854Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:57.3306408Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:57.3397462Z Entering 'third_party/pocketfft' 2024-12-18T00:19:57.3473878Z Entering 'third_party/protobuf' 2024-12-18T00:19:57.3547257Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:57.3621339Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:57.3694808Z Entering 'third_party/psimd' 2024-12-18T00:19:57.3763858Z Entering 'third_party/pthreadpool' 2024-12-18T00:19:57.3833402Z Entering 'third_party/pybind11' 2024-12-18T00:19:57.3905569Z Entering 'third_party/python-peachpy' 2024-12-18T00:19:57.3974954Z Entering 'third_party/sleef' 2024-12-18T00:19:57.4043931Z Entering 'third_party/tensorpipe' 2024-12-18T00:19:57.4112356Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:57.4184410Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:57.4259691Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:57.4327717Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:57.4398595Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:57.4497460Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T00:19:57.4868606Z Entering 'android/libs/fbjni' 2024-12-18T00:19:57.4931537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T00:19:57.4955678Z Entering 'third_party/FP16' 2024-12-18T00:19:57.5021925Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T00:19:57.5045216Z Entering 'third_party/FXdiv' 2024-12-18T00:19:57.5107676Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T00:19:57.5132953Z Entering 'third_party/NNPACK' 2024-12-18T00:19:57.5196947Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T00:19:57.5223007Z Entering 'third_party/NVTX' 2024-12-18T00:19:57.5285859Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T00:19:57.5309866Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:57.5377671Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T00:19:57.5400724Z Entering 'third_party/XNNPACK' 2024-12-18T00:19:57.5467882Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T00:19:57.5504901Z Entering 'third_party/benchmark' 2024-12-18T00:19:57.5569429Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:19:57.5591941Z Entering 'third_party/composable_kernel' 2024-12-18T00:19:57.5653435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T00:19:57.5682064Z Entering 'third_party/cpp-httplib' 2024-12-18T00:19:57.5744502Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T00:19:57.5766948Z Entering 'third_party/cpuinfo' 2024-12-18T00:19:57.5830809Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:19:57.5854608Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:19:57.5918474Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T00:19:57.5943195Z Entering 'third_party/cutlass' 2024-12-18T00:19:57.6007455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:19:57.6037492Z Entering 'third_party/eigen' 2024-12-18T00:19:57.6099883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T00:19:57.6126365Z Entering 'third_party/fbgemm' 2024-12-18T00:19:57.6198348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T00:19:57.6223271Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:19:57.6286122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T00:19:57.6307173Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:19:57.6370196Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T00:19:57.6392402Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:19:57.6454321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T00:19:57.6482822Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:19:57.6548060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:57.6571930Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:19:57.6634875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T00:19:57.6659584Z Entering 'third_party/flatbuffers' 2024-12-18T00:19:57.6721244Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T00:19:57.6746296Z Entering 'third_party/fmt' 2024-12-18T00:19:57.6811291Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:19:57.6833438Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:57.6899494Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T00:19:57.6923991Z Entering 'third_party/gloo' 2024-12-18T00:19:57.6985551Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T00:19:57.7008052Z Entering 'third_party/googletest' 2024-12-18T00:19:57.7069341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:57.7091903Z Entering 'third_party/ideep' 2024-12-18T00:19:57.7155598Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T00:19:57.7176776Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:19:57.7241340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T00:19:57.7271013Z Entering 'third_party/ittapi' 2024-12-18T00:19:57.7333558Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T00:19:57.7355791Z Entering 'third_party/kineto' 2024-12-18T00:19:57.7419945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T00:19:57.7441666Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:57.7505872Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T00:19:57.7527043Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:57.7592580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T00:19:57.7616441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:57.7680493Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T00:19:57.7703508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:57.7768375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T00:19:57.7790266Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:57.7857905Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T00:19:57.7877926Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:57.7947444Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T00:19:57.7972347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:57.8035727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T00:19:57.8057597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:57.8126808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:57.8149005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:57.8211231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T00:19:57.8234425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:57.8302891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T00:19:57.8327814Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:57.8390454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T00:19:57.8411846Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:57.8475973Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T00:19:57.8500024Z Entering 'third_party/mimalloc' 2024-12-18T00:19:57.8564690Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T00:19:57.8587585Z Entering 'third_party/nccl/nccl' 2024-12-18T00:19:57.8654730Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T00:19:57.8680261Z Entering 'third_party/nlohmann' 2024-12-18T00:19:57.8741871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T00:19:57.8766208Z Entering 'third_party/onnx' 2024-12-18T00:19:57.8829929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T00:19:57.8869361Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:57.8931335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:19:57.8958290Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:19:57.9022970Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T00:19:57.9046346Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:57.9109392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:19:57.9131415Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:57.9195045Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:57.9217898Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:57.9282511Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T00:19:57.9304134Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:57.9367246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T00:19:57.9390671Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:57.9457137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T00:19:57.9481408Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:57.9547814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T00:19:57.9572191Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:57.9635041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T00:19:57.9654316Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:57.9717166Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T00:19:57.9741515Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:57.9803995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T00:19:57.9829431Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:57.9891821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T00:19:57.9934318Z Entering 'third_party/pocketfft' 2024-12-18T00:19:57.9997876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T00:19:58.0019807Z Entering 'third_party/protobuf' 2024-12-18T00:19:58.0087992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T00:19:58.0111446Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:58.0178370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T00:19:58.0198353Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:58.0262492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:58.0288102Z Entering 'third_party/psimd' 2024-12-18T00:19:58.0352361Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T00:19:58.0375556Z Entering 'third_party/pthreadpool' 2024-12-18T00:19:58.0439016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T00:19:58.0463653Z Entering 'third_party/pybind11' 2024-12-18T00:19:58.0526282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:19:58.0548809Z Entering 'third_party/python-peachpy' 2024-12-18T00:19:58.0615309Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T00:19:58.0637000Z Entering 'third_party/sleef' 2024-12-18T00:19:58.0702146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T00:19:58.0724820Z Entering 'third_party/tensorpipe' 2024-12-18T00:19:58.0788557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T00:19:58.0809397Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:58.0872308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T00:19:58.0896263Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:58.0957892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T00:19:58.0980546Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:58.1044645Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T00:19:58.1067487Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:58.1129977Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T00:19:58.1149214Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:58.1213250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T00:19:58.2291146Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T00:19:58.2670118Z Entering 'android/libs/fbjni' 2024-12-18T00:19:58.2726555Z Entering 'third_party/FP16' 2024-12-18T00:19:58.2781601Z Entering 'third_party/FXdiv' 2024-12-18T00:19:58.2835556Z Entering 'third_party/NNPACK' 2024-12-18T00:19:58.2889466Z Entering 'third_party/NVTX' 2024-12-18T00:19:58.2946015Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:58.2999597Z Entering 'third_party/XNNPACK' 2024-12-18T00:19:58.3072450Z Entering 'third_party/benchmark' 2024-12-18T00:19:58.3127697Z Entering 'third_party/composable_kernel' 2024-12-18T00:19:58.3189257Z Entering 'third_party/cpp-httplib' 2024-12-18T00:19:58.3242380Z Entering 'third_party/cpuinfo' 2024-12-18T00:19:58.3298519Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:19:58.3351928Z Entering 'third_party/cutlass' 2024-12-18T00:19:58.3412489Z Entering 'third_party/eigen' 2024-12-18T00:19:58.3470263Z Entering 'third_party/fbgemm' 2024-12-18T00:19:58.3524454Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:19:58.3578659Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:19:58.3631096Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:19:58.3690520Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:19:58.3744507Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:19:58.3801830Z Entering 'third_party/flatbuffers' 2024-12-18T00:19:58.3860531Z Entering 'third_party/fmt' 2024-12-18T00:19:58.3915307Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:58.3972240Z Entering 'third_party/gloo' 2024-12-18T00:19:58.4027632Z Entering 'third_party/googletest' 2024-12-18T00:19:58.4080818Z Entering 'third_party/ideep' 2024-12-18T00:19:58.4131607Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:19:58.4192507Z Entering 'third_party/ittapi' 2024-12-18T00:19:58.4248778Z Entering 'third_party/kineto' 2024-12-18T00:19:58.4301384Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:58.4353195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:58.4410863Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:58.4463394Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:58.4515981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:58.4574474Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:58.4632554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:58.4686665Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:58.4743700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:58.4797505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:58.4857633Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:58.4909878Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:58.4970275Z Entering 'third_party/mimalloc' 2024-12-18T00:19:58.5023916Z Entering 'third_party/nccl/nccl' 2024-12-18T00:19:58.5080329Z Entering 'third_party/nlohmann' 2024-12-18T00:19:58.5137309Z Entering 'third_party/onnx' 2024-12-18T00:19:58.5214415Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:58.5272921Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:19:58.5327200Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:58.5383078Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:58.5434538Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:58.5487062Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:58.5543148Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:58.5595620Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:58.5651879Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:58.5703458Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:58.5758105Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:58.5818204Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:58.5892055Z Entering 'third_party/pocketfft' 2024-12-18T00:19:58.5946090Z Entering 'third_party/protobuf' 2024-12-18T00:19:58.6003795Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:58.6056156Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:58.6113786Z Entering 'third_party/psimd' 2024-12-18T00:19:58.6173271Z Entering 'third_party/pthreadpool' 2024-12-18T00:19:58.6227286Z Entering 'third_party/pybind11' 2024-12-18T00:19:58.6285371Z Entering 'third_party/python-peachpy' 2024-12-18T00:19:58.6339010Z Entering 'third_party/sleef' 2024-12-18T00:19:58.6398226Z Entering 'third_party/tensorpipe' 2024-12-18T00:19:58.6452293Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:58.6505622Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:58.6561511Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:58.6615552Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:58.6664999Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:58.6742305Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T00:19:58.7112783Z Entering 'android/libs/fbjni' 2024-12-18T00:19:58.7168087Z Entering 'third_party/FP16' 2024-12-18T00:19:58.7221492Z Entering 'third_party/FXdiv' 2024-12-18T00:19:58.7276069Z Entering 'third_party/NNPACK' 2024-12-18T00:19:58.7334170Z Entering 'third_party/NVTX' 2024-12-18T00:19:58.7389177Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T00:19:58.7442890Z Entering 'third_party/XNNPACK' 2024-12-18T00:19:58.7511772Z Entering 'third_party/benchmark' 2024-12-18T00:19:58.7572750Z Entering 'third_party/composable_kernel' 2024-12-18T00:19:58.7633175Z Entering 'third_party/cpp-httplib' 2024-12-18T00:19:58.7687643Z Entering 'third_party/cpuinfo' 2024-12-18T00:19:58.7745380Z Entering 'third_party/cudnn_frontend' 2024-12-18T00:19:58.7798656Z Entering 'third_party/cutlass' 2024-12-18T00:19:58.7862377Z Entering 'third_party/eigen' 2024-12-18T00:19:58.7920278Z Entering 'third_party/fbgemm' 2024-12-18T00:19:58.7974116Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T00:19:58.8025912Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T00:19:58.8077998Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T00:19:58.8136455Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T00:19:58.8187944Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T00:19:58.8245880Z Entering 'third_party/flatbuffers' 2024-12-18T00:19:58.8303589Z Entering 'third_party/fmt' 2024-12-18T00:19:58.8356101Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T00:19:58.8410039Z Entering 'third_party/gloo' 2024-12-18T00:19:58.8463284Z Entering 'third_party/googletest' 2024-12-18T00:19:58.8517891Z Entering 'third_party/ideep' 2024-12-18T00:19:58.8572734Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T00:19:58.8636274Z Entering 'third_party/ittapi' 2024-12-18T00:19:58.8690583Z Entering 'third_party/kineto' 2024-12-18T00:19:58.8743765Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T00:19:58.8798403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T00:19:58.8855584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T00:19:58.8909191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T00:19:58.8967253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T00:19:58.9017211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T00:19:58.9073442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T00:19:58.9132154Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T00:19:58.9186956Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T00:19:58.9249148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T00:19:58.9307653Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T00:19:58.9359068Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T00:19:58.9418096Z Entering 'third_party/mimalloc' 2024-12-18T00:19:58.9473542Z Entering 'third_party/nccl/nccl' 2024-12-18T00:19:58.9527354Z Entering 'third_party/nlohmann' 2024-12-18T00:19:58.9587141Z Entering 'third_party/onnx' 2024-12-18T00:19:58.9655188Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T00:19:58.9715122Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T00:19:58.9775412Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T00:19:58.9828049Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T00:19:58.9885412Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T00:19:58.9937521Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T00:19:58.9995826Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T00:19:59.0047495Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T00:19:59.0099792Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T00:19:59.0151396Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T00:19:59.0205507Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T00:19:59.0267010Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T00:19:59.0340851Z Entering 'third_party/pocketfft' 2024-12-18T00:19:59.0398612Z Entering 'third_party/protobuf' 2024-12-18T00:19:59.0457609Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T00:19:59.0512144Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T00:19:59.0571111Z Entering 'third_party/psimd' 2024-12-18T00:19:59.0625679Z Entering 'third_party/pthreadpool' 2024-12-18T00:19:59.0685787Z Entering 'third_party/pybind11' 2024-12-18T00:19:59.0743093Z Entering 'third_party/python-peachpy' 2024-12-18T00:19:59.0797343Z Entering 'third_party/sleef' 2024-12-18T00:19:59.0852775Z Entering 'third_party/tensorpipe' 2024-12-18T00:19:59.0907144Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T00:19:59.0960297Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T00:19:59.1012730Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T00:19:59.1064557Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T00:19:59.1116872Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T00:19:59.1193202Z ##[endgroup] 2024-12-18T00:19:59.1238094Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T00:19:59.1264225Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T00:19:59.1431755Z Prepare all required actions 2024-12-18T00:19:59.1432251Z Getting action download info 2024-12-18T00:19:59.3084026Z ##[group]Run ./.github/actions/setup-linux 2024-12-18T00:19:59.3084352Z env: 2024-12-18T00:19:59.3084568Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:19:59.3084840Z ##[endgroup] 2024-12-18T00:19:59.3128959Z ##[group]Run set -euo pipefail 2024-12-18T00:19:59.3129297Z set -euo pipefail 2024-12-18T00:19:59.3129595Z function get_ec2_metadata() { 2024-12-18T00:19:59.3129979Z  # Pulled from instance metadata endpoint for EC2 2024-12-18T00:19:59.3130604Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-12-18T00:19:59.3131156Z  category=$1 2024-12-18T00:19:59.3131534Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-12-18T00:19:59.3131966Z  runner_name_str=i-079e9e3a7e7495371 2024-12-18T00:19:59.3132343Z  if [[ -f /.inarc ]]; then 2024-12-18T00:19:59.3132697Z  echo "ARC Runner, no info on ec2 metadata" 2024-12-18T00:19:59.3133079Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-12-18T00:19:59.3133536Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-12-18T00:19:59.3133961Z  else 2024-12-18T00:19:59.3134779Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-12-18T00:19:59.3135640Z  fi 2024-12-18T00:19:59.3135868Z } 2024-12-18T00:19:59.3136141Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-12-18T00:19:59.3136579Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-12-18T00:19:59.3137070Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-12-18T00:19:59.3137488Z echo "system info $(uname -a)" 2024-12-18T00:19:59.3147339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:19:59.3147714Z env: 2024-12-18T00:19:59.3147940Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:19:59.3148221Z ##[endgroup] 2024-12-18T00:19:59.3307002Z ami-id: ami-0fff1b9a61dec8a5f 2024-12-18T00:19:59.3430736Z instance-id: i-079e9e3a7e7495371 2024-12-18T00:19:59.3574759Z instance-type: g5.12xlarge 2024-12-18T00:19:59.3589358Z system info Linux ip-10-0-71-197.ec2.internal 6.1.109-118.189.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 10 08:59:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-12-18T00:19:59.3621778Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:19:59.3622702Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:19:59.3631958Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:19:59.3632352Z env: 2024-12-18T00:19:59.3632587Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:19:59.3632860Z ##[endgroup] 2024-12-18T00:19:59.3697932Z ##[group]Run if systemctl is-active --quiet docker; then 2024-12-18T00:19:59.3698383Z if systemctl is-active --quiet docker; then 2024-12-18T00:19:59.3698768Z  echo "Docker daemon is running..."; 2024-12-18T00:19:59.3699095Z else 2024-12-18T00:19:59.3699493Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-12-18T00:19:59.3699937Z fi 2024-12-18T00:19:59.3708248Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:19:59.3708628Z env: 2024-12-18T00:19:59.3708864Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:19:59.3709317Z ##[endgroup] 2024-12-18T00:19:59.3798307Z Docker daemon is running... 2024-12-18T00:19:59.3846361Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:19:59.3846671Z with: 2024-12-18T00:19:59.3846884Z shell: bash 2024-12-18T00:19:59.3847334Z timeout_minutes: 5 2024-12-18T00:19:59.3847590Z max_attempts: 3 2024-12-18T00:19:59.3847857Z retry_wait_seconds: 30 2024-12-18T00:19:59.3849950Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-12-18T00:19:59.3852053Z polling_interval_seconds: 1 2024-12-18T00:19:59.3852343Z warning_on_retry: true 2024-12-18T00:19:59.3852612Z continue_on_error: false 2024-12-18T00:19:59.3852869Z env: 2024-12-18T00:19:59.3853103Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:19:59.3853378Z AWS_RETRY_MODE: standard 2024-12-18T00:19:59.3853646Z AWS_MAX_ATTEMPTS: 5 2024-12-18T00:19:59.3853915Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T00:19:59.3854192Z ##[endgroup] 2024-12-18T00:20:00.5554504Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:20:00.5555111Z Configure a credential helper to remove this warning. See 2024-12-18T00:20:00.5555698Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:20:00.5556077Z 2024-12-18T00:20:00.5556178Z Login Succeeded 2024-12-18T00:20:01.4706135Z Command completed after 1 attempt(s). 2024-12-18T00:20:01.4788742Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:20:01.4789283Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:20:01.4789739Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T00:20:01.4798811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:01.4799187Z env: 2024-12-18T00:20:01.4799421Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.4799699Z ##[endgroup] 2024-12-18T00:20:01.4901028Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:20:01.4901596Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T00:20:01.4902020Z # shellcheck disable=SC2046 2024-12-18T00:20:01.4902353Z docker stop $(docker ps -q) || true 2024-12-18T00:20:01.4902706Z # Prune all of the docker images 2024-12-18T00:20:01.4903043Z docker system prune -af 2024-12-18T00:20:01.4911893Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:01.4912276Z env: 2024-12-18T00:20:01.4912525Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.4912794Z ##[endgroup] 2024-12-18T00:20:01.5265344Z "docker stop" requires at least 1 argument. 2024-12-18T00:20:01.5265698Z See 'docker stop --help'. 2024-12-18T00:20:01.5265880Z 2024-12-18T00:20:01.5266061Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T00:20:01.5266331Z 2024-12-18T00:20:01.5266446Z Stop one or more running containers 2024-12-18T00:20:01.5465799Z Total reclaimed space: 0B 2024-12-18T00:20:01.5519072Z ##[group]Run set +e 2024-12-18T00:20:01.5519378Z set +e 2024-12-18T00:20:01.5519624Z set -x 2024-12-18T00:20:01.5519870Z  2024-12-18T00:20:01.5520127Z PT_DOMAIN=download.pytorch.org 2024-12-18T00:20:01.5520751Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-12-18T00:20:01.5521484Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-12-18T00:20:01.5522193Z # one is returned at random 2024-12-18T00:20:01.5522593Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-12-18T00:20:01.5523069Z  2024-12-18T00:20:01.5523527Z if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:20:01.5523966Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-12-18T00:20:01.5524478Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-12-18T00:20:01.5524866Z  2024-12-18T00:20:01.5525117Z  if [ -z "${RESOLVED_IP}" ]; then 2024-12-18T00:20:01.5525514Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-12-18T00:20:01.5525882Z  exit 1 2024-12-18T00:20:01.5526134Z  fi 2024-12-18T00:20:01.5526375Z fi 2024-12-18T00:20:01.5526603Z  2024-12-18T00:20:01.5526885Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-12-18T00:20:01.5527262Z  # Clean up any old records first 2024-12-18T00:20:01.5527646Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-12-18T00:20:01.5527980Z fi 2024-12-18T00:20:01.5528206Z  2024-12-18T00:20:01.5528538Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-12-18T00:20:01.5528934Z cat /etc/hosts 2024-12-18T00:20:01.5537791Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:01.5538173Z env: 2024-12-18T00:20:01.5538414Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.5538695Z ##[endgroup] 2024-12-18T00:20:01.5568190Z + PT_DOMAIN=download.pytorch.org 2024-12-18T00:20:01.5574609Z ++ dig -4 +short download.pytorch.org 2024-12-18T00:20:01.5575567Z ++ tail -n1 2024-12-18T00:20:01.5798935Z + RESOLVED_IP=3.171.85.110 2024-12-18T00:20:01.5799224Z + '[' -z 3.171.85.110 ']' 2024-12-18T00:20:01.5799511Z + grep -r download.pytorch.org /etc/hosts 2024-12-18T00:20:01.5814698Z 3.167.99.70 download.pytorch.org 2024-12-18T00:20:01.5817058Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-12-18T00:20:01.6916182Z + echo '3.171.85.110 download.pytorch.org' 2024-12-18T00:20:01.6917208Z + sudo tee -a /etc/hosts 2024-12-18T00:20:01.7240542Z 3.171.85.110 download.pytorch.org 2024-12-18T00:20:01.7262796Z + cat /etc/hosts 2024-12-18T00:20:01.7275855Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-12-18T00:20:01.7281076Z ::1 localhost6 localhost6.localdomain6 2024-12-18T00:20:01.7281439Z 3.171.85.110 download.pytorch.org 2024-12-18T00:20:01.7427192Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T00:20:01.7427668Z with: 2024-12-18T00:20:01.7428462Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7429344Z docker-build-dir: .ci/docker 2024-12-18T00:20:01.7429632Z working-directory: . 2024-12-18T00:20:01.7429986Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:01.7430437Z force-push: false 2024-12-18T00:20:01.7430668Z env: 2024-12-18T00:20:01.7430891Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.7431153Z ##[endgroup] 2024-12-18T00:20:01.7456411Z ##[group]Run set -ex 2024-12-18T00:20:01.7456707Z set -ex 2024-12-18T00:20:01.7456948Z  2024-12-18T00:20:01.7457347Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T00:20:01.7458023Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T00:20:01.7458571Z # job could then download the pre-built image as usual 2024-12-18T00:20:01.7459085Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T00:20:01.7459561Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7460020Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7460616Z  2024-12-18T00:20:01.7460976Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T00:20:01.7461412Z  exit 0 2024-12-18T00:20:01.7461652Z else 2024-12-18T00:20:01.7461934Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7462269Z fi 2024-12-18T00:20:01.7462498Z  2024-12-18T00:20:01.7462846Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T00:20:01.7463626Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T00:20:01.7464161Z  # use it as it is, but first let's extract the tag 2024-12-18T00:20:01.7464648Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T00:20:01.7465168Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7465679Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7466106Z else 2024-12-18T00:20:01.7466441Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T00:20:01.7466923Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7467558Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7468122Z fi 2024-12-18T00:20:01.7476804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:01.7477187Z env: 2024-12-18T00:20:01.7477413Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.7477697Z REPO_NAME: pytorch 2024-12-18T00:20:01.7478525Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7479394Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:20:01.7479768Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:01.7480163Z ##[endgroup] 2024-12-18T00:20:01.7508228Z + [[ ! -d .ci/docker ]] 2024-12-18T00:20:01.7508514Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T00:20:01.7508797Z + echo skip=false 2024-12-18T00:20:01.7510031Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T00:20:01.7516615Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7517446Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T00:20:01.7543755Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7544500Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7546182Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7578328Z ##[group]Run set +e 2024-12-18T00:20:01.7578642Z set +e 2024-12-18T00:20:01.7578884Z set -x 2024-12-18T00:20:01.7579124Z  2024-12-18T00:20:01.7579335Z login() { 2024-12-18T00:20:01.7579828Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:20:01.7580358Z } 2024-12-18T00:20:01.7580578Z  2024-12-18T00:20:01.7580802Z retry () { 2024-12-18T00:20:01.7581092Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:20:01.7581413Z } 2024-12-18T00:20:01.7581639Z  2024-12-18T00:20:01.7581883Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:20:01.7582191Z  2024-12-18T00:20:01.7582429Z START_TIME=$(date +%s) 2024-12-18T00:20:01.7582737Z # Wait up to 90 minutes 2024-12-18T00:20:01.7583262Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T00:20:01.7583751Z  # Check if image already exists, if it does then skip building it 2024-12-18T00:20:01.7584248Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T00:20:01.7584617Z  exit 0 2024-12-18T00:20:01.7584870Z  fi 2024-12-18T00:20:01.7585100Z  2024-12-18T00:20:01.7585488Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T00:20:01.7586130Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T00:20:01.7586780Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T00:20:01.7587298Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T00:20:01.7587709Z  # It's a Docker build job, let's build the image 2024-12-18T00:20:01.7588068Z  break 2024-12-18T00:20:01.7588305Z  else 2024-12-18T00:20:01.7588651Z  # It's a regular build job, wait for the image to become available 2024-12-18T00:20:01.7589070Z  sleep 300 2024-12-18T00:20:01.7589330Z  fi 2024-12-18T00:20:01.7589564Z done 2024-12-18T00:20:01.7589781Z  2024-12-18T00:20:01.7590168Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T00:20:01.7590755Z # be empty. The default action would be to continue rebuild the image 2024-12-18T00:20:01.7591261Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T00:20:01.7591715Z  # if we're on the base branch then use the parent commit 2024-12-18T00:20:01.7592122Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T00:20:01.7592434Z else 2024-12-18T00:20:01.7592769Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T00:20:01.7593252Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T00:20:01.7593616Z fi 2024-12-18T00:20:01.7593838Z  2024-12-18T00:20:01.7594082Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T00:20:01.7594439Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7594775Z  2024-12-18T00:20:01.7595382Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T00:20:01.7595924Z  exit 0 2024-12-18T00:20:01.7596161Z fi 2024-12-18T00:20:01.7596380Z  2024-12-18T00:20:01.7596707Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T00:20:01.7597386Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T00:20:01.7597957Z  exit 1 2024-12-18T00:20:01.7598201Z fi 2024-12-18T00:20:01.7598429Z  2024-12-18T00:20:01.7598807Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T00:20:01.7599465Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T00:20:01.7600063Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T00:20:01.7600789Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T00:20:01.7601552Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T00:20:01.7602020Z fi 2024-12-18T00:20:01.7602251Z  2024-12-18T00:20:01.7602535Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T00:20:01.7610729Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:01.7611122Z env: 2024-12-18T00:20:01.7611371Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:01.7611754Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T00:20:01.7612122Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:20:01.7613052Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7613958Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:01.7614428Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:01.7614841Z DOCKER_PUSH: 2024-12-18T00:20:01.7615088Z ##[endgroup] 2024-12-18T00:20:01.7642238Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:01.7642686Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:01.7645525Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:20:01.7647099Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:02.3238956Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:20:02.3239681Z Configure a credential helper to remove this warning. See 2024-12-18T00:20:02.3240476Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:20:02.3240889Z 2024-12-18T00:20:02.3241537Z Login Succeeded 2024-12-18T00:20:02.3271636Z ++ date +%s 2024-12-18T00:20:02.3283983Z + START_TIME=1734481202 2024-12-18T00:20:02.3288626Z ++ date +%s 2024-12-18T00:20:02.3300651Z + [[ 1734475802 -lt 1734481202 ]] 2024-12-18T00:20:02.3301780Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:02.5378831Z { 2024-12-18T00:20:02.5379238Z "schemaVersion": 2, 2024-12-18T00:20:02.5380444Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T00:20:02.5381043Z "config": { 2024-12-18T00:20:02.5381488Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T00:20:02.5382010Z "size": 52680, 2024-12-18T00:20:02.5382548Z "digest": "sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b" 2024-12-18T00:20:02.5383782Z }, 2024-12-18T00:20:02.5384078Z "layers": [ 2024-12-18T00:20:02.5384843Z { 2024-12-18T00:20:02.5385710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5386252Z "size": 28584506, 2024-12-18T00:20:02.5387041Z "digest": "sha256:80888bc6716fcbb8874e75ac88898d3e38e6f1bc55678f0e97ca9d706b7f3733" 2024-12-18T00:20:02.5388160Z }, 2024-12-18T00:20:02.5388589Z { 2024-12-18T00:20:02.5389563Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5390030Z "size": 7944698, 2024-12-18T00:20:02.5390653Z "digest": "sha256:fbcd35dc5bc3a7bda41926aadd083020f942b001ebac6f1d30480f0f065394c0" 2024-12-18T00:20:02.5391270Z }, 2024-12-18T00:20:02.5391626Z { 2024-12-18T00:20:02.5392079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5392561Z "size": 57593527, 2024-12-18T00:20:02.5393144Z "digest": "sha256:c7232af9ae05f7de83f8d6171bd0c35a4dd0a85ebafb15b950dbc08f89ea5fb5" 2024-12-18T00:20:02.5393734Z }, 2024-12-18T00:20:02.5394009Z { 2024-12-18T00:20:02.5394498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5395023Z "size": 185, 2024-12-18T00:20:02.5395492Z "digest": "sha256:db6cdef1932a0d9ca6ef9a539e08d491f66d1b1ed81926ae1525375bdd8100cc" 2024-12-18T00:20:02.5396258Z }, 2024-12-18T00:20:02.5396539Z { 2024-12-18T00:20:02.5397010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5397535Z "size": 6886, 2024-12-18T00:20:02.5397992Z "digest": "sha256:56dc8550293751a1604e97ac949cfae82ba20cb2a28e034737bafd7382559609" 2024-12-18T00:20:02.5398581Z }, 2024-12-18T00:20:02.5398888Z { 2024-12-18T00:20:02.5399299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5400392Z "size": 1374287670, 2024-12-18T00:20:02.5400902Z "digest": "sha256:30c0ea6140d07e2a8deb70d780f277c63cf61836ff33d66eef944728a4bef6bd" 2024-12-18T00:20:02.5401397Z }, 2024-12-18T00:20:02.5401763Z { 2024-12-18T00:20:02.5402183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5402743Z "size": 62641, 2024-12-18T00:20:02.5403369Z "digest": "sha256:71bdb1a72c2d6dc97bbdbca82383f0260c4ee87556701e8e606c08a6bb0f0da5" 2024-12-18T00:20:02.5403891Z }, 2024-12-18T00:20:02.5404373Z { 2024-12-18T00:20:02.5404802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5405319Z "size": 1685, 2024-12-18T00:20:02.5405877Z "digest": "sha256:4829486be7c30f19f4136fa56adbb3de206ed0bbf0705b59fb2147406778ce38" 2024-12-18T00:20:02.5406401Z }, 2024-12-18T00:20:02.5406706Z { 2024-12-18T00:20:02.5407174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5407677Z "size": 1523, 2024-12-18T00:20:02.5408180Z "digest": "sha256:1f4e68d7b5e4224ba1da78ef461ff7f01e8d59c09d39281277521384105a9441" 2024-12-18T00:20:02.5408756Z }, 2024-12-18T00:20:02.5409142Z { 2024-12-18T00:20:02.5409583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5410173Z "size": 2626741021, 2024-12-18T00:20:02.5410721Z "digest": "sha256:7c373e2d9b7e82a6878d4a31293dd857915a0fe47d07dce541cea03b043d57fc" 2024-12-18T00:20:02.5411315Z }, 2024-12-18T00:20:02.5411601Z { 2024-12-18T00:20:02.5412057Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5412620Z "size": 86545, 2024-12-18T00:20:02.5413265Z "digest": "sha256:6223811417458a3c93b84ee3b65f8b08d9e2828b926f0aed863041610d7d95d4" 2024-12-18T00:20:02.5413971Z }, 2024-12-18T00:20:02.5414273Z { 2024-12-18T00:20:02.5414785Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5415358Z "size": 1824, 2024-12-18T00:20:02.5415874Z "digest": "sha256:f003d065e31e5e8c0cea490c88a912dd331bddcbd2f5ff22ccc692a5343bcbeb" 2024-12-18T00:20:02.5416463Z }, 2024-12-18T00:20:02.5416831Z { 2024-12-18T00:20:02.5417225Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5417743Z "size": 246420139, 2024-12-18T00:20:02.5418339Z "digest": "sha256:cd6ce71168153e485b99493ce07032bbd9fd5cefceb68ef33ce088244306a653" 2024-12-18T00:20:02.5418928Z }, 2024-12-18T00:20:02.5419262Z { 2024-12-18T00:20:02.5419849Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5420423Z "size": 703, 2024-12-18T00:20:02.5420961Z "digest": "sha256:e843066a8f68042266cfe814f1729975619fb8f733f371be115e8b2fe81cbc9d" 2024-12-18T00:20:02.5421557Z }, 2024-12-18T00:20:02.5421861Z { 2024-12-18T00:20:02.5422342Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5422830Z "size": 1256, 2024-12-18T00:20:02.5423346Z "digest": "sha256:bac50ca2930bc4b1afa8a2f1b4b0d1d499d63b7922b784af690c3b31ec0c2f70" 2024-12-18T00:20:02.5424001Z }, 2024-12-18T00:20:02.5424290Z { 2024-12-18T00:20:02.5424706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5425280Z "size": 484, 2024-12-18T00:20:02.5425885Z "digest": "sha256:57e52c6751f4d865ada4cae919859fe563fe5dcbb93ca82cb4746f98caccbd3a" 2024-12-18T00:20:02.5426447Z }, 2024-12-18T00:20:02.5426872Z { 2024-12-18T00:20:02.5427317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5427781Z "size": 91715265, 2024-12-18T00:20:02.5428395Z "digest": "sha256:a9c5c14d77c035c7501566b1247024dfd241bd6bc12fbe2bca28ec7d90b26069" 2024-12-18T00:20:02.5428966Z }, 2024-12-18T00:20:02.5429321Z { 2024-12-18T00:20:02.5429778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5430291Z "size": 3706, 2024-12-18T00:20:02.5430810Z "digest": "sha256:b86099b8619ad88e8fe6c8e693180d52e0f2fbeac5927f0b403c8a9ef76d7f6b" 2024-12-18T00:20:02.5431575Z }, 2024-12-18T00:20:02.5431899Z { 2024-12-18T00:20:02.5432383Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5432896Z "size": 1860, 2024-12-18T00:20:02.5433400Z "digest": "sha256:3c8ea8a600fff6cce4c1b36acc5f3949ecd664e55b5eaf4a50fe5499351574a6" 2024-12-18T00:20:02.5434019Z }, 2024-12-18T00:20:02.5434338Z { 2024-12-18T00:20:02.5434750Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5435339Z "size": 701, 2024-12-18T00:20:02.5435847Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:20:02.5436415Z }, 2024-12-18T00:20:02.5436863Z { 2024-12-18T00:20:02.5437316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5437876Z "size": 478, 2024-12-18T00:20:02.5438358Z "digest": "sha256:292b18ec2dab9bd60d00192387c1269d39b0b4886b782585d2c879808218fc66" 2024-12-18T00:20:02.5438960Z }, 2024-12-18T00:20:02.5439302Z { 2024-12-18T00:20:02.5439778Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5440310Z "size": 2941869535, 2024-12-18T00:20:02.5440938Z "digest": "sha256:76e3817b4b4cec0a5b005161867dc257681b792271ba9d23c9b79b3a6801dcd9" 2024-12-18T00:20:02.5441498Z }, 2024-12-18T00:20:02.5441795Z { 2024-12-18T00:20:02.5442395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5442933Z "size": 380, 2024-12-18T00:20:02.5443506Z "digest": "sha256:50b1f245e9f5be8670d8ba5790e3b96450eee3428dcac7060367db48ae00234e" 2024-12-18T00:20:02.5444140Z }, 2024-12-18T00:20:02.5444438Z { 2024-12-18T00:20:02.5444857Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5445505Z "size": 12901, 2024-12-18T00:20:02.5446036Z "digest": "sha256:ad93cf5205f24ada86b5f7eab5b63ed97ae7858368f6fdbce373343d2568d66e" 2024-12-18T00:20:02.5446567Z }, 2024-12-18T00:20:02.5446971Z { 2024-12-18T00:20:02.5447424Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5448009Z "size": 863, 2024-12-18T00:20:02.5448648Z "digest": "sha256:8d17c59eaf7d84d49e3660380debcc70734b2a178e6cab86f0d4fd3e7a8100d1" 2024-12-18T00:20:02.5449198Z }, 2024-12-18T00:20:02.5449556Z { 2024-12-18T00:20:02.5450019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5450629Z "size": 106, 2024-12-18T00:20:02.5451326Z "digest": "sha256:b36a8ccf5c8c8ef0bf3e168ec9f568a0ada36c49103c9cacfef6dd072ddd681a" 2024-12-18T00:20:02.5451927Z }, 2024-12-18T00:20:02.5452211Z { 2024-12-18T00:20:02.5452701Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5453255Z "size": 504, 2024-12-18T00:20:02.5453764Z "digest": "sha256:aa0a3b87e9e34ab15903206018bfd9fb1eb54fd21f05a61ef254557703890732" 2024-12-18T00:20:02.5454507Z }, 2024-12-18T00:20:02.5454796Z { 2024-12-18T00:20:02.5455221Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5455847Z "size": 121477314, 2024-12-18T00:20:02.5456406Z "digest": "sha256:72a632052c5be4900c60ce09e32f8a0fd3b637ab0b7685dc5b07bda4776ddd15" 2024-12-18T00:20:02.5457035Z }, 2024-12-18T00:20:02.5457315Z { 2024-12-18T00:20:02.5457791Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5458356Z "size": 109, 2024-12-18T00:20:02.5458949Z "digest": "sha256:ceed6316bc249ffffee9563cb7505023b3e70fba306f187bd612fece63508263" 2024-12-18T00:20:02.5459544Z }, 2024-12-18T00:20:02.5459929Z { 2024-12-18T00:20:02.5460416Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5460942Z "size": 489, 2024-12-18T00:20:02.5461537Z "digest": "sha256:d0ca0f491f809a9f1dedd391f5b25cc3f59d8e95e7f76299f18ec3c931e56ed1" 2024-12-18T00:20:02.5462132Z }, 2024-12-18T00:20:02.5462394Z { 2024-12-18T00:20:02.5462904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5464167Z "size": 296, 2024-12-18T00:20:02.5464668Z "digest": "sha256:cf02c706cd567c781eeb567c90960801dc1ef78872b73fbc047df898951ecbb8" 2024-12-18T00:20:02.5465356Z }, 2024-12-18T00:20:02.5465738Z { 2024-12-18T00:20:02.5466142Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5466746Z "size": 103, 2024-12-18T00:20:02.5467277Z "digest": "sha256:16073d3f107dab837d7edb5e6044449660fec7e2ded263ca81d5dcfea5beb486" 2024-12-18T00:20:02.5467937Z }, 2024-12-18T00:20:02.5468261Z { 2024-12-18T00:20:02.5468648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5469238Z "size": 1473, 2024-12-18T00:20:02.5469793Z "digest": "sha256:d234faf2f1bfb6cb9d4ba14103226d20d540e568b47690b67c2a2b2c5a49645f" 2024-12-18T00:20:02.5470433Z }, 2024-12-18T00:20:02.5471074Z { 2024-12-18T00:20:02.5471475Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5472121Z "size": 430258758, 2024-12-18T00:20:02.5472733Z "digest": "sha256:3f0ade86af1a5bc72c9088735bb1a7ab9819fa20fdd96437fc080b2c711dfd3d" 2024-12-18T00:20:02.5473347Z }, 2024-12-18T00:20:02.5473630Z { 2024-12-18T00:20:02.5474091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5474643Z "size": 164, 2024-12-18T00:20:02.5475152Z "digest": "sha256:034b6a03f5f751ed12f7505175ce3443eb3ed7b34d02d1bb25de0ee52d93a44b" 2024-12-18T00:20:02.5475806Z }, 2024-12-18T00:20:02.5476146Z { 2024-12-18T00:20:02.5476614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5477125Z "size": 564, 2024-12-18T00:20:02.5477748Z "digest": "sha256:21f8f066852dd484ccbf79470452fe8005cde08773dd4fedc07fed783a0b9bf8" 2024-12-18T00:20:02.5478360Z }, 2024-12-18T00:20:02.5478703Z { 2024-12-18T00:20:02.5479141Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5479664Z "size": 35865656, 2024-12-18T00:20:02.5480267Z "digest": "sha256:9dc1f315a979ec860a882d7df33584e02c1d18f86293521839ce42e21082e639" 2024-12-18T00:20:02.5480851Z }, 2024-12-18T00:20:02.5481214Z { 2024-12-18T00:20:02.5481644Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5482133Z "size": 104, 2024-12-18T00:20:02.5482714Z "digest": "sha256:cd6a6063178b945c150fe21a31df6e33203295dd814802e0658835e810fb2bf9" 2024-12-18T00:20:02.5483436Z }, 2024-12-18T00:20:02.5483725Z { 2024-12-18T00:20:02.5484406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5484895Z "size": 425, 2024-12-18T00:20:02.5485411Z "digest": "sha256:2116738f4ea9299766a55629d39b881c4d33fc4fc0ce915965be1a850fcd7aaa" 2024-12-18T00:20:02.5486078Z }, 2024-12-18T00:20:02.5486375Z { 2024-12-18T00:20:02.5486866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5487420Z "size": 20262160, 2024-12-18T00:20:02.5487919Z "digest": "sha256:d7fa134316d91d2ec39bceb121619d44cdfdf1c7ac30c72768a9fd48f9763bd1" 2024-12-18T00:20:02.5488572Z }, 2024-12-18T00:20:02.5488983Z { 2024-12-18T00:20:02.5489449Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5490037Z "size": 643, 2024-12-18T00:20:02.5490524Z "digest": "sha256:af03dc3fd15ea8f5f1fa41d514400f6a1d963985cf046b09d789f7f0359b2aba" 2024-12-18T00:20:02.5491130Z }, 2024-12-18T00:20:02.5491476Z { 2024-12-18T00:20:02.5491945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5492451Z "size": 701, 2024-12-18T00:20:02.5493004Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:20:02.5493618Z }, 2024-12-18T00:20:02.5493905Z { 2024-12-18T00:20:02.5494499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5495048Z "size": 142, 2024-12-18T00:20:02.5495511Z "digest": "sha256:fd9de3dc425828e7b8984d4ced05f4102762137b81a5dc67104fc6b0805c570d" 2024-12-18T00:20:02.5496326Z }, 2024-12-18T00:20:02.5496635Z { 2024-12-18T00:20:02.5497128Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5497705Z "size": 135, 2024-12-18T00:20:02.5498232Z "digest": "sha256:b85093661f7768ac691f4b3ee5f553b9ab14cb2a6877c8edc6133d74caa88f69" 2024-12-18T00:20:02.5498799Z }, 2024-12-18T00:20:02.5499147Z { 2024-12-18T00:20:02.5499591Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5500290Z "size": 4995482131, 2024-12-18T00:20:02.5500838Z "digest": "sha256:ab6ef6c2ac4d23af7a3249cee1d4761f63a89513796ba720da7cc425c8afa6d2" 2024-12-18T00:20:02.5501385Z }, 2024-12-18T00:20:02.5501752Z { 2024-12-18T00:20:02.5502196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5502717Z "size": 196, 2024-12-18T00:20:02.5503324Z "digest": "sha256:56c896077120aaa4cbcc2203662d13b4ea14cf998e9657e91b252535ceae5348" 2024-12-18T00:20:02.5503860Z }, 2024-12-18T00:20:02.5504173Z { 2024-12-18T00:20:02.5504678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5505211Z "size": 564, 2024-12-18T00:20:02.5505877Z "digest": "sha256:d88e9c5e7c70fe36bd1bd2e94d43d4192d94a1f28e323234a73a75a2d1da48f1" 2024-12-18T00:20:02.5506438Z }, 2024-12-18T00:20:02.5506746Z { 2024-12-18T00:20:02.5507234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5507789Z "size": 43164612, 2024-12-18T00:20:02.5508350Z "digest": "sha256:53617ee83302d7f97eb56f6756d78d18f8dc41bf252bf2d04cb643198f400f6f" 2024-12-18T00:20:02.5509056Z }, 2024-12-18T00:20:02.5509321Z { 2024-12-18T00:20:02.5509770Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5510364Z "size": 106, 2024-12-18T00:20:02.5510913Z "digest": "sha256:3edbde10d4043bf2ebf32fddfa55c8d0923435465637df7dc8f4addcc03a0b84" 2024-12-18T00:20:02.5511618Z }, 2024-12-18T00:20:02.5511891Z { 2024-12-18T00:20:02.5512355Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5512910Z "size": 1402, 2024-12-18T00:20:02.5513487Z "digest": "sha256:f4038f7524c3ed705e7560db6dd6efe76f548363f16bc7a93cfe360d5bc6dcaa" 2024-12-18T00:20:02.5514069Z }, 2024-12-18T00:20:02.5514362Z { 2024-12-18T00:20:02.5514844Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5515366Z "size": 701, 2024-12-18T00:20:02.5516077Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:20:02.5516679Z }, 2024-12-18T00:20:02.5517021Z { 2024-12-18T00:20:02.5517552Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5518088Z "size": 137, 2024-12-18T00:20:02.5518584Z "digest": "sha256:4a51173f77576ab68fba2f862dd4d12b9ff1f250145c090520b05f88f5c81391" 2024-12-18T00:20:02.5519257Z }, 2024-12-18T00:20:02.5519561Z { 2024-12-18T00:20:02.5519948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5520596Z "size": 120, 2024-12-18T00:20:02.5521123Z "digest": "sha256:ed6a71d41274e5b1eea4025d24a4851d6880b902b08f7d4dd0618ea81154f575" 2024-12-18T00:20:02.5521761Z }, 2024-12-18T00:20:02.5522093Z { 2024-12-18T00:20:02.5522582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5523250Z "size": 4886226234, 2024-12-18T00:20:02.5523805Z "digest": "sha256:389dc939e2a414ef26b642b1e8657a710358afa6a6be3ebc4380b01ffacbbacc" 2024-12-18T00:20:02.5524409Z }, 2024-12-18T00:20:02.5524764Z { 2024-12-18T00:20:02.5525166Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5525723Z "size": 174, 2024-12-18T00:20:02.5526300Z "digest": "sha256:0341eaeed33df462f8bd28003bb9b09739b0786abba58ccc17ba4cec535ee4ef" 2024-12-18T00:20:02.5526894Z }, 2024-12-18T00:20:02.5527204Z { 2024-12-18T00:20:02.5527670Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5528406Z "size": 908, 2024-12-18T00:20:02.5528919Z "digest": "sha256:537cdb2d31685c0dbaf612d24633539a141cfcb69d90c5ce932923a6e4ab4474" 2024-12-18T00:20:02.5529578Z }, 2024-12-18T00:20:02.5529906Z { 2024-12-18T00:20:02.5530385Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5530891Z "size": 701, 2024-12-18T00:20:02.5531448Z "digest": "sha256:bebd539a1d598fe86afc0da77bf16ae4c7ebfb9f031f21ed7634150fbe23f996" 2024-12-18T00:20:02.5532090Z }, 2024-12-18T00:20:02.5532431Z { 2024-12-18T00:20:02.5532866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5533379Z "size": 135, 2024-12-18T00:20:02.5534022Z "digest": "sha256:76d4a07024975dd727ff7b1c8873c513043bc9a5d310b469e51c3b110fdb6084" 2024-12-18T00:20:02.5534606Z }, 2024-12-18T00:20:02.5534986Z { 2024-12-18T00:20:02.5535419Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5535910Z "size": 32, 2024-12-18T00:20:02.5536512Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:20:02.5537103Z }, 2024-12-18T00:20:02.5537389Z { 2024-12-18T00:20:02.5537914Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5538396Z "size": 159, 2024-12-18T00:20:02.5538888Z "digest": "sha256:3287a539d61e3913b93941ecd49e55887b5122d63403e0fe9e7a3a63b6ed03c6" 2024-12-18T00:20:02.5539567Z }, 2024-12-18T00:20:02.5539959Z { 2024-12-18T00:20:02.5540444Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5540981Z "size": 1932, 2024-12-18T00:20:02.5541455Z "digest": "sha256:9b763e24a7d42c731eb337815675c9478832146e73305559d0db1c88a0731492" 2024-12-18T00:20:02.5542070Z }, 2024-12-18T00:20:02.5542394Z { 2024-12-18T00:20:02.5542856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5543435Z "size": 196810750, 2024-12-18T00:20:02.5543928Z "digest": "sha256:f6ef3d4c814e330a230273579dc978970563774df75f73fba78181ddcf6ab98e" 2024-12-18T00:20:02.5544536Z }, 2024-12-18T00:20:02.5544887Z { 2024-12-18T00:20:02.5545414Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5545946Z "size": 165, 2024-12-18T00:20:02.5546484Z "digest": "sha256:06914b00eb49b8005245cba4a631c34d0a2aaf8bf6291bc6477705206de11a06" 2024-12-18T00:20:02.5547065Z }, 2024-12-18T00:20:02.5547364Z { 2024-12-18T00:20:02.5547963Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5548529Z "size": 7943, 2024-12-18T00:20:02.5548987Z "digest": "sha256:724553524224508e03de9422ca47b1a1c047e53cd305c2a1344147bb0d82d1d0" 2024-12-18T00:20:02.5549640Z }, 2024-12-18T00:20:02.5549966Z { 2024-12-18T00:20:02.5550439Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5551089Z "size": 8070, 2024-12-18T00:20:02.5551616Z "digest": "sha256:47d9fb6dae26902b0fdb744916689adbd42add22a476d1c5ec89ae4e30b91e39" 2024-12-18T00:20:02.5552201Z }, 2024-12-18T00:20:02.5552540Z { 2024-12-18T00:20:02.5552978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5553587Z "size": 303, 2024-12-18T00:20:02.5554118Z "digest": "sha256:acce7b690dcb2fde77ba3ce65df70452fc7c98de73a42fe72652e477f86fc62d" 2024-12-18T00:20:02.5554678Z }, 2024-12-18T00:20:02.5555054Z { 2024-12-18T00:20:02.5555503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5556018Z "size": 7633763, 2024-12-18T00:20:02.5556744Z "digest": "sha256:58dc9f64def75bcb2abecdf39b798c3800e837d37da971e766a05731ea5537c2" 2024-12-18T00:20:02.5557285Z }, 2024-12-18T00:20:02.5557574Z { 2024-12-18T00:20:02.5558101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5558631Z "size": 108, 2024-12-18T00:20:02.5559207Z "digest": "sha256:94dfe5ef1b7e91ebd6a1a5acc1d7732e051334d5102d8a69c509c1d2f6e95598" 2024-12-18T00:20:02.5559859Z }, 2024-12-18T00:20:02.5560162Z { 2024-12-18T00:20:02.5560648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5561199Z "size": 54145665, 2024-12-18T00:20:02.5561737Z "digest": "sha256:40a2c9135b0683a7e2432748a9a6ff19926f6a2a49722d844c07e307320f7362" 2024-12-18T00:20:02.5562443Z }, 2024-12-18T00:20:02.5562703Z { 2024-12-18T00:20:02.5563222Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5564087Z "size": 484, 2024-12-18T00:20:02.5587140Z "digest": "sha256:8839d6b0c8218eccb426367e5bfe066b172b8c6059bae8473e915e11c4e582a9" 2024-12-18T00:20:02.5587663Z }, 2024-12-18T00:20:02.5587859Z { 2024-12-18T00:20:02.5588201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5588621Z "size": 1374859271, 2024-12-18T00:20:02.5589042Z "digest": "sha256:1c588f0a9e97f3e192252a179d987d1ba1e5201ba4b6e74c429074e9e34df1b2" 2024-12-18T00:20:02.5589499Z }, 2024-12-18T00:20:02.5589702Z { 2024-12-18T00:20:02.5590033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5590452Z "size": 106, 2024-12-18T00:20:02.5590862Z "digest": "sha256:6727e7e279360b8b66f0137a15b357efede06e69bfd0800ebad6ed7c8ed71df4" 2024-12-18T00:20:02.5591320Z }, 2024-12-18T00:20:02.5591512Z { 2024-12-18T00:20:02.5591841Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5592265Z "size": 568, 2024-12-18T00:20:02.5592678Z "digest": "sha256:3a55100d1957b103c43e314deb55697c459aa6fa225f91e7b1c69bc4c01dca4e" 2024-12-18T00:20:02.5593151Z }, 2024-12-18T00:20:02.5593340Z { 2024-12-18T00:20:02.5593674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5594096Z "size": 303706071, 2024-12-18T00:20:02.5594517Z "digest": "sha256:3d6f160d335161c5321e13ce2823ec68379a6a3401b07bd7378602a69c85b80c" 2024-12-18T00:20:02.5594980Z }, 2024-12-18T00:20:02.5595168Z { 2024-12-18T00:20:02.5595507Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5595929Z "size": 111, 2024-12-18T00:20:02.5596344Z "digest": "sha256:b2a8bb947a87562144fe67a39c280d4dee50cbe9d8ff349140f5db6c377517e6" 2024-12-18T00:20:02.5596824Z }, 2024-12-18T00:20:02.5597017Z { 2024-12-18T00:20:02.5597349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5597772Z "size": 529, 2024-12-18T00:20:02.5598361Z "digest": "sha256:69a74b2360f4ede95654274ce7ef6cc1597a7234cc17e86d5cdf4c98286d97c3" 2024-12-18T00:20:02.5598841Z }, 2024-12-18T00:20:02.5599035Z { 2024-12-18T00:20:02.5599370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5599802Z "size": 48289052, 2024-12-18T00:20:02.5600275Z "digest": "sha256:f9e12a253d89c16863b9d4d90138ffa673447c9b7f27afcba54a2a679e58b5ab" 2024-12-18T00:20:02.5600754Z }, 2024-12-18T00:20:02.5600943Z { 2024-12-18T00:20:02.5601282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5601717Z "size": 106, 2024-12-18T00:20:02.5602134Z "digest": "sha256:8fdb06d51b96d538045b9566a381de3dc25e59d1adb9d0a4e567afcb3213f676" 2024-12-18T00:20:02.5602611Z }, 2024-12-18T00:20:02.5602799Z { 2024-12-18T00:20:02.5603213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5603632Z "size": 32, 2024-12-18T00:20:02.5604050Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:20:02.5604529Z }, 2024-12-18T00:20:02.5604716Z { 2024-12-18T00:20:02.5605044Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5605467Z "size": 32, 2024-12-18T00:20:02.5605884Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:20:02.5606370Z }, 2024-12-18T00:20:02.5606562Z { 2024-12-18T00:20:02.5606894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5607455Z "size": 32, 2024-12-18T00:20:02.5607869Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:20:02.5608348Z }, 2024-12-18T00:20:02.5608541Z { 2024-12-18T00:20:02.5608881Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T00:20:02.5609304Z "size": 32, 2024-12-18T00:20:02.5609721Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-12-18T00:20:02.5610257Z } 2024-12-18T00:20:02.5610445Z ] 2024-12-18T00:20:02.5610638Z } 2024-12-18T00:20:02.5610844Z + exit 0 2024-12-18T00:20:02.5651088Z ##[group]Run set -eux 2024-12-18T00:20:02.5651395Z set -eux 2024-12-18T00:20:02.5652233Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T00:20:02.5661834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:02.5662230Z env: 2024-12-18T00:20:02.5662462Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:02.5662747Z ##[endgroup] 2024-12-18T00:20:02.5701317Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T00:20:02.5701789Z + jq --raw-output .SecretString 2024-12-18T00:20:02.5703320Z + jq -r .docker_hub_readonly_token 2024-12-18T00:20:02.5704412Z + docker login --username pytorchbot --password-stdin 2024-12-18T00:20:03.1887333Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:20:03.1887920Z Configure a credential helper to remove this warning. See 2024-12-18T00:20:03.1888481Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:20:03.1888854Z 2024-12-18T00:20:03.1888959Z Login Succeeded 2024-12-18T00:20:03.1987837Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:20:03.1988214Z tag=${ECR_DOCKER_IMAGE##*/} 2024-12-18T00:20:03.1988617Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-12-18T00:20:03.1997335Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:03.1997729Z env: 2024-12-18T00:20:03.1997969Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:03.1998820Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.1999670Z ##[endgroup] 2024-12-18T00:20:03.2029184Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks-45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.2084802Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T00:20:03.2085267Z with: 2024-12-18T00:20:03.2086050Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.2086993Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:03.2087399Z env: 2024-12-18T00:20:03.2087627Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:03.2087901Z ##[endgroup] 2024-12-18T00:20:03.2109265Z ##[group]Run set -x 2024-12-18T00:20:03.2109561Z set -x 2024-12-18T00:20:03.2109803Z set +e 2024-12-18T00:20:03.2110038Z  2024-12-18T00:20:03.2110272Z login() { 2024-12-18T00:20:03.2110752Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T00:20:03.2111263Z } 2024-12-18T00:20:03.2111488Z  2024-12-18T00:20:03.2111746Z retry () { 2024-12-18T00:20:03.2112047Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T00:20:03.2112366Z } 2024-12-18T00:20:03.2112594Z  2024-12-18T00:20:03.2112854Z retry login "${DOCKER_REGISTRY}" 2024-12-18T00:20:03.2113179Z  2024-12-18T00:20:03.2113409Z set -e 2024-12-18T00:20:03.2113936Z # ignore output since only exit code is used for conditional 2024-12-18T00:20:03.2114441Z # only pull docker image if it's not available locally 2024-12-18T00:20:03.2114977Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T00:20:03.2115482Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T00:20:03.2115805Z fi 2024-12-18T00:20:03.2124820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:20:03.2125196Z env: 2024-12-18T00:20:03.2125429Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:20:03.2126269Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.2127212Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:03.2127599Z ##[endgroup] 2024-12-18T00:20:03.2157717Z + set +e 2024-12-18T00:20:03.2158036Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:03.2158478Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:03.2161767Z + aws ecr get-login-password --region us-east-1 2024-12-18T00:20:03.2162797Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T00:20:03.7654509Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-12-18T00:20:03.7655261Z Configure a credential helper to remove this warning. See 2024-12-18T00:20:03.7655988Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T00:20:03.7656496Z 2024-12-18T00:20:03.7656604Z Login Succeeded 2024-12-18T00:20:03.7679169Z + set -e 2024-12-18T00:20:03.7680167Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.7862089Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:03.7863784Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:20:04.0861182Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks 2024-12-18T00:20:04.0863878Z 80888bc6716f: Pulling fs layer 2024-12-18T00:20:04.0864194Z fbcd35dc5bc3: Pulling fs layer 2024-12-18T00:20:04.0864796Z c7232af9ae05: Pulling fs layer 2024-12-18T00:20:04.0865121Z db6cdef1932a: Pulling fs layer 2024-12-18T00:20:04.0865405Z 56dc85502937: Pulling fs layer 2024-12-18T00:20:04.0865694Z 30c0ea6140d0: Pulling fs layer 2024-12-18T00:20:04.0865986Z 71bdb1a72c2d: Pulling fs layer 2024-12-18T00:20:04.0866273Z 4829486be7c3: Pulling fs layer 2024-12-18T00:20:04.0866552Z 1f4e68d7b5e4: Pulling fs layer 2024-12-18T00:20:04.0866854Z 7c373e2d9b7e: Pulling fs layer 2024-12-18T00:20:04.0867140Z 622381141745: Pulling fs layer 2024-12-18T00:20:04.0867425Z f003d065e31e: Pulling fs layer 2024-12-18T00:20:04.0867729Z cd6ce7116815: Pulling fs layer 2024-12-18T00:20:04.0868013Z e843066a8f68: Pulling fs layer 2024-12-18T00:20:04.0868302Z bac50ca2930b: Pulling fs layer 2024-12-18T00:20:04.0868589Z 57e52c6751f4: Pulling fs layer 2024-12-18T00:20:04.0868882Z a9c5c14d77c0: Pulling fs layer 2024-12-18T00:20:04.0869175Z b86099b8619a: Pulling fs layer 2024-12-18T00:20:04.0869460Z 3c8ea8a600ff: Pulling fs layer 2024-12-18T00:20:04.0869752Z bebd539a1d59: Pulling fs layer 2024-12-18T00:20:04.0870039Z db6cdef1932a: Waiting 2024-12-18T00:20:04.0870303Z 292b18ec2dab: Pulling fs layer 2024-12-18T00:20:04.0870591Z 76e3817b4b4c: Pulling fs layer 2024-12-18T00:20:04.0870877Z 50b1f245e9f5: Pulling fs layer 2024-12-18T00:20:04.0871145Z 56dc85502937: Waiting 2024-12-18T00:20:04.0871405Z 30c0ea6140d0: Waiting 2024-12-18T00:20:04.0871846Z ad93cf5205f2: Pulling fs layer 2024-12-18T00:20:04.0872148Z 8d17c59eaf7d: Pulling fs layer 2024-12-18T00:20:04.0872444Z b36a8ccf5c8c: Pulling fs layer 2024-12-18T00:20:04.0872730Z aa0a3b87e9e3: Pulling fs layer 2024-12-18T00:20:04.0873022Z 72a632052c5b: Pulling fs layer 2024-12-18T00:20:04.0875194Z 7c373e2d9b7e: Waiting 2024-12-18T00:20:04.0875468Z ceed6316bc24: Pulling fs layer 2024-12-18T00:20:04.0875746Z 4829486be7c3: Waiting 2024-12-18T00:20:04.0875980Z 622381141745: Waiting 2024-12-18T00:20:04.0876228Z 71bdb1a72c2d: Waiting 2024-12-18T00:20:04.0876490Z d0ca0f491f80: Pulling fs layer 2024-12-18T00:20:04.0876776Z 57e52c6751f4: Waiting 2024-12-18T00:20:04.0877037Z cf02c706cd56: Pulling fs layer 2024-12-18T00:20:04.0877316Z a9c5c14d77c0: Waiting 2024-12-18T00:20:04.0877567Z b86099b8619a: Waiting 2024-12-18T00:20:04.0877815Z 3c8ea8a600ff: Waiting 2024-12-18T00:20:04.0878074Z 16073d3f107d: Pulling fs layer 2024-12-18T00:20:04.0878365Z d234faf2f1bf: Pulling fs layer 2024-12-18T00:20:04.0878655Z 3f0ade86af1a: Pulling fs layer 2024-12-18T00:20:04.0878941Z 034b6a03f5f7: Pulling fs layer 2024-12-18T00:20:04.0879215Z cd6ce7116815: Waiting 2024-12-18T00:20:04.0879472Z 21f8f066852d: Pulling fs layer 2024-12-18T00:20:04.0879766Z 9dc1f315a979: Pulling fs layer 2024-12-18T00:20:04.0880035Z e843066a8f68: Waiting 2024-12-18T00:20:04.0880281Z 292b18ec2dab: Waiting 2024-12-18T00:20:04.0880539Z cd6a6063178b: Pulling fs layer 2024-12-18T00:20:04.0880824Z 2116738f4ea9: Pulling fs layer 2024-12-18T00:20:04.0881102Z 50b1f245e9f5: Waiting 2024-12-18T00:20:04.0881344Z bac50ca2930b: Waiting 2024-12-18T00:20:04.0881594Z ceed6316bc24: Waiting 2024-12-18T00:20:04.0881859Z d7fa134316d9: Pulling fs layer 2024-12-18T00:20:04.0882138Z aa0a3b87e9e3: Waiting 2024-12-18T00:20:04.0882398Z af03dc3fd15e: Pulling fs layer 2024-12-18T00:20:04.0882670Z f003d065e31e: Waiting 2024-12-18T00:20:04.0882915Z d234faf2f1bf: Waiting 2024-12-18T00:20:04.0883288Z d0ca0f491f80: Waiting 2024-12-18T00:20:04.0883539Z 3f0ade86af1a: Waiting 2024-12-18T00:20:04.0883804Z fd9de3dc4258: Pulling fs layer 2024-12-18T00:20:04.0884084Z 034b6a03f5f7: Waiting 2024-12-18T00:20:04.0884331Z 2116738f4ea9: Waiting 2024-12-18T00:20:04.0884574Z cf02c706cd56: Waiting 2024-12-18T00:20:04.0884825Z 8d17c59eaf7d: Waiting 2024-12-18T00:20:04.0885082Z d7fa134316d9: Waiting 2024-12-18T00:20:04.0885364Z b85093661f77: Pulling fs layer 2024-12-18T00:20:04.0885631Z 16073d3f107d: Waiting 2024-12-18T00:20:04.0885878Z bebd539a1d59: Waiting 2024-12-18T00:20:04.0886142Z ab6ef6c2ac4d: Pulling fs layer 2024-12-18T00:20:04.0886424Z 21f8f066852d: Waiting 2024-12-18T00:20:04.0886670Z 56c896077120: Pulling fs layer 2024-12-18T00:20:04.0887087Z cd6a6063178b: Waiting 2024-12-18T00:20:04.0887338Z af03dc3fd15e: Waiting 2024-12-18T00:20:04.0887601Z d88e9c5e7c70: Pulling fs layer 2024-12-18T00:20:04.0887880Z b36a8ccf5c8c: Waiting 2024-12-18T00:20:04.0888122Z 76e3817b4b4c: Waiting 2024-12-18T00:20:04.0888376Z 53617ee83302: Pulling fs layer 2024-12-18T00:20:04.0888667Z 3edbde10d404: Pulling fs layer 2024-12-18T00:20:04.0888948Z 72a632052c5b: Waiting 2024-12-18T00:20:04.0889204Z f4038f7524c3: Pulling fs layer 2024-12-18T00:20:04.0889475Z fd9de3dc4258: Waiting 2024-12-18T00:20:04.0889724Z 53617ee83302: Waiting 2024-12-18T00:20:04.0889974Z 3edbde10d404: Waiting 2024-12-18T00:20:04.0890220Z f4038f7524c3: Waiting 2024-12-18T00:20:04.0890472Z 4a51173f7757: Pulling fs layer 2024-12-18T00:20:04.0890755Z ed6a71d41274: Pulling fs layer 2024-12-18T00:20:04.0891041Z 389dc939e2a4: Pulling fs layer 2024-12-18T00:20:04.0891313Z 4a51173f7757: Waiting 2024-12-18T00:20:04.0891571Z 0341eaeed33d: Pulling fs layer 2024-12-18T00:20:04.0891847Z 389dc939e2a4: Waiting 2024-12-18T00:20:04.0892091Z ed6a71d41274: Waiting 2024-12-18T00:20:04.0892343Z ad93cf5205f2: Waiting 2024-12-18T00:20:04.0892589Z b85093661f77: Waiting 2024-12-18T00:20:04.0892845Z 537cdb2d3168: Pulling fs layer 2024-12-18T00:20:04.0893135Z 76d4a0702497: Pulling fs layer 2024-12-18T00:20:04.0893411Z 0341eaeed33d: Waiting 2024-12-18T00:20:04.0893673Z 4f4fb700ef54: Pulling fs layer 2024-12-18T00:20:04.0894042Z 537cdb2d3168: Waiting 2024-12-18T00:20:04.0894300Z 3287a539d61e: Pulling fs layer 2024-12-18T00:20:04.0894580Z 9b763e24a7d4: Pulling fs layer 2024-12-18T00:20:04.0894868Z f6ef3d4c814e: Pulling fs layer 2024-12-18T00:20:04.0895144Z 76d4a0702497: Waiting 2024-12-18T00:20:04.0895401Z 06914b00eb49: Pulling fs layer 2024-12-18T00:20:04.0895676Z 4f4fb700ef54: Waiting 2024-12-18T00:20:04.0895914Z 3287a539d61e: Waiting 2024-12-18T00:20:04.0896157Z 9b763e24a7d4: Waiting 2024-12-18T00:20:04.0896406Z f6ef3d4c814e: Waiting 2024-12-18T00:20:04.0896660Z 724553524224: Pulling fs layer 2024-12-18T00:20:04.0896952Z 47d9fb6dae26: Pulling fs layer 2024-12-18T00:20:04.0897225Z 06914b00eb49: Waiting 2024-12-18T00:20:04.0897485Z acce7b690dcb: Pulling fs layer 2024-12-18T00:20:04.0897761Z 724553524224: Waiting 2024-12-18T00:20:04.0898008Z 47d9fb6dae26: Waiting 2024-12-18T00:20:04.0898253Z d88e9c5e7c70: Waiting 2024-12-18T00:20:04.0898513Z 58dc9f64def7: Pulling fs layer 2024-12-18T00:20:04.0898799Z acce7b690dcb: Waiting 2024-12-18T00:20:04.0899062Z 94dfe5ef1b7e: Pulling fs layer 2024-12-18T00:20:04.0899354Z 40a2c9135b06: Pulling fs layer 2024-12-18T00:20:04.0899627Z 58dc9f64def7: Waiting 2024-12-18T00:20:04.0899889Z 8839d6b0c821: Pulling fs layer 2024-12-18T00:20:04.0900198Z 94dfe5ef1b7e: Waiting 2024-12-18T00:20:04.0900473Z 40a2c9135b06: Waiting 2024-12-18T00:20:04.0900723Z 8839d6b0c821: Waiting 2024-12-18T00:20:04.0900973Z 1c588f0a9e97: Pulling fs layer 2024-12-18T00:20:04.0901268Z 6727e7e27936: Pulling fs layer 2024-12-18T00:20:04.0901543Z 1c588f0a9e97: Waiting 2024-12-18T00:20:04.0901810Z 3a55100d1957: Pulling fs layer 2024-12-18T00:20:04.0902100Z 3d6f160d3351: Pulling fs layer 2024-12-18T00:20:04.0902367Z 6727e7e27936: Waiting 2024-12-18T00:20:04.0902625Z b2a8bb947a87: Pulling fs layer 2024-12-18T00:20:04.0902919Z 69a74b2360f4: Pulling fs layer 2024-12-18T00:20:04.0903207Z f9e12a253d89: Pulling fs layer 2024-12-18T00:20:04.0903487Z b2a8bb947a87: Waiting 2024-12-18T00:20:04.0903728Z 69a74b2360f4: Waiting 2024-12-18T00:20:04.0903991Z 8fdb06d51b96: Pulling fs layer 2024-12-18T00:20:04.0904267Z 3a55100d1957: Waiting 2024-12-18T00:20:04.0904521Z f9e12a253d89: Waiting 2024-12-18T00:20:04.0904765Z 8fdb06d51b96: Waiting 2024-12-18T00:20:04.0905007Z 3d6f160d3351: Waiting 2024-12-18T00:20:04.2263949Z fbcd35dc5bc3: Download complete 2024-12-18T00:20:04.3148761Z db6cdef1932a: Verifying Checksum 2024-12-18T00:20:04.3149143Z db6cdef1932a: Download complete 2024-12-18T00:20:04.3987823Z 56dc85502937: Verifying Checksum 2024-12-18T00:20:04.3988141Z 56dc85502937: Download complete 2024-12-18T00:20:04.4370286Z 80888bc6716f: Verifying Checksum 2024-12-18T00:20:04.4370624Z 80888bc6716f: Download complete 2024-12-18T00:20:04.5123901Z 71bdb1a72c2d: Download complete 2024-12-18T00:20:04.6036890Z 4829486be7c3: Verifying Checksum 2024-12-18T00:20:04.6037222Z 4829486be7c3: Download complete 2024-12-18T00:20:04.6868808Z 1f4e68d7b5e4: Verifying Checksum 2024-12-18T00:20:04.6869159Z 1f4e68d7b5e4: Download complete 2024-12-18T00:20:04.7304016Z c7232af9ae05: Verifying Checksum 2024-12-18T00:20:04.7304367Z c7232af9ae05: Download complete 2024-12-18T00:20:04.8162342Z 622381141745: Download complete 2024-12-18T00:20:04.8963208Z f003d065e31e: Verifying Checksum 2024-12-18T00:20:04.8963875Z f003d065e31e: Download complete 2024-12-18T00:20:05.4844519Z 80888bc6716f: Pull complete 2024-12-18T00:20:05.7665958Z fbcd35dc5bc3: Pull complete 2024-12-18T00:20:06.4891423Z c7232af9ae05: Pull complete 2024-12-18T00:20:06.5063732Z db6cdef1932a: Pull complete 2024-12-18T00:20:06.5226007Z 56dc85502937: Pull complete 2024-12-18T00:20:07.4534594Z cd6ce7116815: Verifying Checksum 2024-12-18T00:20:07.4535031Z cd6ce7116815: Download complete 2024-12-18T00:20:07.5482009Z e843066a8f68: Download complete 2024-12-18T00:20:07.6110730Z bac50ca2930b: Download complete 2024-12-18T00:20:07.6889460Z 57e52c6751f4: Verifying Checksum 2024-12-18T00:20:07.6889840Z 57e52c6751f4: Download complete 2024-12-18T00:20:08.6610091Z a9c5c14d77c0: Verifying Checksum 2024-12-18T00:20:08.6610729Z a9c5c14d77c0: Download complete 2024-12-18T00:20:08.7275663Z b86099b8619a: Verifying Checksum 2024-12-18T00:20:08.7275974Z b86099b8619a: Download complete 2024-12-18T00:20:08.7999774Z 3c8ea8a600ff: Verifying Checksum 2024-12-18T00:20:08.8000095Z 3c8ea8a600ff: Download complete 2024-12-18T00:20:08.8732602Z bebd539a1d59: Verifying Checksum 2024-12-18T00:20:08.8732922Z bebd539a1d59: Download complete 2024-12-18T00:20:08.9491634Z 292b18ec2dab: Verifying Checksum 2024-12-18T00:20:08.9491945Z 292b18ec2dab: Download complete 2024-12-18T00:20:18.2007722Z 30c0ea6140d0: Verifying Checksum 2024-12-18T00:20:18.2008258Z 30c0ea6140d0: Download complete 2024-12-18T00:20:18.2894437Z 50b1f245e9f5: Download complete 2024-12-18T00:20:18.3802507Z ad93cf5205f2: Verifying Checksum 2024-12-18T00:20:18.3803284Z ad93cf5205f2: Download complete 2024-12-18T00:20:18.4459581Z 8d17c59eaf7d: Verifying Checksum 2024-12-18T00:20:18.4459923Z 8d17c59eaf7d: Download complete 2024-12-18T00:20:18.5496606Z b36a8ccf5c8c: Download complete 2024-12-18T00:20:18.6219434Z aa0a3b87e9e3: Verifying Checksum 2024-12-18T00:20:18.6219801Z aa0a3b87e9e3: Download complete 2024-12-18T00:20:19.9056962Z 72a632052c5b: Verifying Checksum 2024-12-18T00:20:19.9057310Z 72a632052c5b: Download complete 2024-12-18T00:20:19.9874558Z ceed6316bc24: Verifying Checksum 2024-12-18T00:20:19.9874908Z ceed6316bc24: Download complete 2024-12-18T00:20:20.0916674Z d0ca0f491f80: Verifying Checksum 2024-12-18T00:20:20.0917083Z d0ca0f491f80: Download complete 2024-12-18T00:20:20.1793842Z cf02c706cd56: Verifying Checksum 2024-12-18T00:20:20.1794327Z cf02c706cd56: Download complete 2024-12-18T00:20:20.2506543Z 16073d3f107d: Verifying Checksum 2024-12-18T00:20:20.2506991Z 16073d3f107d: Download complete 2024-12-18T00:20:20.3430105Z d234faf2f1bf: Verifying Checksum 2024-12-18T00:20:20.3430462Z d234faf2f1bf: Download complete 2024-12-18T00:20:24.7018646Z 3f0ade86af1a: Verifying Checksum 2024-12-18T00:20:24.7019118Z 3f0ade86af1a: Download complete 2024-12-18T00:20:24.7904510Z 034b6a03f5f7: Download complete 2024-12-18T00:20:24.8692003Z 21f8f066852d: Download complete 2024-12-18T00:20:25.2870669Z 9dc1f315a979: Verifying Checksum 2024-12-18T00:20:25.2871000Z 9dc1f315a979: Download complete 2024-12-18T00:20:25.3860795Z cd6a6063178b: Download complete 2024-12-18T00:20:25.4863358Z 2116738f4ea9: Verifying Checksum 2024-12-18T00:20:25.4863941Z 2116738f4ea9: Download complete 2024-12-18T00:20:25.7513388Z d7fa134316d9: Verifying Checksum 2024-12-18T00:20:25.7514013Z d7fa134316d9: Download complete 2024-12-18T00:20:25.8414831Z af03dc3fd15e: Verifying Checksum 2024-12-18T00:20:25.8415482Z af03dc3fd15e: Download complete 2024-12-18T00:20:25.9313916Z fd9de3dc4258: Download complete 2024-12-18T00:20:26.0263802Z b85093661f77: Verifying Checksum 2024-12-18T00:20:26.0264270Z b85093661f77: Download complete 2024-12-18T00:20:30.6182405Z 30c0ea6140d0: Pull complete 2024-12-18T00:20:30.9310684Z 71bdb1a72c2d: Pull complete 2024-12-18T00:20:31.0048218Z 7c373e2d9b7e: Verifying Checksum 2024-12-18T00:20:31.0048762Z 7c373e2d9b7e: Download complete 2024-12-18T00:20:31.1119413Z 56c896077120: Verifying Checksum 2024-12-18T00:20:31.1119718Z 56c896077120: Download complete 2024-12-18T00:20:31.1993932Z d88e9c5e7c70: Verifying Checksum 2024-12-18T00:20:31.1994245Z d88e9c5e7c70: Download complete 2024-12-18T00:20:31.2056931Z 4829486be7c3: Pull complete 2024-12-18T00:20:31.5762345Z 1f4e68d7b5e4: Pull complete 2024-12-18T00:20:31.6936890Z 53617ee83302: Verifying Checksum 2024-12-18T00:20:31.6937282Z 53617ee83302: Download complete 2024-12-18T00:20:31.7719706Z 3edbde10d404: Download complete 2024-12-18T00:20:31.8633628Z f4038f7524c3: Verifying Checksum 2024-12-18T00:20:31.8634046Z f4038f7524c3: Download complete 2024-12-18T00:20:31.9522209Z 4a51173f7757: Download complete 2024-12-18T00:20:32.0377107Z ed6a71d41274: Verifying Checksum 2024-12-18T00:20:32.0377427Z ed6a71d41274: Download complete 2024-12-18T00:20:43.2710028Z 76e3817b4b4c: Verifying Checksum 2024-12-18T00:20:43.2710509Z 76e3817b4b4c: Download complete 2024-12-18T00:20:43.3809094Z 0341eaeed33d: Verifying Checksum 2024-12-18T00:20:43.3809428Z 0341eaeed33d: Download complete 2024-12-18T00:20:43.4962740Z 537cdb2d3168: Verifying Checksum 2024-12-18T00:20:43.4965541Z 537cdb2d3168: Download complete 2024-12-18T00:20:43.5913160Z 76d4a0702497: Verifying Checksum 2024-12-18T00:20:43.5913490Z 76d4a0702497: Download complete 2024-12-18T00:20:43.5972723Z 4f4fb700ef54: Download complete 2024-12-18T00:20:43.7325924Z 3287a539d61e: Download complete 2024-12-18T00:20:43.8407431Z 9b763e24a7d4: Download complete 2024-12-18T00:20:48.4659733Z f6ef3d4c814e: Verifying Checksum 2024-12-18T00:20:48.4660133Z f6ef3d4c814e: Download complete 2024-12-18T00:20:48.5670957Z 06914b00eb49: Verifying Checksum 2024-12-18T00:20:48.5671284Z 06914b00eb49: Download complete 2024-12-18T00:20:48.6742987Z 724553524224: Verifying Checksum 2024-12-18T00:20:48.6743291Z 724553524224: Download complete 2024-12-18T00:20:48.7701214Z 47d9fb6dae26: Verifying Checksum 2024-12-18T00:20:48.7701532Z 47d9fb6dae26: Download complete 2024-12-18T00:20:48.8841776Z acce7b690dcb: Download complete 2024-12-18T00:20:49.0715859Z 58dc9f64def7: Verifying Checksum 2024-12-18T00:20:49.0716196Z 58dc9f64def7: Download complete 2024-12-18T00:20:49.1490514Z 94dfe5ef1b7e: Verifying Checksum 2024-12-18T00:20:49.1490846Z 94dfe5ef1b7e: Download complete 2024-12-18T00:20:50.4187932Z 40a2c9135b06: Verifying Checksum 2024-12-18T00:20:50.4188371Z 40a2c9135b06: Download complete 2024-12-18T00:20:50.5036925Z 8839d6b0c821: Verifying Checksum 2024-12-18T00:20:50.5037235Z 8839d6b0c821: Download complete 2024-12-18T00:20:57.5527355Z 7c373e2d9b7e: Pull complete 2024-12-18T00:20:58.0555748Z 622381141745: Pull complete 2024-12-18T00:20:58.5274083Z f003d065e31e: Pull complete 2024-12-18T00:21:06.6670514Z cd6ce7116815: Pull complete 2024-12-18T00:21:07.1491652Z e843066a8f68: Pull complete 2024-12-18T00:21:07.6741419Z bac50ca2930b: Pull complete 2024-12-18T00:21:08.1853422Z 57e52c6751f4: Pull complete 2024-12-18T00:21:10.7754030Z a9c5c14d77c0: Pull complete 2024-12-18T00:21:11.1882881Z b86099b8619a: Pull complete 2024-12-18T00:21:11.6349382Z 3c8ea8a600ff: Pull complete 2024-12-18T00:21:12.1221148Z bebd539a1d59: Pull complete 2024-12-18T00:21:12.6179087Z 292b18ec2dab: Pull complete 2024-12-18T00:21:18.2389247Z 1c588f0a9e97: Verifying Checksum 2024-12-18T00:21:18.2389651Z 1c588f0a9e97: Download complete 2024-12-18T00:21:18.3704792Z 6727e7e27936: Verifying Checksum 2024-12-18T00:21:18.3705128Z 6727e7e27936: Download complete 2024-12-18T00:21:18.4705680Z 3a55100d1957: Verifying Checksum 2024-12-18T00:21:18.4706017Z 3a55100d1957: Download complete 2024-12-18T00:21:25.5580055Z 3d6f160d3351: Verifying Checksum 2024-12-18T00:21:25.5580410Z 3d6f160d3351: Download complete 2024-12-18T00:21:25.6682332Z b2a8bb947a87: Verifying Checksum 2024-12-18T00:21:25.6682684Z b2a8bb947a87: Download complete 2024-12-18T00:21:25.7788415Z 69a74b2360f4: Download complete 2024-12-18T00:21:27.0066628Z f9e12a253d89: Verifying Checksum 2024-12-18T00:21:27.0066981Z f9e12a253d89: Download complete 2024-12-18T00:21:27.1004483Z 8fdb06d51b96: Download complete 2024-12-18T00:21:46.9461537Z 389dc939e2a4: Verifying Checksum 2024-12-18T00:21:46.9461917Z 389dc939e2a4: Download complete 2024-12-18T00:21:51.9963115Z ab6ef6c2ac4d: Verifying Checksum 2024-12-18T00:21:51.9963738Z ab6ef6c2ac4d: Download complete 2024-12-18T00:22:05.0054175Z 76e3817b4b4c: Pull complete 2024-12-18T00:22:05.4073504Z 50b1f245e9f5: Pull complete 2024-12-18T00:22:05.8603185Z ad93cf5205f2: Pull complete 2024-12-18T00:22:06.2508537Z 8d17c59eaf7d: Pull complete 2024-12-18T00:22:06.7031642Z b36a8ccf5c8c: Pull complete 2024-12-18T00:22:07.1804385Z aa0a3b87e9e3: Pull complete 2024-12-18T00:22:09.8885699Z 72a632052c5b: Pull complete 2024-12-18T00:22:10.3679017Z ceed6316bc24: Pull complete 2024-12-18T00:22:10.8621675Z d0ca0f491f80: Pull complete 2024-12-18T00:22:11.1767629Z cf02c706cd56: Pull complete 2024-12-18T00:22:11.5598319Z 16073d3f107d: Pull complete 2024-12-18T00:22:12.0146144Z d234faf2f1bf: Pull complete 2024-12-18T00:22:21.3900741Z 3f0ade86af1a: Pull complete 2024-12-18T00:22:21.8417025Z 034b6a03f5f7: Pull complete 2024-12-18T00:22:22.2192258Z 21f8f066852d: Pull complete 2024-12-18T00:22:23.2002416Z 9dc1f315a979: Pull complete 2024-12-18T00:22:23.5939311Z cd6a6063178b: Pull complete 2024-12-18T00:22:24.0603403Z 2116738f4ea9: Pull complete 2024-12-18T00:22:24.7801100Z d7fa134316d9: Pull complete 2024-12-18T00:22:25.2442741Z af03dc3fd15e: Pull complete 2024-12-18T00:22:26.0724212Z fd9de3dc4258: Pull complete 2024-12-18T00:22:26.4528470Z b85093661f77: Pull complete 2024-12-18T00:22:57.4634699Z ab6ef6c2ac4d: Pull complete 2024-12-18T00:22:57.7997101Z 56c896077120: Pull complete 2024-12-18T00:22:58.3332057Z d88e9c5e7c70: Pull complete 2024-12-18T00:23:01.3510012Z 53617ee83302: Pull complete 2024-12-18T00:23:01.5889188Z 3edbde10d404: Pull complete 2024-12-18T00:23:01.9351578Z f4038f7524c3: Pull complete 2024-12-18T00:23:02.7904876Z 4a51173f7757: Pull complete 2024-12-18T00:23:03.2977881Z ed6a71d41274: Pull complete 2024-12-18T00:24:12.8339449Z 389dc939e2a4: Pull complete 2024-12-18T00:24:13.3247142Z 0341eaeed33d: Pull complete 2024-12-18T00:24:13.7910276Z 537cdb2d3168: Pull complete 2024-12-18T00:24:14.6935326Z 76d4a0702497: Pull complete 2024-12-18T00:24:15.1588741Z 4f4fb700ef54: Pull complete 2024-12-18T00:24:15.6158782Z 3287a539d61e: Pull complete 2024-12-18T00:24:16.0112211Z 9b763e24a7d4: Pull complete 2024-12-18T00:24:23.7499115Z f6ef3d4c814e: Pull complete 2024-12-18T00:24:24.1544148Z 06914b00eb49: Pull complete 2024-12-18T00:24:24.5513537Z 724553524224: Pull complete 2024-12-18T00:24:24.8261109Z 47d9fb6dae26: Pull complete 2024-12-18T00:24:25.3305935Z acce7b690dcb: Pull complete 2024-12-18T00:24:26.8942937Z 58dc9f64def7: Pull complete 2024-12-18T00:24:27.2759914Z 94dfe5ef1b7e: Pull complete 2024-12-18T00:24:29.2672601Z 40a2c9135b06: Pull complete 2024-12-18T00:24:29.7093513Z 8839d6b0c821: Pull complete 2024-12-18T00:24:42.4009773Z 1c588f0a9e97: Pull complete 2024-12-18T00:24:42.7508798Z 6727e7e27936: Pull complete 2024-12-18T00:24:43.2759731Z 3a55100d1957: Pull complete 2024-12-18T00:24:46.4075873Z 3d6f160d3351: Pull complete 2024-12-18T00:24:46.8459648Z b2a8bb947a87: Pull complete 2024-12-18T00:24:47.3392251Z 69a74b2360f4: Pull complete 2024-12-18T00:24:48.1806393Z f9e12a253d89: Pull complete 2024-12-18T00:24:48.5731874Z 8fdb06d51b96: Pull complete 2024-12-18T00:24:50.4935030Z Digest: sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb 2024-12-18T00:24:50.5964627Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:24:50.6365227Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:24:50.6427032Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:24:50.6427972Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T00:24:50.6440151Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:24:50.6440539Z env: 2024-12-18T00:24:50.6440769Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:24:50.6441048Z ##[endgroup] 2024-12-18T00:24:50.6708949Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@release/2.6 2024-12-18T00:24:50.6709393Z with: 2024-12-18T00:24:50.6709629Z driver-version: 550.54.15 2024-12-18T00:24:50.6709897Z env: 2024-12-18T00:24:50.6710127Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:24:50.6710395Z ##[endgroup] 2024-12-18T00:24:50.6870842Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-12-18T00:24:50.6871244Z with: 2024-12-18T00:24:50.6871463Z timeout_minutes: 10 2024-12-18T00:24:50.6871718Z max_attempts: 3 2024-12-18T00:24:50.6895411Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-12-18T00:24:50.6919075Z retry_wait_seconds: 10 2024-12-18T00:24:50.6919356Z polling_interval_seconds: 1 2024-12-18T00:24:50.6919651Z warning_on_retry: true 2024-12-18T00:24:50.6919921Z continue_on_error: false 2024-12-18T00:24:50.6920176Z env: 2024-12-18T00:24:50.6920393Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:24:50.6920670Z DRIVER_VERSION: 550.54.15 2024-12-18T00:24:50.6920938Z ##[endgroup] 2024-12-18T00:24:50.8578045Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-12-18T00:24:50.8579069Z + pre_install_nvidia_driver_amzn2 2024-12-18T00:24:50.8582719Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-12-18T00:24:51.1936224Z No match for argument: nvidia-driver-latest-dkms 2024-12-18T00:24:51.1936716Z No packages marked for removal. 2024-12-18T00:24:51.2001075Z Dependencies resolved. 2024-12-18T00:24:51.2011093Z Nothing to do. 2024-12-18T00:24:51.2011470Z Complete! 2024-12-18T00:24:51.3248902Z + install_nvidia_driver_common 2024-12-18T00:24:51.3252943Z + echo 'Before installing NVIDIA driver' 2024-12-18T00:24:51.3253498Z Before installing NVIDIA driver 2024-12-18T00:24:51.3255741Z + lspci 2024-12-18T00:24:51.3382054Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:24:51.3382644Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:24:51.3383813Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:24:51.3384894Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:24:51.3385894Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:24:51.3387010Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:24:51.3388035Z 00:1b.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.3388937Z 00:1c.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.3390187Z 00:1d.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.3391090Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.3392088Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:24:51.3392651Z + lsmod 2024-12-18T00:24:51.3433869Z Module Size Used by 2024-12-18T00:24:51.3434335Z ib_core 454656 0 2024-12-18T00:24:51.3434649Z veth 36864 0 2024-12-18T00:24:51.3434932Z nvidia_modeset 1351680 0 2024-12-18T00:24:51.3435231Z video 65536 1 nvidia_modeset 2024-12-18T00:24:51.3436811Z wmi 36864 1 video 2024-12-18T00:24:51.3437104Z nvidia_uvm 4706304 0 2024-12-18T00:24:51.3437464Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-12-18T00:24:51.3437825Z drm 602112 1 nvidia 2024-12-18T00:24:51.3438154Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:24:51.3438543Z backlight 24576 3 video,drm,nvidia_modeset 2024-12-18T00:24:51.3438913Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:24:51.3439225Z xt_conntrack 16384 1 2024-12-18T00:24:51.3439504Z nft_chain_nat 16384 3 2024-12-18T00:24:51.3439782Z xt_MASQUERADE 20480 1 2024-12-18T00:24:51.3440092Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:24:51.3440441Z nf_conntrack_netlink 57344 0 2024-12-18T00:24:51.3440866Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:24:51.3441322Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:24:51.3441660Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:24:51.3441970Z xfrm_user 57344 1 2024-12-18T00:24:51.3442251Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:24:51.3442560Z xt_addrtype 16384 2 2024-12-18T00:24:51.3442834Z nft_compat 20480 4 2024-12-18T00:24:51.3443239Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:24:51.3443676Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:24:51.3444060Z br_netfilter 36864 0 2024-12-18T00:24:51.3444356Z bridge 323584 1 br_netfilter 2024-12-18T00:24:51.3444663Z stp 16384 1 bridge 2024-12-18T00:24:51.3444960Z llc 16384 2 bridge,stp 2024-12-18T00:24:51.3445254Z overlay 167936 0 2024-12-18T00:24:51.3445511Z tls 114688 0 2024-12-18T00:24:51.3445774Z nls_ascii 16384 1 2024-12-18T00:24:51.3446039Z sunrpc 692224 1 2024-12-18T00:24:51.3446307Z nls_cp437 20480 1 2024-12-18T00:24:51.3446564Z vfat 24576 1 2024-12-18T00:24:51.3446820Z fat 86016 1 vfat 2024-12-18T00:24:51.3447093Z ena 167936 0 2024-12-18T00:24:51.3447349Z i8042 45056 0 2024-12-18T00:24:51.3447618Z serio 28672 3 i8042 2024-12-18T00:24:51.3447900Z button 24576 0 2024-12-18T00:24:51.3448164Z ghash_clmulni_intel 16384 0 2024-12-18T00:24:51.3448441Z sch_fq_codel 20480 33 2024-12-18T00:24:51.3448711Z dm_mod 188416 0 2024-12-18T00:24:51.3448971Z fuse 163840 1 2024-12-18T00:24:51.3449224Z loop 36864 0 2024-12-18T00:24:51.3449489Z configfs 57344 1 2024-12-18T00:24:51.3449757Z dax 45056 1 dm_mod 2024-12-18T00:24:51.3450044Z dmi_sysfs 20480 0 2024-12-18T00:24:51.3450311Z crc32_pclmul 16384 0 2024-12-18T00:24:51.3450586Z crc32c_intel 24576 0 2024-12-18T00:24:51.3450857Z efivarfs 24576 1 2024-12-18T00:24:51.3451123Z + modinfo nvidia 2024-12-18T00:24:51.3456332Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:24:51.3456829Z alias: char-major-195-* 2024-12-18T00:24:51.3457231Z version: 550.54.15 2024-12-18T00:24:51.3457494Z supported: external 2024-12-18T00:24:51.3457757Z license: NVIDIA 2024-12-18T00:24:51.3458045Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:24:51.3458405Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:24:51.3458743Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:24:51.3459079Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:24:51.3459437Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:24:51.3459792Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:24:51.3460122Z depends: i2c-core,drm 2024-12-18T00:24:51.3460486Z retpoline: Y 2024-12-18T00:24:51.3460724Z name: nvidia 2024-12-18T00:24:51.3461105Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:24:51.3461596Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:24:51.3462069Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:24:51.3462535Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:24:51.3462885Z parm: NVreg_RmLogonRC:int 2024-12-18T00:24:51.3463202Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:24:51.3463710Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:24:51.3464035Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:24:51.3464357Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:24:51.3464741Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:24:51.3465145Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:24:51.3465521Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:24:51.3465844Z parm: NVreg_EnableMSI:int 2024-12-18T00:24:51.3466152Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:24:51.3466491Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:24:51.3466877Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:24:51.3467304Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:24:51.3467702Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:24:51.3468136Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:24:51.3468573Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:24:51.3469016Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:24:51.3469449Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:24:51.3469807Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:24:51.3470197Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:24:51.3470599Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:24:51.3470969Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:24:51.3471311Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:24:51.3471658Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:24:51.3472007Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:24:51.3472345Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:24:51.3472715Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:24:51.3473099Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:24:51.3473443Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:24:51.3473798Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:24:51.3474163Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:24:51.3474523Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:24:51.3474886Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:24:51.3475235Z parm: NVreg_RmMsg:charp 2024-12-18T00:24:51.3475538Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:24:51.3475892Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:24:51.3476239Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:24:51.3476576Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:24:51.3476927Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:24:51.3477417Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:24:51.3477754Z parm: rm_firmware_active:charp 2024-12-18T00:24:51.3478069Z + HAS_NVIDIA_DRIVER=0 2024-12-18T00:24:51.3478333Z ++ command -v nvidia-smi 2024-12-18T00:24:51.3478610Z + '[' -x /usr/bin/nvidia-smi ']' 2024-12-18T00:24:51.3478880Z + set +e 2024-12-18T00:24:51.3479216Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-12-18T00:24:51.3772060Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-12-18T00:24:51.3772375Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:24:51.3773178Z + '[' 0 -ne 0 ']' 2024-12-18T00:24:51.3773413Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-12-18T00:24:51.3773866Z + HAS_NVIDIA_DRIVER=1 2024-12-18T00:24:51.3774316Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-12-18T00:24:51.3774797Z + set -e 2024-12-18T00:24:51.3775009Z + '[' 1 -eq 0 ']' 2024-12-18T00:24:51.3775412Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-12-18T00:24:51.3776043Z + post_install_nvidia_driver_common 2024-12-18T00:24:51.3780143Z + sudo modprobe nvidia 2024-12-18T00:24:51.5259857Z + echo 'After installing NVIDIA driver' 2024-12-18T00:24:51.5260186Z + lspci 2024-12-18T00:24:51.5260425Z After installing NVIDIA driver 2024-12-18T00:24:51.5384581Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-12-18T00:24:51.5385754Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-12-18T00:24:51.5386833Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-12-18T00:24:51.5387833Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-12-18T00:24:51.5388781Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-12-18T00:24:51.5389801Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-12-18T00:24:51.5390734Z 00:1b.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.5391586Z 00:1c.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.5392405Z 00:1d.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.5393033Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-12-18T00:24:51.5393533Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-12-18T00:24:51.5393957Z + lsmod 2024-12-18T00:24:51.5422206Z Module Size Used by 2024-12-18T00:24:51.5422529Z ib_core 454656 0 2024-12-18T00:24:51.5422915Z veth 36864 0 2024-12-18T00:24:51.5423317Z nvidia_modeset 1351680 0 2024-12-18T00:24:51.5423777Z video 65536 1 nvidia_modeset 2024-12-18T00:24:51.5424151Z wmi 36864 1 video 2024-12-18T00:24:51.5424442Z nvidia_uvm 4706304 0 2024-12-18T00:24:51.5424761Z nvidia 54071296 19 nvidia_uvm,nvidia_modeset 2024-12-18T00:24:51.5425110Z drm 602112 1 nvidia 2024-12-18T00:24:51.5425439Z drm_panel_orientation_quirks 32768 1 drm 2024-12-18T00:24:51.5425823Z backlight 24576 3 video,drm,nvidia_modeset 2024-12-18T00:24:51.5426193Z i2c_core 106496 2 nvidia,drm 2024-12-18T00:24:51.5426500Z xt_conntrack 16384 1 2024-12-18T00:24:51.5426772Z nft_chain_nat 16384 3 2024-12-18T00:24:51.5427051Z xt_MASQUERADE 20480 1 2024-12-18T00:24:51.5427375Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-12-18T00:24:51.5427729Z nf_conntrack_netlink 57344 0 2024-12-18T00:24:51.5428152Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-12-18T00:24:51.5428614Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-12-18T00:24:51.5428947Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-12-18T00:24:51.5429266Z xfrm_user 57344 1 2024-12-18T00:24:51.5429563Z xfrm_algo 16384 1 xfrm_user 2024-12-18T00:24:51.5429879Z xt_addrtype 16384 2 2024-12-18T00:24:51.5430351Z nft_compat 20480 4 2024-12-18T00:24:51.5430689Z nf_tables 311296 57 nft_compat,nft_chain_nat 2024-12-18T00:24:51.5431131Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-12-18T00:24:51.5431527Z br_netfilter 36864 0 2024-12-18T00:24:51.5431834Z bridge 323584 1 br_netfilter 2024-12-18T00:24:51.5432141Z stp 16384 1 bridge 2024-12-18T00:24:51.5432444Z llc 16384 2 bridge,stp 2024-12-18T00:24:51.5432748Z overlay 167936 0 2024-12-18T00:24:51.5433021Z tls 114688 0 2024-12-18T00:24:51.5433459Z nls_ascii 16384 1 2024-12-18T00:24:51.5433732Z sunrpc 692224 1 2024-12-18T00:24:51.5434013Z nls_cp437 20480 1 2024-12-18T00:24:51.5434280Z vfat 24576 1 2024-12-18T00:24:51.5434552Z fat 86016 1 vfat 2024-12-18T00:24:51.5434840Z ena 167936 0 2024-12-18T00:24:51.5435104Z i8042 45056 0 2024-12-18T00:24:51.5435378Z serio 28672 3 i8042 2024-12-18T00:24:51.5435674Z button 24576 0 2024-12-18T00:24:51.5435955Z ghash_clmulni_intel 16384 0 2024-12-18T00:24:51.5436249Z sch_fq_codel 20480 33 2024-12-18T00:24:51.5436519Z dm_mod 188416 0 2024-12-18T00:24:51.5436787Z fuse 163840 1 2024-12-18T00:24:51.5437062Z loop 36864 0 2024-12-18T00:24:51.5437333Z configfs 57344 1 2024-12-18T00:24:51.5437609Z dax 45056 1 dm_mod 2024-12-18T00:24:51.5437895Z dmi_sysfs 20480 0 2024-12-18T00:24:51.5438173Z crc32_pclmul 16384 0 2024-12-18T00:24:51.5438448Z crc32c_intel 24576 0 2024-12-18T00:24:51.5438721Z efivarfs 24576 1 2024-12-18T00:24:51.5438996Z + modinfo nvidia 2024-12-18T00:24:51.5446002Z filename: /lib/modules/6.1.109-118.189.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-12-18T00:24:51.5446642Z alias: char-major-195-* 2024-12-18T00:24:51.5446937Z version: 550.54.15 2024-12-18T00:24:51.5447207Z supported: external 2024-12-18T00:24:51.5447469Z license: NVIDIA 2024-12-18T00:24:51.5447758Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-12-18T00:24:51.5448116Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-12-18T00:24:51.5448462Z srcversion: 833721318DA517F0C2FEC97 2024-12-18T00:24:51.5448803Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-12-18T00:24:51.5449167Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-12-18T00:24:51.5449528Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-12-18T00:24:51.5449863Z depends: i2c-core,drm 2024-12-18T00:24:51.5450149Z retpoline: Y 2024-12-18T00:24:51.5450396Z name: nvidia 2024-12-18T00:24:51.5450784Z vermagic: 6.1.109-118.189.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-12-18T00:24:51.5451288Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-12-18T00:24:51.5451759Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-12-18T00:24:51.5452231Z parm: NVreg_ResmanDebugLevel:int 2024-12-18T00:24:51.5452592Z parm: NVreg_RmLogonRC:int 2024-12-18T00:24:51.5452917Z parm: NVreg_ModifyDeviceFiles:int 2024-12-18T00:24:51.5453258Z parm: NVreg_DeviceFileUID:int 2024-12-18T00:24:51.5453577Z parm: NVreg_DeviceFileGID:int 2024-12-18T00:24:51.5453913Z parm: NVreg_DeviceFileMode:int 2024-12-18T00:24:51.5454305Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-12-18T00:24:51.5454720Z parm: NVreg_UsePageAttributeTable:int 2024-12-18T00:24:51.5455082Z parm: NVreg_EnablePCIeGen3:int 2024-12-18T00:24:51.5455401Z parm: NVreg_EnableMSI:int 2024-12-18T00:24:51.5455716Z parm: NVreg_TCEBypassMode:int 2024-12-18T00:24:51.5456058Z parm: NVreg_EnableStreamMemOPs:int 2024-12-18T00:24:51.5456568Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-12-18T00:24:51.5456999Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-12-18T00:24:51.5457406Z parm: NVreg_EnableS0ixPowerManagement:int 2024-12-18T00:24:51.5457846Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-12-18T00:24:51.5458281Z parm: NVreg_DynamicPowerManagement:int 2024-12-18T00:24:51.5458728Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-12-18T00:24:51.5459164Z parm: NVreg_EnableGpuFirmware:int 2024-12-18T00:24:51.5459529Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-12-18T00:24:51.5459918Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-12-18T00:24:51.5460403Z parm: NVreg_EnableUserNUMAManagement:int 2024-12-18T00:24:51.5460772Z parm: NVreg_MemoryPoolSize:int 2024-12-18T00:24:51.5461117Z parm: NVreg_KMallocHeapMaxSize:int 2024-12-18T00:24:51.5461471Z parm: NVreg_VMallocHeapMaxSize:int 2024-12-18T00:24:51.5461840Z parm: NVreg_IgnoreMMIOCheck:int 2024-12-18T00:24:51.5462175Z parm: NVreg_NvLinkDisable:int 2024-12-18T00:24:51.5462542Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-12-18T00:24:51.5462929Z parm: NVreg_RegisterPCIDriver:int 2024-12-18T00:24:51.5463281Z parm: NVreg_EnableResizableBar:int 2024-12-18T00:24:51.5463810Z parm: NVreg_EnableDbgBreakpoint:int 2024-12-18T00:24:51.5464181Z parm: NVreg_EnableNonblockingOpen:int 2024-12-18T00:24:51.5464538Z parm: NVreg_RegistryDwords:charp 2024-12-18T00:24:51.5464908Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-12-18T00:24:51.5465267Z parm: NVreg_RmMsg:charp 2024-12-18T00:24:51.5465581Z parm: NVreg_GpuBlacklist:charp 2024-12-18T00:24:51.5465931Z parm: NVreg_TemporaryFilePath:charp 2024-12-18T00:24:51.5466275Z parm: NVreg_ExcludedGpus:charp 2024-12-18T00:24:51.5466614Z parm: NVreg_DmaRemapPeerMmio:int 2024-12-18T00:24:51.5466976Z parm: NVreg_RmNvlinkBandwidth:charp 2024-12-18T00:24:51.5467329Z parm: NVreg_ImexChannelCount:int 2024-12-18T00:24:51.5467663Z parm: rm_firmware_active:charp 2024-12-18T00:24:51.5467958Z + set +e 2024-12-18T00:24:51.5468172Z + nvidia-smi 2024-12-18T00:24:51.5956884Z Wed Dec 18 00:24:51 2024 2024-12-18T00:24:51.5957948Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:24:51.5959157Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:24:51.5960167Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:24:51.5961205Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:24:51.5962294Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:24:51.5962876Z | | | MIG M. | 2024-12-18T00:24:51.5963300Z |=========================================+========================+======================| 2024-12-18T00:24:51.6198085Z | 0 NVIDIA A10G On | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:24:51.6198759Z | 0% 21C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:24:51.6199329Z | | | N/A | 2024-12-18T00:24:51.6199811Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:24:51.6436678Z | 1 NVIDIA A10G On | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:24:51.6437344Z | 0% 20C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:24:51.6437895Z | | | N/A | 2024-12-18T00:24:51.6438573Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:24:51.6671473Z | 2 NVIDIA A10G On | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:24:51.6672100Z | 0% 21C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:24:51.6672652Z | | | N/A | 2024-12-18T00:24:51.6673079Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:24:51.6899566Z | 3 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:24:51.6900394Z | 0% 21C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:24:51.6900909Z | | | N/A | 2024-12-18T00:24:51.6901340Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:24:51.7033415Z 2024-12-18T00:24:51.7034012Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:24:51.7034624Z | Processes: | 2024-12-18T00:24:51.7035129Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:24:51.7035580Z | ID ID Usage | 2024-12-18T00:24:51.7035954Z |=========================================================================================| 2024-12-18T00:24:51.7066336Z | No running processes found | 2024-12-18T00:24:51.7066982Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:24:52.9059252Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-12-18T00:24:52.9263964Z NVIDIA A10G 2024-12-18T00:24:52.9329303Z + NVIDIA_SMI_STATUS=0 2024-12-18T00:24:52.9329620Z + '[' 0 -eq 0 ']' 2024-12-18T00:24:52.9329981Z + echo 'INFO: Ignoring allowed status 0' 2024-12-18T00:24:52.9330366Z + set -e 2024-12-18T00:24:52.9330656Z INFO: Ignoring allowed status 0 2024-12-18T00:24:52.9340048Z == Installing nvidia container toolkit for amzn2023 == 2024-12-18T00:24:52.9344167Z + sudo yum install -y yum-utils 2024-12-18T00:24:53.3881047Z Last metadata expiration check: 0:31:11 ago on Tue Dec 17 23:53:42 2024. 2024-12-18T00:24:53.4114556Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-12-18T00:24:53.4453557Z Dependencies resolved. 2024-12-18T00:24:53.4594440Z Nothing to do. 2024-12-18T00:24:53.4594917Z Complete! 2024-12-18T00:24:53.5836511Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-12-18T00:24:53.5837106Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:24:53.5838007Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:24:53.8137714Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-12-18T00:24:53.8909746Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2 2024-12-18T00:24:54.3777955Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-12-18T00:24:54.4010092Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-12-18T00:24:54.4016103Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed. 2024-12-18T00:24:54.4348589Z Dependencies resolved. 2024-12-18T00:24:54.4488655Z Nothing to do. 2024-12-18T00:24:54.4489115Z Complete! 2024-12-18T00:24:54.5791824Z + sudo systemctl restart docker 2024-12-18T00:26:47.4336417Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-12-18T00:26:47.4816765Z Wed Dec 18 00:26:47 2024 2024-12-18T00:26:47.4817550Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:26:47.4818586Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-12-18T00:26:47.4819575Z |-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:26:47.4820593Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-12-18T00:26:47.4821579Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-12-18T00:26:47.4822296Z | | | MIG M. | 2024-12-18T00:26:47.4822657Z |=========================================+========================+======================| 2024-12-18T00:26:47.5017174Z | 0 NVIDIA A10G On | 00000000:00:1B.0 Off | 0 | 2024-12-18T00:26:47.5017638Z | 0% 22C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:26:47.5018053Z | | | N/A | 2024-12-18T00:26:47.5018474Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:26:47.5213761Z | 1 NVIDIA A10G On | 00000000:00:1C.0 Off | 0 | 2024-12-18T00:26:47.5214250Z | 0% 21C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:26:47.5214664Z | | | N/A | 2024-12-18T00:26:47.5215098Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:26:47.5406040Z | 2 NVIDIA A10G On | 00000000:00:1D.0 Off | 0 | 2024-12-18T00:26:47.5406582Z | 0% 21C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:26:47.5406992Z | | | N/A | 2024-12-18T00:26:47.5407404Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:26:47.5598489Z | 3 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-12-18T00:26:47.5599393Z | 0% 22C P8 15W / 300W | 0MiB / 23028MiB | 0% Default | 2024-12-18T00:26:47.5600173Z | | | N/A | 2024-12-18T00:26:47.5601002Z +-----------------------------------------+------------------------+----------------------+ 2024-12-18T00:26:47.5746550Z 2024-12-18T00:26:47.5747411Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:26:47.5748293Z | Processes: | 2024-12-18T00:26:47.5749231Z | GPU GI CI PID Type Process name GPU Memory | 2024-12-18T00:26:47.5750100Z | ID ID Usage | 2024-12-18T00:26:47.5750818Z |=========================================================================================| 2024-12-18T00:26:47.5794370Z | No running processes found | 2024-12-18T00:26:47.5795354Z +-----------------------------------------------------------------------------------------+ 2024-12-18T00:26:48.9523024Z Command completed after 1 attempt(s). 2024-12-18T00:26:48.9618251Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:26:48.9618826Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-12-18T00:26:48.9619317Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-12-18T00:26:48.9619794Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-12-18T00:26:48.9632407Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:26:48.9632783Z env: 2024-12-18T00:26:48.9633012Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:26:48.9633358Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:26:48.9633717Z ##[endgroup] 2024-12-18T00:26:49.2065127Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:26:49.2253880Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-12-18T00:26:49.2258961Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-12-18T00:26:49.3604961Z Prepare all required actions 2024-12-18T00:26:49.3606087Z Getting action download info 2024-12-18T00:26:49.5425169Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T00:26:49.8315460Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T00:26:50.1958683Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T00:26:50.1959043Z with: 2024-12-18T00:26:50.1959314Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:26:50.1959655Z s3-bucket: gha-artifacts 2024-12-18T00:26:50.1959917Z env: 2024-12-18T00:26:50.1960133Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:26:50.1960478Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:26:50.1960851Z ##[endgroup] 2024-12-18T00:26:50.1998992Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:26:50.1999334Z with: 2024-12-18T00:26:50.1999631Z name: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:26:50.1999980Z s3-bucket: gha-artifacts 2024-12-18T00:26:50.2000252Z region: us-east-1 2024-12-18T00:26:50.2000566Z env: 2024-12-18T00:26:50.2000829Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:26:50.2001166Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:26:50.2001521Z ##[endgroup] 2024-12-18T00:26:50.6823166Z (node:191308) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:26:50.6823636Z 2024-12-18T00:26:50.6823837Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:26:50.6824359Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:26:50.6824893Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:26:50.7834953Z Found 1 objects with prefix pytorch/pytorch/12383255690/linux-focal-cuda12.4-py3.10-gcc9-sm86/ 2024-12-18T00:26:50.7835830Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:27:13.3384817Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T00:27:13.3390654Z Artifact download has finished successfully 2024-12-18T00:27:13.3753560Z ##[group]Run unzip -o artifacts.zip 2024-12-18T00:27:13.3753909Z unzip -o artifacts.zip 2024-12-18T00:27:13.3764878Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:13.3765275Z env: 2024-12-18T00:27:13.3765507Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:13.3765861Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:13.3766232Z ##[endgroup] 2024-12-18T00:27:13.3815427Z Archive: artifacts.zip 2024-12-18T00:27:13.3817018Z creating: dist/ 2024-12-18T00:27:15.5813854Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:27:15.5814377Z creating: build/custom_test_artifacts/ 2024-12-18T00:27:15.5814783Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T00:27:15.5815278Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T00:27:15.5815854Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:27:15.5823584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:27:15.5824238Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T00:27:15.5824884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:27:15.5825572Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:27:15.5826234Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:27:15.5828947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:27:15.5830517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:27:15.5831220Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:27:15.5831910Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:27:15.5834456Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:27:15.5835723Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:27:15.5837875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:27:15.5838679Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:27:15.5840588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:27:15.5841892Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:27:15.5842621Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:27:15.5843405Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:27:15.5885466Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:27:15.5926948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:27:15.5927962Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:27:15.5975116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:27:15.5976114Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:27:15.5977685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:27:15.5978762Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:27:15.5979761Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:27:15.5980899Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:27:15.5981873Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:27:15.5982864Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:27:15.5983831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:27:15.5984741Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:27:15.5985622Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:27:15.5986478Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:27:15.5987374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:27:15.5988507Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:27:15.5990218Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:27:15.6065039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:27:15.6138137Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:27:15.6139308Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:27:15.6140690Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:27:15.6141888Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:27:15.6143127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T00:27:15.6144427Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T00:27:15.6145845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T00:27:15.6147461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T00:27:15.6148903Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T00:27:15.6149629Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T00:27:15.6150375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T00:27:15.6151132Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T00:27:15.6151885Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T00:27:15.6152630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T00:27:15.6153377Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T00:27:15.6170636Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T00:27:15.6315136Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T00:27:15.6315843Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T00:27:15.6316603Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T00:27:15.6317454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T00:27:15.6318263Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T00:27:15.6319046Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T00:27:15.6319865Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T00:27:15.6320668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T00:27:15.6321470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T00:27:15.6322259Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T00:27:15.6323127Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T00:27:15.6344679Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T00:27:15.6428396Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T00:27:15.6429738Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:27:15.6431560Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:27:15.6433149Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T00:27:15.6434415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T00:27:15.6435656Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T00:27:15.6436898Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-12-18T00:27:15.6438070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T00:27:15.6438924Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T00:27:15.6439475Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T00:27:15.6560867Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T00:27:15.6624189Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T00:27:15.6625193Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T00:27:15.6626140Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T00:27:15.6627243Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:27:15.6632318Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:27:15.6632967Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T00:27:15.6633606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:27:15.6634283Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:27:15.6634938Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:27:15.6637234Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:27:15.6638792Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:27:15.6639482Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:27:15.6640160Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:27:15.6642712Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:27:15.6644299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:27:15.6646633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:27:15.6647955Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:27:15.6650097Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:27:15.6651247Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:27:15.6651976Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:27:15.6652664Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:27:15.6695078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:27:15.6735823Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:27:15.6736829Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:27:15.6784521Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:27:15.6785668Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:27:15.6786784Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:27:15.6787809Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:27:15.6788796Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:27:15.6789810Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:27:15.6790774Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:27:15.6791741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:27:15.6793260Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:27:15.6794194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:27:15.6795057Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:27:15.6795910Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:27:15.6796773Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:27:15.6797757Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:27:15.6799153Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:27:15.6873533Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:27:15.6947080Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:27:15.6947913Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:27:15.6948716Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:27:15.6949933Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:27:15.6951188Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T00:27:15.6952488Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T00:27:15.6953959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T00:27:15.6955624Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T00:27:15.6957220Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T00:27:15.6958695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T00:27:15.6959689Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T00:27:15.6960479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T00:27:15.6961250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T00:27:15.6962030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T00:27:15.6962797Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T00:27:15.6980116Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T00:27:15.7044264Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T00:27:15.7045096Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:27:15.7046030Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:27:15.7046731Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T00:27:15.7047375Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T00:27:15.7049072Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T00:27:15.7049699Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-12-18T00:27:15.7052486Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T00:27:15.7053331Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T00:27:15.7054193Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T00:27:15.7104932Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T00:27:15.7105988Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T00:27:15.7107033Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T00:27:15.7108255Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T00:27:15.7113334Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T00:27:15.7114032Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T00:27:15.7114719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T00:27:15.7115457Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T00:27:15.7116181Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T00:27:15.7118098Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T00:27:15.7119687Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T00:27:15.7120438Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T00:27:15.7121185Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T00:27:15.7123751Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T00:27:15.7125166Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T00:27:15.7127351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T00:27:15.7128199Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T00:27:15.7129721Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T00:27:15.7131946Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T00:27:15.7132725Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-12-18T00:27:15.7133465Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-12-18T00:27:15.7175028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-12-18T00:27:15.7217131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-12-18T00:27:15.7218364Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-12-18T00:27:15.7265470Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-12-18T00:27:15.7266571Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-12-18T00:27:15.7267631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-12-18T00:27:15.7268722Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-12-18T00:27:15.7269831Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-12-18T00:27:15.7270874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-12-18T00:27:15.7271909Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-12-18T00:27:15.7272937Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-12-18T00:27:15.7273943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-12-18T00:27:15.7274898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-12-18T00:27:15.7275823Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-12-18T00:27:15.7276730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-12-18T00:27:15.7277651Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-12-18T00:27:15.7278548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-12-18T00:27:15.7279920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-12-18T00:27:15.7354116Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-12-18T00:27:15.7428860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-12-18T00:27:15.7429799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-12-18T00:27:15.7430546Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T00:27:15.7431182Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T00:27:15.7431852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T00:27:15.7432555Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T00:27:15.7433377Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T00:27:15.7434275Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T00:27:15.7435135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T00:27:15.7435948Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T00:27:15.7436933Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T00:27:15.7437883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T00:27:15.7438730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T00:27:15.7439564Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T00:27:15.7440392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T00:27:15.7443912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T00:27:15.7566152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T00:27:15.7566981Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T00:27:15.7567817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T00:27:15.7568758Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T00:27:15.7569652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T00:27:15.7570504Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T00:27:15.7571370Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T00:27:15.7572250Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T00:27:15.7573127Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T00:27:15.7574016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T00:27:15.7574886Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T00:27:15.7595576Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T00:27:15.7651035Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T00:27:15.7651965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T00:27:15.7652789Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T00:27:15.7653526Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T00:27:15.7654222Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T00:27:15.7655998Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T00:27:15.7656671Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-12-18T00:27:15.7659466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T00:27:15.7660261Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T00:27:15.7661152Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T00:27:15.7762016Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T00:27:15.7805469Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T00:27:15.7805957Z creating: build/lib/ 2024-12-18T00:27:15.7894537Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T00:27:15.8350180Z inflating: build/lib/libprotobuf.a 2024-12-18T00:27:15.8360049Z inflating: build/lib/libpthreadpool.a 2024-12-18T00:27:15.8368642Z inflating: build/lib/libcpuinfo.a 2024-12-18T00:27:15.8377002Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T00:27:15.8377413Z inflating: build/lib/libclog.a 2024-12-18T00:27:15.8396638Z inflating: build/lib/libnnpack.a 2024-12-18T00:27:15.8399188Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T00:27:15.8587121Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T00:27:15.8655345Z inflating: build/lib/libgtest.a 2024-12-18T00:27:15.8730391Z inflating: build/lib/libbenchmark.a 2024-12-18T00:27:15.8794817Z inflating: build/lib/libasmjit.a 2024-12-18T00:27:15.8802183Z inflating: build/lib/libittnotify.a 2024-12-18T00:27:15.8830288Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T00:27:15.8957543Z inflating: build/lib/libgloo.a 2024-12-18T00:27:15.8978633Z inflating: build/lib/libfmt.a 2024-12-18T00:27:15.9077957Z inflating: build/lib/libc10.so 2024-12-18T00:27:15.9080179Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T00:27:15.9081413Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T00:27:15.9587310Z inflating: build/lib/libprotoc.a 2024-12-18T00:27:15.9607101Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T00:27:15.9624472Z inflating: build/lib/libgmock.a 2024-12-18T00:27:15.9625012Z inflating: build/lib/libgtest_main.a 2024-12-18T00:27:15.9626092Z inflating: build/lib/libbenchmark_main.a 2024-12-18T00:27:16.0198984Z inflating: build/lib/libtensorpipe.a 2024-12-18T00:27:16.0587785Z inflating: build/lib/libgloo_cuda.a 2024-12-18T00:27:17.0915982Z inflating: build/lib/libdnnl.a 2024-12-18T00:27:17.0975757Z inflating: build/lib/libc10_cuda.so 2024-12-18T00:27:17.0976286Z inflating: build/lib/libgmock_main.a 2024-12-18T00:27:17.2235780Z inflating: build/lib/libfbgemm.a 2024-12-18T00:27:17.2492671Z inflating: build/lib/libtensorpipe_cuda.a 2024-12-18T00:27:17.3000748Z inflating: build/lib/libkineto.a 2024-12-18T00:27:17.3090069Z inflating: build/lib/libXNNPACK.a 2024-12-18T00:27:17.3132515Z inflating: build/lib/libonnx_proto.a 2024-12-18T00:27:17.3867331Z inflating: build/lib/libonnx.a 2024-12-18T00:27:17.4735173Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T00:27:20.0021132Z inflating: build/lib/libtorch_cpu.so 2024-12-18T00:27:20.0026225Z inflating: build/lib/libunbox_lib.a 2024-12-18T00:27:20.0030314Z inflating: build/lib/libshm.so 2024-12-18T00:27:22.0978151Z inflating: build/lib/libtorch_cuda.so 2024-12-18T00:27:22.0980163Z inflating: build/lib/libtorch.so 2024-12-18T00:27:22.0983066Z inflating: build/lib/libc10d_cuda_test.so 2024-12-18T00:27:22.9767528Z inflating: build/lib/libtorch_cuda_linalg.so 2024-12-18T00:27:23.1851783Z inflating: build/lib/libtorch_python.so 2024-12-18T00:27:23.1922604Z inflating: build/lib/libtorchbind_test.so 2024-12-18T00:27:23.1942978Z inflating: build/lib/libjitbackend_test.so 2024-12-18T00:27:23.1967729Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T00:27:23.1993391Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T00:27:23.2027465Z inflating: build/lib/libnnapi_backend.so 2024-12-18T00:27:23.2027815Z creating: build/bin/ 2024-12-18T00:27:23.2081628Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T00:27:23.2136599Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T00:27:23.2192110Z inflating: build/bin/c10_Device_test 2024-12-18T00:27:23.2254943Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T00:27:23.2312335Z inflating: build/bin/c10_Scalar_test 2024-12-18T00:27:23.2364254Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T00:27:23.2419202Z inflating: build/bin/c10_SymInt_test 2024-12-18T00:27:23.2476723Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T00:27:23.2536021Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T00:27:23.2596097Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T00:27:23.2669974Z inflating: build/bin/c10_cow_test 2024-12-18T00:27:23.2723109Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T00:27:23.2779554Z inflating: build/bin/c10_Bitset_test 2024-12-18T00:27:23.2832113Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T00:27:23.2885148Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T00:27:23.2939260Z inflating: build/bin/c10_Half_test 2024-12-18T00:27:23.2999289Z inflating: build/bin/c10_LeftRight_test 2024-12-18T00:27:23.3057651Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T00:27:23.3114734Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T00:27:23.3167858Z inflating: build/bin/c10_Synchronized_test 2024-12-18T00:27:23.3226851Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T00:27:23.3282222Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T00:27:23.3337196Z inflating: build/bin/c10_TypeList_test 2024-12-18T00:27:23.3389467Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T00:27:23.3444703Z inflating: build/bin/c10_accumulate_test 2024-12-18T00:27:23.3504641Z inflating: build/bin/c10_bfloat16_test 2024-12-18T00:27:23.3558209Z inflating: build/bin/c10_bit_cast_test 2024-12-18T00:27:23.3618769Z inflating: build/bin/c10_complex_math_test 2024-12-18T00:27:23.3678299Z inflating: build/bin/c10_complex_test 2024-12-18T00:27:23.3731872Z inflating: build/bin/c10_error_test 2024-12-18T00:27:23.3787846Z inflating: build/bin/c10_exception_test 2024-12-18T00:27:23.3842817Z inflating: build/bin/c10_flags_test 2024-12-18T00:27:23.3897252Z inflating: build/bin/c10_generic_math_test 2024-12-18T00:27:23.4068760Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T00:27:23.4123323Z inflating: build/bin/c10_irange_test 2024-12-18T00:27:23.4180145Z inflating: build/bin/c10_lazy_test 2024-12-18T00:27:23.4240979Z inflating: build/bin/c10_logging_test 2024-12-18T00:27:23.4321560Z inflating: build/bin/c10_optional_test 2024-12-18T00:27:23.4387728Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T00:27:23.4444839Z inflating: build/bin/c10_registry_test 2024-12-18T00:27:23.4605404Z inflating: build/bin/c10_small_vector_test 2024-12-18T00:27:23.4660267Z inflating: build/bin/c10_ssize_test 2024-12-18T00:27:23.4716366Z inflating: build/bin/c10_string_util_test 2024-12-18T00:27:23.4779676Z inflating: build/bin/c10_string_view_test 2024-12-18T00:27:23.4833930Z inflating: build/bin/c10_tempfile_test 2024-12-18T00:27:23.4896097Z inflating: build/bin/c10_typeid_test 2024-12-18T00:27:23.4945388Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T00:27:23.5393290Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T00:27:23.5841375Z inflating: build/bin/protoc 2024-12-18T00:27:23.5897846Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-12-18T00:27:23.5954157Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-12-18T00:27:23.6010380Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-12-18T00:27:23.6066406Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-12-18T00:27:23.6122343Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T00:27:23.6178828Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T00:27:23.6234754Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-12-18T00:27:23.6287283Z inflating: build/bin/c10_cuda_CUDATest 2024-12-18T00:27:23.6689412Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T00:27:23.7102211Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T00:27:23.7528198Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T00:27:23.7584501Z inflating: build/bin/BackoffTest 2024-12-18T00:27:23.7640982Z inflating: build/bin/FileStoreTest 2024-12-18T00:27:23.7700086Z inflating: build/bin/TCPStoreTest 2024-12-18T00:27:23.7756831Z inflating: build/bin/HashStoreTest 2024-12-18T00:27:23.7770808Z inflating: build/bin/ProcessGroupMPITest 2024-12-18T00:27:23.7827997Z inflating: build/bin/test_edge_op_registration 2024-12-18T00:27:23.7832202Z inflating: build/bin/torch_shm_manager 2024-12-18T00:27:23.7835315Z inflating: build/bin/example_allreduce 2024-12-18T00:27:23.7894016Z inflating: build/bin/test_dist_autograd 2024-12-18T00:27:23.7967561Z inflating: build/bin/test_cpp_rpc 2024-12-18T00:27:23.7970643Z inflating: build/bin/parallel_benchmark 2024-12-18T00:27:23.8041125Z inflating: build/bin/test_mobile_nnc 2024-12-18T00:27:23.8049820Z inflating: build/bin/aot_model_compiler_test 2024-12-18T00:27:23.8403551Z inflating: build/bin/test_lazy 2024-12-18T00:27:23.9593315Z inflating: build/bin/test_api 2024-12-18T00:27:23.9672234Z inflating: build/bin/Dict_test 2024-12-18T00:27:23.9727838Z inflating: build/bin/Dimname_test 2024-12-18T00:27:23.9796223Z inflating: build/bin/MaybeOwned_test 2024-12-18T00:27:23.9857762Z inflating: build/bin/NamedTensor_test 2024-12-18T00:27:23.9920460Z inflating: build/bin/apply_utils_test 2024-12-18T00:27:23.9983674Z inflating: build/bin/atest 2024-12-18T00:27:24.0050457Z inflating: build/bin/basic 2024-12-18T00:27:24.0109386Z inflating: build/bin/broadcast_test 2024-12-18T00:27:24.0164884Z inflating: build/bin/cpu_allocator_test 2024-12-18T00:27:24.0227675Z inflating: build/bin/cpu_generator_test 2024-12-18T00:27:24.0284362Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T00:27:24.0381208Z inflating: build/bin/cpu_rng_test 2024-12-18T00:27:24.0435054Z inflating: build/bin/dispatch_key_set_test 2024-12-18T00:27:24.0489111Z inflating: build/bin/dlconvertor_test 2024-12-18T00:27:24.0550862Z inflating: build/bin/extension_backend_test 2024-12-18T00:27:24.0609783Z inflating: build/bin/half_test 2024-12-18T00:27:24.0710291Z inflating: build/bin/ivalue_test 2024-12-18T00:27:24.0763052Z inflating: build/bin/lazy_tensor_test 2024-12-18T00:27:24.0820868Z inflating: build/bin/math_kernel_test 2024-12-18T00:27:24.0879538Z inflating: build/bin/memory_format_test 2024-12-18T00:27:24.0936836Z inflating: build/bin/memory_overlapping_test 2024-12-18T00:27:24.0993377Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T00:27:24.1053703Z inflating: build/bin/native_test 2024-12-18T00:27:24.1107918Z inflating: build/bin/operator_name_test 2024-12-18T00:27:24.1162333Z inflating: build/bin/operators_test 2024-12-18T00:27:24.1218138Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T00:27:24.1289104Z inflating: build/bin/pow_test 2024-12-18T00:27:24.1351271Z inflating: build/bin/quantized_test 2024-12-18T00:27:24.1404966Z inflating: build/bin/reduce_ops_test 2024-12-18T00:27:24.1459704Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T00:27:24.1520690Z inflating: build/bin/scalar_tensor_test 2024-12-18T00:27:24.1583279Z inflating: build/bin/scalar_test 2024-12-18T00:27:24.1638749Z inflating: build/bin/StorageUtils_test 2024-12-18T00:27:24.1694862Z inflating: build/bin/stride_properties_test 2024-12-18T00:27:24.1778489Z inflating: build/bin/tensor_iterator_test 2024-12-18T00:27:24.1836499Z inflating: build/bin/test_parallel 2024-12-18T00:27:24.1839659Z inflating: build/bin/thread_init_test 2024-12-18T00:27:24.1898993Z inflating: build/bin/type_ptr_test 2024-12-18T00:27:24.1962544Z inflating: build/bin/type_test 2024-12-18T00:27:24.2018726Z inflating: build/bin/undefined_tensor_test 2024-12-18T00:27:24.2020343Z inflating: build/bin/verify_api_visibility 2024-12-18T00:27:24.2094639Z inflating: build/bin/legacy_vmap_test 2024-12-18T00:27:24.2150880Z inflating: build/bin/weakref_test 2024-12-18T00:27:24.2205690Z inflating: build/bin/wrapdim_test 2024-12-18T00:27:24.2260899Z inflating: build/bin/xla_tensor_test 2024-12-18T00:27:24.2324406Z inflating: build/bin/IListRef_test 2024-12-18T00:27:24.2436128Z inflating: build/bin/List_test 2024-12-18T00:27:24.2508105Z inflating: build/bin/KernelFunction_test 2024-12-18T00:27:24.2635128Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T00:27:24.2736126Z inflating: build/bin/kernel_function_test 2024-12-18T00:27:24.2869353Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T00:27:24.2977120Z inflating: build/bin/kernel_lambda_test 2024-12-18T00:27:24.3043008Z inflating: build/bin/kernel_stackbased_test 2024-12-18T00:27:24.3143415Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T00:27:24.3198020Z inflating: build/bin/CppSignature_test 2024-12-18T00:27:24.3258185Z inflating: build/bin/backend_fallback_test 2024-12-18T00:27:24.3310524Z inflating: build/bin/op_allowlist_test 2024-12-18T00:27:24.3617929Z inflating: build/bin/op_registration_test 2024-12-18T00:27:24.3685762Z inflating: build/bin/inline_container_test 2024-12-18T00:27:24.3742077Z inflating: build/bin/cuda_apply_test 2024-12-18T00:27:24.3797993Z inflating: build/bin/cuda_allocator_test 2024-12-18T00:27:24.3857453Z inflating: build/bin/cuda_caching_host_allocator_test 2024-12-18T00:27:24.3920085Z inflating: build/bin/cuda_atomic_ops_test 2024-12-18T00:27:24.3994319Z inflating: build/bin/cuda_complex_math_test 2024-12-18T00:27:24.4057155Z inflating: build/bin/cuda_complex_test 2024-12-18T00:27:24.4110954Z inflating: build/bin/cuda_device_test 2024-12-18T00:27:24.4173053Z inflating: build/bin/cuda_cub_test 2024-12-18T00:27:24.4227474Z inflating: build/bin/cuda_dlconvertor_test 2024-12-18T00:27:24.4297242Z inflating: build/bin/cuda_distributions_test 2024-12-18T00:27:24.4357847Z inflating: build/bin/cuda_generator_test 2024-12-18T00:27:24.4411450Z inflating: build/bin/cuda_half_test 2024-12-18T00:27:24.4465767Z inflating: build/bin/cuda_integer_divider_test 2024-12-18T00:27:24.4519268Z inflating: build/bin/cuda_optional_test 2024-12-18T00:27:24.4574264Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-12-18T00:27:24.4630790Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-12-18T00:27:24.4683704Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-12-18T00:27:24.4747980Z inflating: build/bin/cuda_stream_test 2024-12-18T00:27:24.4801707Z inflating: build/bin/cuda_cudnn_test 2024-12-18T00:27:24.4856842Z inflating: build/bin/cuda_vectorized_test 2024-12-18T00:27:24.4871616Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T00:27:24.4941274Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T00:27:24.5002272Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-12-18T00:27:24.5070582Z inflating: build/bin/ProcessGroupNCCLTest 2024-12-18T00:27:24.5136199Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-12-18T00:27:24.5980168Z inflating: build/bin/test_tensorexpr 2024-12-18T00:27:24.6565321Z inflating: build/bin/test_jit 2024-12-18T00:27:24.6565662Z creating: .additional_ci_files/ 2024-12-18T00:27:24.6656244Z inflating: .additional_ci_files/test-times.json 2024-12-18T00:27:24.7016390Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T00:27:24.7057888Z ##[group]Run rm artifacts.zip 2024-12-18T00:27:24.7058204Z rm artifacts.zip 2024-12-18T00:27:24.7067505Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:24.7067880Z env: 2024-12-18T00:27:24.7068118Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:24.7068462Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:24.7068827Z ##[endgroup] 2024-12-18T00:27:24.8377164Z ##[group]Run df -H 2024-12-18T00:27:24.8377417Z df -H 2024-12-18T00:27:24.8385293Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:24.8385854Z env: 2024-12-18T00:27:24.8386084Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:24.8386432Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:24.8386792Z ##[endgroup] 2024-12-18T00:27:24.8436209Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T00:27:24.8436995Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-12-18T00:27:24.8437655Z tmpfs 101G 0 101G 0% /dev/shm 2024-12-18T00:27:24.8438325Z tmpfs 41G 697k 41G 1% /run 2024-12-18T00:27:24.8438967Z /dev/nvme0n1p1 161G 59G 103G 37% / 2024-12-18T00:27:24.8439714Z tmpfs 101G 25k 101G 1% /tmp 2024-12-18T00:27:24.8440396Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-12-18T00:27:24.8441047Z tmpfs 21G 0 21G 0% /run/user/0 2024-12-18T00:27:24.8473369Z Prepare all required actions 2024-12-18T00:27:24.8473746Z Getting action download info 2024-12-18T00:27:25.0227561Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T00:27:25.0227929Z with: 2024-12-18T00:27:25.0228140Z env: 2024-12-18T00:27:25.0228368Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.0228704Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.0229058Z ##[endgroup] 2024-12-18T00:27:25.0263661Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T00:27:25.0264009Z with: 2024-12-18T00:27:25.0264220Z name: td_results 2024-12-18T00:27:25.0264467Z s3-bucket: gha-artifacts 2024-12-18T00:27:25.0264726Z region: us-east-1 2024-12-18T00:27:25.0264954Z env: 2024-12-18T00:27:25.0265182Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.0265515Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.0265870Z ##[endgroup] 2024-12-18T00:27:25.4766103Z (node:191333) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T00:27:25.4766577Z 2024-12-18T00:27:25.4766782Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T00:27:25.4767306Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T00:27:25.4767867Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T00:27:25.5721889Z Found 0 objects with prefix pytorch/pytorch/12383255690/td_results/ 2024-12-18T00:27:25.5729674Z Artifact download has finished successfully 2024-12-18T00:27:25.6086045Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T00:27:25.6086421Z mkdir -p .additional_ci_files 2024-12-18T00:27:25.6086858Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T00:27:25.6097431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:25.6097808Z env: 2024-12-18T00:27:25.6098034Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.6098390Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.6098756Z ##[endgroup] 2024-12-18T00:27:25.6157756Z mv: cannot stat 'td_results.json': No such file or directory 2024-12-18T00:27:25.6198809Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T00:27:25.6199217Z .github/scripts/parse_ref.py 2024-12-18T00:27:25.6208045Z shell: /usr/bin/bash -e {0} 2024-12-18T00:27:25.6208326Z env: 2024-12-18T00:27:25.6208551Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.6208900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.6209262Z ##[endgroup] 2024-12-18T00:27:25.6488881Z Prepare all required actions 2024-12-18T00:27:25.6525591Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T00:27:25.6525926Z with: 2024-12-18T00:27:25.6526320Z github-token: *** 2024-12-18T00:27:25.6526566Z env: 2024-12-18T00:27:25.6526798Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.6527144Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.6527498Z ##[endgroup] 2024-12-18T00:27:25.6549408Z ##[group]Run set -eux 2024-12-18T00:27:25.6549673Z set -eux 2024-12-18T00:27:25.6550101Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T00:27:25.6558427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:25.6558796Z env: 2024-12-18T00:27:25.6559033Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:25.6559374Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:25.6559883Z GITHUB_TOKEN: *** 2024-12-18T00:27:25.6560129Z ##[endgroup] 2024-12-18T00:27:25.6587568Z + python3 .github/scripts/get_workflow_job_id.py 12383255690 i-079e9e3a7e7495371 2024-12-18T00:27:26.7687107Z setting job-id=34567098055 2024-12-18T00:27:26.7687709Z setting job-name=unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:26.7856763Z Prepare all required actions 2024-12-18T00:27:26.7857138Z Getting action download info 2024-12-18T00:27:26.9158002Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T00:27:26.9158361Z with: 2024-12-18T00:27:26.9158768Z github-token: *** 2024-12-18T00:27:26.9160117Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-12-18T00:27:26.9161792Z job-name: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:26.9162355Z env: 2024-12-18T00:27:26.9162580Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:26.9163218Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:26.9163752Z ##[endgroup] 2024-12-18T00:27:26.9206702Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T00:27:26.9207011Z with: 2024-12-18T00:27:26.9207224Z shell: bash 2024-12-18T00:27:26.9207455Z timeout_minutes: 10 2024-12-18T00:27:26.9207706Z max_attempts: 5 2024-12-18T00:27:26.9207953Z retry_wait_seconds: 30 2024-12-18T00:27:26.9208707Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:27:26.9209483Z polling_interval_seconds: 1 2024-12-18T00:27:26.9209766Z warning_on_retry: true 2024-12-18T00:27:26.9210028Z continue_on_error: false 2024-12-18T00:27:26.9210293Z env: 2024-12-18T00:27:26.9210507Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:26.9210839Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:26.9211358Z GITHUB_TOKEN: *** 2024-12-18T00:27:26.9211607Z ##[endgroup] 2024-12-18T00:27:27.0219202Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T00:27:27.2588576Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T00:27:27.2772719Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-12-18T00:27:27.2777065Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-12-18T00:27:27.2894745Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-12-18T00:27:27.2898603Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-12-18T00:27:27.2902398Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.12.14) 2024-12-18T00:27:27.2911766Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-12-18T00:27:27.9994411Z Command completed after 1 attempt(s). 2024-12-18T00:27:28.0070057Z ##[group]Run set -x 2024-12-18T00:27:28.0070349Z set -x 2024-12-18T00:27:28.0070622Z  2024-12-18T00:27:28.0071002Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:27:28.0071692Z # in runner workspace 2024-12-18T00:27:28.0072104Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T00:27:28.0081099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:28.0081476Z env: 2024-12-18T00:27:28.0081715Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:28.0082104Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:28.0082456Z ##[endgroup] 2024-12-18T00:27:28.0110769Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T00:27:28.0356771Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:27:28.0357202Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T00:27:28.0357539Z echo "Job name: ${JOB_NAME}" 2024-12-18T00:27:28.0357835Z  2024-12-18T00:27:28.0358225Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T00:27:28.0358720Z # in runner workspace 2024-12-18T00:27:28.0359156Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T00:27:28.0359638Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T00:27:28.0359977Z  --job-name "${JOB_NAME}" \ 2024-12-18T00:27:28.0361425Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-12-18T00:27:28.0362971Z  --selected-test-configs "" \ 2024-12-18T00:27:28.0363323Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T00:27:28.0363834Z  --tag "${TAG}" \ 2024-12-18T00:27:28.0364140Z  --event-name "${EVENT_NAME}" \ 2024-12-18T00:27:28.0364467Z  --schedule "${SCHEDULE}" \ 2024-12-18T00:27:28.0364774Z  --branch "${HEAD_BRANCH}" 2024-12-18T00:27:28.0372724Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:28.0373094Z env: 2024-12-18T00:27:28.0373318Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:28.0373658Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:28.0374200Z GITHUB_TOKEN: *** 2024-12-18T00:27:28.0374724Z JOB_NAME: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:28.0375284Z PR_NUMBER: 2024-12-18T00:27:28.0375511Z TAG: 2024-12-18T00:27:28.0375726Z EVENT_NAME: push 2024-12-18T00:27:28.0375966Z SCHEDULE: 2024-12-18T00:27:28.0376191Z HEAD_BRANCH: 2024-12-18T00:27:28.0376417Z ##[endgroup] 2024-12-18T00:27:28.0407516Z Workflow: inductor 2024-12-18T00:27:28.0408227Z Job name: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:28.2919572Z ##[group]Run echo "Filtered matrix:" 2024-12-18T00:27:28.2919918Z echo "Filtered matrix:" 2024-12-18T00:27:28.2921310Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-12-18T00:27:28.2922677Z  2024-12-18T00:27:28.2922979Z echo 2024-12-18T00:27:28.2923263Z echo "Is the current job unstable? False" 2024-12-18T00:27:28.2923597Z  2024-12-18T00:27:28.2923809Z echo 2024-12-18T00:27:28.2924080Z echo "Is keep-going label set? False" 2024-12-18T00:27:28.2924608Z  2024-12-18T00:27:28.2924818Z echo 2024-12-18T00:27:28.2925065Z echo "Renabled issues? " 2024-12-18T00:27:28.2933751Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:28.2934127Z env: 2024-12-18T00:27:28.2934361Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:28.2934696Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:28.2935065Z ##[endgroup] 2024-12-18T00:27:28.2963547Z Filtered matrix: 2024-12-18T00:27:28.2965376Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-12-18T00:27:28.2966676Z 2024-12-18T00:27:28.2966828Z Is the current job unstable? False 2024-12-18T00:27:28.2967056Z 2024-12-18T00:27:28.2967169Z Is keep-going label set? False 2024-12-18T00:27:28.2967363Z 2024-12-18T00:27:28.2967462Z Renabled issues? 2024-12-18T00:27:28.3010126Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:27:28.3010646Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T00:27:28.3019578Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T00:27:28.3019963Z env: 2024-12-18T00:27:28.3020193Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:28.3020542Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:28.3020903Z JOB_TIMEOUT: 240 2024-12-18T00:27:28.3021144Z ##[endgroup] 2024-12-18T00:27:28.3106927Z ##[group]Run set -x 2024-12-18T00:27:28.3107241Z set -x 2024-12-18T00:27:28.3107470Z  2024-12-18T00:27:28.3107729Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T00:27:28.3108126Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T00:27:28.3108517Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T00:27:28.3108897Z  TEST_COMMAND=.ci/onnx/test.sh 2024-12-18T00:27:28.3109197Z else 2024-12-18T00:27:28.3109458Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:27:28.3109768Z fi 2024-12-18T00:27:28.3109983Z  2024-12-18T00:27:28.3110320Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T00:27:28.3110841Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T00:27:28.3111306Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T00:27:28.3111704Z # shellcheck disable=SC2086,SC2090 2024-12-18T00:27:28.3112037Z container_name=$(docker run \ 2024-12-18T00:27:28.3112356Z  ${GPU_FLAG:-} \ 2024-12-18T00:27:28.3112667Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2024-12-18T00:27:28.3113015Z  -e BUILD_ENVIRONMENT \ 2024-12-18T00:27:28.3113314Z  -e PR_NUMBER \ 2024-12-18T00:27:28.3113600Z  -e GITHUB_ACTIONS \ 2024-12-18T00:27:28.3113900Z  -e GITHUB_REPOSITORY \ 2024-12-18T00:27:28.3114204Z  -e GITHUB_WORKFLOW \ 2024-12-18T00:27:28.3114494Z  -e GITHUB_JOB \ 2024-12-18T00:27:28.3114766Z  -e GITHUB_RUN_ID \ 2024-12-18T00:27:28.3115054Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T00:27:28.3115364Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T00:27:28.3115667Z  -e JOB_ID \ 2024-12-18T00:27:28.3115938Z  -e JOB_NAME \ 2024-12-18T00:27:28.3116203Z  -e BASE_SHA \ 2024-12-18T00:27:28.3116474Z  -e BRANCH \ 2024-12-18T00:27:28.3116740Z  -e SHA1 \ 2024-12-18T00:27:28.3117007Z  -e AWS_DEFAULT_REGION \ 2024-12-18T00:27:28.3117310Z  -e IN_WHEEL_TEST \ 2024-12-18T00:27:28.3117591Z  -e SHARD_NUMBER \ 2024-12-18T00:27:28.3117879Z  -e TEST_CONFIG \ 2024-12-18T00:27:28.3118176Z  -e NUM_TEST_SHARDS \ 2024-12-18T00:27:28.3118663Z  -e REENABLED_ISSUES \ 2024-12-18T00:27:28.3118975Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T00:27:28.3119297Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T00:27:28.3119596Z  -e TEST_SHOWLOCALS \ 2024-12-18T00:27:28.3119892Z  -e NO_TEST_TIMEOUT \ 2024-12-18T00:27:28.3120178Z  -e NO_TD \ 2024-12-18T00:27:28.3120443Z  -e TD_DISTRIBUTED \ 2024-12-18T00:27:28.3120732Z  -e PR_LABELS \ 2024-12-18T00:27:28.3121030Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T00:27:28.3121377Z  -e SCCACHE_BUCKET \ 2024-12-18T00:27:28.3121683Z  -e SCCACHE_REGION \ 2024-12-18T00:27:28.3121987Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-12-18T00:27:28.3122305Z  -e XLA_CUDA \ 2024-12-18T00:27:28.3122609Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T00:27:28.3123088Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T00:27:28.3123472Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T00:27:28.3123862Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-12-18T00:27:28.3124216Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-12-18T00:27:28.3124582Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-12-18T00:27:28.3146331Z  -e DASHBOARD_TAG \ 2024-12-18T00:27:28.3146681Z  -e IS_A100_RUNNER \ 2024-12-18T00:27:28.3147016Z  -e ARTIFACTS_FILE_SUFFIX \ 2024-12-18T00:27:28.3147395Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T00:27:28.3147806Z  --security-opt seccomp=unconfined \ 2024-12-18T00:27:28.3148333Z  --cap-add=SYS_PTRACE \ 2024-12-18T00:27:28.3148642Z  --ipc=host \ 2024-12-18T00:27:28.3148926Z  --shm-size="${SHM_SIZE}" \ 2024-12-18T00:27:28.3149231Z  --tty \ 2024-12-18T00:27:28.3149481Z  --detach \ 2024-12-18T00:27:28.3149757Z  --name="${container_name}" \ 2024-12-18T00:27:28.3150122Z  --user jenkins \ 2024-12-18T00:27:28.3150497Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T00:27:28.3150902Z  -w /var/lib/jenkins/workspace \ 2024-12-18T00:27:28.3151229Z  "${DOCKER_IMAGE}" 2024-12-18T00:27:28.3151493Z ) 2024-12-18T00:27:28.3151796Z # Propagate download.pytorch.org IP to container 2024-12-18T00:27:28.3152445Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-12-18T00:27:28.3153126Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-12-18T00:27:28.3153810Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-12-18T00:27:28.3162320Z shell: /usr/bin/bash -e {0} 2024-12-18T00:27:28.3162601Z env: 2024-12-18T00:27:28.3162826Z GIT_DEFAULT_BRANCH: main 2024-12-18T00:27:28.3163267Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:28.3163884Z BUILD_ENVIRONMENT: linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:27:28.3164298Z PR_NUMBER: 2024-12-18T00:27:28.3164612Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T00:27:28.3165009Z GITHUB_WORKFLOW: inductor 2024-12-18T00:27:28.3165383Z GITHUB_JOB: test 2024-12-18T00:27:28.3165716Z GITHUB_RUN_ID: 12383255690 2024-12-18T00:27:28.3169816Z GITHUB_RUN_NUMBER: 109429 2024-12-18T00:27:28.3170117Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T00:27:28.3170376Z JOB_ID: 34567098055 2024-12-18T00:27:28.3170900Z JOB_NAME: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:28.3171463Z BRANCH: release/2.6 2024-12-18T00:27:28.3171751Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:28.3172124Z BASE_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:28.3172474Z TEST_CONFIG: inductor_distributed 2024-12-18T00:27:28.3172767Z SHARD_NUMBER: 1 2024-12-18T00:27:28.3173195Z NUM_TEST_SHARDS: 1 2024-12-18T00:27:28.3173436Z REENABLED_ISSUES: 2024-12-18T00:27:28.3173690Z CONTINUE_THROUGH_ERROR: False 2024-12-18T00:27:28.3173970Z VERBOSE_TEST_LOGS: False 2024-12-18T00:27:28.3174237Z TEST_SHOWLOCALS: False 2024-12-18T00:27:28.3174495Z NO_TEST_TIMEOUT: False 2024-12-18T00:27:28.3195447Z NO_TD: False 2024-12-18T00:27:28.3195692Z TD_DISTRIBUTED: False 2024-12-18T00:27:28.3196007Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T00:27:28.3196362Z SCCACHE_REGION: us-east-1 2024-12-18T00:27:28.3196643Z SCCACHE_S3_KEY_PREFIX: inductor 2024-12-18T00:27:28.3196919Z SHM_SIZE: 2g 2024-12-18T00:27:28.3197701Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:27:28.3198541Z XLA_CUDA: 2024-12-18T00:27:28.3198904Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:27:28.3199359Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T00:27:28.3199697Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T00:27:28.3199999Z DASHBOARD_TAG: 2024-12-18T00:27:28.3200426Z HUGGING_FACE_HUB_TOKEN: *** 2024-12-18T00:27:28.3200843Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2024-12-18T00:27:28.3201229Z IS_A100_RUNNER: 0 2024-12-18T00:27:28.3201722Z ARTIFACTS_FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T00:27:28.3202327Z ##[endgroup] 2024-12-18T00:27:28.3230040Z + [[ inductor_distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T00:27:28.3230487Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *onnx* ]] 2024-12-18T00:27:28.3231097Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T00:27:28.3239819Z +++ nproc --ignore=2 2024-12-18T00:27:28.3255945Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=46 -e SCCACHE_BUCKET -e SCCACHE_REGION -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --env-file=/tmp/github_env_12383255690 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T00:27:40.3069287Z + container_name=06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T00:27:40.3072650Z + grep download.pytorch.org /etc/hosts 2024-12-18T00:27:40.3073718Z + docker exec -i 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d sudo bash -c '/bin/cat >> /etc/hosts' 2024-12-18T00:27:40.4633739Z + echo DOCKER_CONTAINER_ID=06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T00:27:40.4639020Z ++ echo dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T00:27:40.4642239Z + docker exec -t 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d sh -c 'python3 -m pip install dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-12-18T00:27:40.8626048Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl (from torch==2.6.0a0+git0cdf8b1) 2024-12-18T00:27:41.1955212Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T00:27:41.1958353Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T00:27:41.1961363Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T00:27:41.1964394Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T00:27:41.1967713Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T00:27:41.1972277Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T00:27:41.1988343Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T00:27:41.2001167Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.3.0) 2024-12-18T00:27:41.2018848Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (1.22.4) 2024-12-18T00:27:41.2408039Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T00:27:41.8406935Z Installing collected packages: torch 2024-12-18T00:27:52.9246690Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-12-18T00:27:52.9247697Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-12-18T00:27:52.9248230Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T00:27:53.0212734Z + export TERM=vt100 2024-12-18T00:27:53.0213225Z + TERM=vt100 2024-12-18T00:27:53.0218353Z ++ dirname .ci/pytorch/test.sh 2024-12-18T00:27:53.0229813Z + source .ci/pytorch/common.sh 2024-12-18T00:27:53.0234200Z +++ dirname .ci/pytorch/common.sh 2024-12-18T00:27:53.0244011Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T00:27:53.0245978Z +++ declare -f -t trap_add 2024-12-18T00:27:53.0251177Z ++ set -ex 2024-12-18T00:27:53.0251672Z ++ [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-12-18T00:27:53.0252073Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T00:27:53.0252985Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-12-18T00:27:53.0253466Z + [[ -d /var/lib/jenkins/workspace ]] 2024-12-18T00:27:53.0257194Z ++ stat -c %u /var/lib/jenkins/workspace 2024-12-18T00:27:53.0279563Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-12-18T00:27:53.0279901Z + trap_add cleanup_workspace EXIT 2024-12-18T00:27:53.0280225Z + trap_add_cmd=cleanup_workspace 2024-12-18T00:27:53.0280504Z + shift 2024-12-18T00:27:53.0280732Z + for trap_add_name in "$@" 2024-12-18T00:27:53.0289634Z +++ trap -p EXIT 2024-12-18T00:27:53.0293120Z ++ eval 'extract_trap_cmd ' 2024-12-18T00:27:53.0293529Z +++ extract_trap_cmd 2024-12-18T00:27:53.0293862Z +++ printf '%s\n' '' 2024-12-18T00:27:53.0294224Z ++ printf '%s\n' cleanup_workspace 2024-12-18T00:27:53.0296613Z + trap -- ' 2024-12-18T00:27:53.0296957Z cleanup_workspace' EXIT 2024-12-18T00:27:53.0297394Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-12-18T00:27:53.7447385Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-12-18T00:27:53.7470587Z + echo 'Environment variables:' 2024-12-18T00:27:53.7471508Z Environment variables: 2024-12-18T00:27:53.7471847Z + env 2024-12-18T00:27:53.7482618Z INSTALLED_DB=yes 2024-12-18T00:27:53.7483162Z NV_LIBCUBLAS_VERSION=12.4.5.8-1 2024-12-18T00:27:53.7483554Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:27:53.7483926Z NV_NVML_DEV_VERSION=12.4.127-1 2024-12-18T00:27:53.7484419Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:27:53.7484877Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:27:53.7485265Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4 2024-12-18T00:27:53.7485638Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:27:53.7486029Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:27:53.7486416Z HOSTNAME=06c5d3cbe8cd 2024-12-18T00:27:53.7486978Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7487573Z GITHUB_ACTION=__self 2024-12-18T00:27:53.7487842Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:27:53.7492167Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-12-18T00:27:53.7496289Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1 2024-12-18T00:27:53.7496657Z NV_NVTX_VERSION=12.4.127-1 2024-12-18T00:27:53.7496925Z GITHUB_RUN_NUMBER=109429 2024-12-18T00:27:53.7497202Z TEST_CONFIG=inductor_distributed 2024-12-18T00:27:53.7497504Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:27:53.7497823Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:27:53.7498128Z IS_A100_RUNNER=0 2024-12-18T00:27:53.7498379Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 2024-12-18T00:27:53.7498699Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1 2024-12-18T00:27:53.7499198Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:27:53.7499510Z NV_LIBNPP_VERSION=12.2.5.30-1 2024-12-18T00:27:53.7499812Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:27:53.7500173Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:27:53.7500552Z GITHUB_REF_TYPE=branch 2024-12-18T00:27:53.7500832Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:27:53.7501121Z NCCL_VERSION=2.21.5-1 2024-12-18T00:27:53.7501419Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7501747Z XLA_CUDA= 2024-12-18T00:27:53.7502089Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:27:53.7502631Z *** 2024-12-18T00:27:53.7502859Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:27:53.7503166Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:27:53.7503442Z GITHUB_ACTIONS=true 2024-12-18T00:27:53.7503702Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:53.7504035Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1 2024-12-18T00:27:53.7504404Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1 2024-12-18T00:27:53.7504763Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7505116Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:27:53.7505476Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7506021Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6 2024-12-18T00:27:53.7506621Z UCC_HOME=/usr 2024-12-18T00:27:53.7506873Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 2024-12-18T00:27:53.7507169Z VERBOSE_TEST_LOGS=False 2024-12-18T00:27:53.7507436Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:27:53.7507752Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-12-18T00:27:53.7508106Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:27:53.7508409Z NV_CUDA_CUDART_VERSION=12.4.127-1 2024-12-18T00:27:53.7508688Z SHARD_NUMBER=1 2024-12-18T00:27:53.7508932Z GITHUB_REF_PROTECTED=true 2024-12-18T00:27:53.7509200Z HOME=/var/lib/jenkins 2024-12-18T00:27:53.7509484Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:27:53.7509813Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:27:53.7510160Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:27:53.7510516Z SCCACHE_S3_KEY_PREFIX=inductor 2024-12-18T00:27:53.7510800Z CUDA_VERSION=12.4.1 2024-12-18T00:27:53.7511079Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1 2024-12-18T00:27:53.7511397Z NUM_TEST_SHARDS=1 2024-12-18T00:27:53.7511640Z UCX_HOME=/usr 2024-12-18T00:27:53.7511983Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1 2024-12-18T00:27:53.7512705Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7513568Z JOB_NAME=unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:53.7514410Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7515262Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:27:53.7515747Z GITHUB_EVENT_NAME=push 2024-12-18T00:27:53.7516004Z DASHBOARD_TAG= 2024-12-18T00:27:53.7516246Z GITHUB_RUN_ID=12383255690 2024-12-18T00:27:53.7516555Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1 2024-12-18T00:27:53.7516918Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-12-18T00:27:53.7517584Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7518224Z GITHUB_ACTOR=malfet 2024-12-18T00:27:53.7518486Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1 2024-12-18T00:27:53.7518770Z PR_NUMBER= 2024-12-18T00:27:53.7518990Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:27:53.7519259Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:27:53.7519597Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:27:53.7519936Z TERM=vt100 2024-12-18T00:27:53.7520176Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1 2024-12-18T00:27:53.7520482Z INSTALLED_VISION=yes 2024-12-18T00:27:53.7520734Z BRANCH=release/2.6 2024-12-18T00:27:53.7520992Z SCCACHE_REGION=us-east-1 2024-12-18T00:27:53.7521270Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:27:53.7521575Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:27:53.7521891Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:27:53.7522397Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:27:53.7523026Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:27:53.7523393Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:27:53.7523742Z REENABLED_ISSUES= 2024-12-18T00:27:53.7523968Z SHLVL=1 2024-12-18T00:27:53.7524193Z MAX_JOBS=46 2024-12-18T00:27:53.7524425Z NV_CUDA_LIB_VERSION=12.4.1-1 2024-12-18T00:27:53.7524696Z NVARCH=x86_64 2024-12-18T00:27:53.7524932Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:27:53.7525274Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7525655Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:27:53.7525954Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-12-18T00:27:53.7526404Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:27:53.7526826Z GITHUB_JOB=test 2024-12-18T00:27:53.7527097Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4 2024-12-18T00:27:53.7527499Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:27:53.7527876Z NO_TEST_TIMEOUT=False 2024-12-18T00:27:53.7528227Z TD_DISTRIBUTED=False 2024-12-18T00:27:53.7528508Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 2024-12-18T00:27:53.7528834Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:27:53.7529130Z NV_NVPROF_VERSION=12.4.127-1 2024-12-18T00:27:53.7529412Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:27:53.7529681Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:27:53.7529962Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:27:53.7530698Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:27:53.7531441Z GITHUB_BASE_REF= 2024-12-18T00:27:53.7531911Z ARTIFACTS_FILE_SUFFIX=test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T00:27:53.7532446Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:27:53.7532727Z CI=true 2024-12-18T00:27:53.7532958Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:27:53.7533270Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:27:53.7533557Z JOB_ID=34567098055 2024-12-18T00:27:53.7533808Z INSTALLED_PROTOBUF=yes 2024-12-18T00:27:53.7534067Z GITHUB_HEAD_REF= 2024-12-18T00:27:53.7534306Z GITHUB_ACTION_REF= 2024-12-18T00:27:53.7534598Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:27:53.7534950Z TEST_SHOWLOCALS=False 2024-12-18T00:27:53.7535212Z GITHUB_WORKFLOW=inductor 2024-12-18T00:27:53.7535500Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:27:53.7536106Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7536705Z NO_TD=False 2024-12-18T00:27:53.7536950Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:27:53.7537326Z _=/usr/bin/env 2024-12-18T00:27:53.7537650Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T00:27:53.7718529Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T00:27:53.7719258Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T00:27:53.7719843Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T00:27:53.7720430Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T00:27:53.7720869Z + BUILD_DIR=build 2024-12-18T00:27:53.7721131Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T00:27:53.7721430Z + BUILD_BIN_DIR=build/bin 2024-12-18T00:27:53.7721690Z + SHARD_NUMBER=1 2024-12-18T00:27:53.7721933Z + NUM_TEST_SHARDS=1 2024-12-18T00:27:53.7722180Z + export VALGRIND=ON 2024-12-18T00:27:53.7722439Z + VALGRIND=ON 2024-12-18T00:27:53.7722752Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *clang9* ]] 2024-12-18T00:27:53.7723302Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-12-18T00:27:53.7723668Z + [[ 0 == \1 ]] 2024-12-18T00:27:53.7723962Z + [[ False == \1 ]] 2024-12-18T00:27:53.7724273Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *bazel* ]] 2024-12-18T00:27:53.7725187Z ++ realpath build/custom_test_artifacts 2024-12-18T00:27:53.7736468Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-12-18T00:27:53.7736987Z + [[ -n '' ]] 2024-12-18T00:27:53.7737241Z + echo 'Environment variables' 2024-12-18T00:27:53.7737536Z Environment variables 2024-12-18T00:27:53.7737787Z + env 2024-12-18T00:27:53.7746744Z INSTALLED_DB=yes 2024-12-18T00:27:53.7747364Z NV_LIBCUBLAS_VERSION=12.4.5.8-1 2024-12-18T00:27:53.7747727Z NVIDIA_VISIBLE_DEVICES=all 2024-12-18T00:27:53.7748028Z NV_NVML_DEV_VERSION=12.4.127-1 2024-12-18T00:27:53.7748559Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T00:27:53.7749065Z CONTINUE_THROUGH_ERROR=False 2024-12-18T00:27:53.7749452Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.21.5-1+cuda12.4 2024-12-18T00:27:53.7749871Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:27:53.7750307Z BUILD_ENVIRONMENT=linux-focal-cuda12.4-py3.10-gcc9-sm86 2024-12-18T00:27:53.7750720Z HOSTNAME=06c5d3cbe8cd 2024-12-18T00:27:53.7751293Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7752102Z GITHUB_ACTION=__self 2024-12-18T00:27:53.7752442Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T00:27:53.7756440Z NVIDIA_REQUIRE_CUDA=cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536 2024-12-18T00:27:53.7760911Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-4=12.4.5.8-1 2024-12-18T00:27:53.7761312Z NV_NVTX_VERSION=12.4.127-1 2024-12-18T00:27:53.7761597Z GITHUB_RUN_NUMBER=109429 2024-12-18T00:27:53.7761897Z TEST_CONFIG=inductor_distributed 2024-12-18T00:27:53.7762209Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T00:27:53.7762713Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-12-18T00:27:53.7763202Z IS_A100_RUNNER=0 2024-12-18T00:27:53.7763930Z NV_CUDA_CUDART_DEV_VERSION=12.4.127-1 2024-12-18T00:27:53.7764372Z NV_LIBCUSPARSE_VERSION=12.3.1.170-1 2024-12-18T00:27:53.7764953Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2024-12-18T00:27:53.7765260Z NV_LIBNPP_VERSION=12.2.5.30-1 2024-12-18T00:27:53.7765575Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T00:27:53.7765920Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-12-18T00:27:53.7766270Z GITHUB_REF_TYPE=branch 2024-12-18T00:27:53.7766543Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-12-18T00:27:53.7766835Z NCCL_VERSION=2.21.5-1 2024-12-18T00:27:53.7767143Z BASE_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7767482Z XLA_CUDA= 2024-12-18T00:27:53.7767813Z HUGGING_FACE_HUB_TOKEN=*** 2024-12-18T00:27:53.7768288Z *** 2024-12-18T00:27:53.7768561Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-12-18T00:27:53.7768877Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T00:27:53.7769170Z GITHUB_ACTIONS=true 2024-12-18T00:27:53.7769440Z NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T00:27:53.7769781Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-4=12.4.127-1 2024-12-18T00:27:53.7770155Z NV_LIBNPP_PACKAGE=libnpp-12-4=12.2.5.30-1 2024-12-18T00:27:53.7770513Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7770869Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-12-18T00:27:53.7771231Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7771778Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/release/2.6 2024-12-18T00:27:53.7772272Z UCC_HOME=/usr 2024-12-18T00:27:53.7772524Z NV_LIBCUBLAS_DEV_VERSION=12.4.5.8-1 2024-12-18T00:27:53.7772831Z VERBOSE_TEST_LOGS=False 2024-12-18T00:27:53.7773104Z NVIDIA_PRODUCT_NAME=CUDA 2024-12-18T00:27:53.7773462Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-4 2024-12-18T00:27:53.7773812Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T00:27:53.7774120Z NV_CUDA_CUDART_VERSION=12.4.127-1 2024-12-18T00:27:53.7774408Z SHARD_NUMBER=1 2024-12-18T00:27:53.7774658Z GITHUB_REF_PROTECTED=true 2024-12-18T00:27:53.7774928Z HOME=/var/lib/jenkins 2024-12-18T00:27:53.7775209Z GITHUB_API_URL=https://api.github.com 2024-12-18T00:27:53.7775542Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T00:27:53.7775894Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-12-18T00:27:53.7776248Z SCCACHE_S3_KEY_PREFIX=inductor 2024-12-18T00:27:53.7776741Z CUDA_VERSION=12.4.1 2024-12-18T00:27:53.7777018Z NV_LIBCUBLAS_PACKAGE=libcublas-12-4=12.4.5.8-1 2024-12-18T00:27:53.7777347Z NUM_TEST_SHARDS=1 2024-12-18T00:27:53.7777584Z UCX_HOME=/usr 2024-12-18T00:27:53.7777935Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-4=12.4.1-1 2024-12-18T00:27:53.7778657Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7779509Z JOB_NAME=unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T00:27:53.7780356Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7781120Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T00:27:53.7781603Z GITHUB_EVENT_NAME=push 2024-12-18T00:27:53.7781869Z DASHBOARD_TAG= 2024-12-18T00:27:53.7782112Z GITHUB_RUN_ID=12383255690 2024-12-18T00:27:53.7782430Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-4=12.2.5.30-1 2024-12-18T00:27:53.7782800Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-4 2024-12-18T00:27:53.7783466Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7784104Z GITHUB_ACTOR=malfet 2024-12-18T00:27:53.7784367Z NV_LIBNPP_DEV_VERSION=12.2.5.30-1 2024-12-18T00:27:53.7784649Z PR_NUMBER= 2024-12-18T00:27:53.7784878Z GITHUB_RUN_ATTEMPT=1 2024-12-18T00:27:53.7785130Z VALGRIND=ON 2024-12-18T00:27:53.7785376Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T00:27:53.7785849Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T00:27:53.7786191Z TERM=vt100 2024-12-18T00:27:53.7786436Z NV_LIBCUSPARSE_DEV_VERSION=12.3.1.170-1 2024-12-18T00:27:53.7786749Z INSTALLED_VISION=yes 2024-12-18T00:27:53.7787005Z BRANCH=release/2.6 2024-12-18T00:27:53.7787255Z SCCACHE_REGION=us-east-1 2024-12-18T00:27:53.7787538Z OPENSSL_ROOT_DIR=/opt/openssl 2024-12-18T00:27:53.7787850Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-12-18T00:27:53.7788169Z CUDA_PATH=/usr/local/cuda 2024-12-18T00:27:53.7788672Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-12-18T00:27:53.7789226Z GITHUB_SERVER_URL=https://github.com 2024-12-18T00:27:53.7789583Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-12-18T00:27:53.7789932Z REENABLED_ISSUES= 2024-12-18T00:27:53.7790163Z SHLVL=1 2024-12-18T00:27:53.7790372Z MAX_JOBS=46 2024-12-18T00:27:53.7790596Z NV_CUDA_LIB_VERSION=12.4.1-1 2024-12-18T00:27:53.7790869Z NVARCH=x86_64 2024-12-18T00:27:53.7791104Z GITHUB_ACTOR_ID=2453524 2024-12-18T00:27:53.7791452Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T00:27:53.7791839Z GITHUB_REF_NAME=release/2.6 2024-12-18T00:27:53.7792134Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-4 2024-12-18T00:27:53.7792579Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T00:27:53.7793002Z GITHUB_JOB=test 2024-12-18T00:27:53.7793270Z NV_LIBNCCL_PACKAGE=libnccl2=2.21.5-1+cuda12.4 2024-12-18T00:27:53.7793678Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-12-18T00:27:53.7794051Z NO_TEST_TIMEOUT=False 2024-12-18T00:27:53.7794313Z TD_DISTRIBUTED=False 2024-12-18T00:27:53.7794590Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.4.1-1 2024-12-18T00:27:53.7794920Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T00:27:53.7795225Z NV_NVPROF_VERSION=12.4.127-1 2024-12-18T00:27:53.7795502Z GITHUB_RETENTION_DAYS=90 2024-12-18T00:27:53.7795778Z OPENSSL_DIR=/opt/openssl 2024-12-18T00:27:53.7796051Z GITHUB_ACTION_REPOSITORY= 2024-12-18T00:27:53.7796795Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:27:53.7797547Z GITHUB_BASE_REF= 2024-12-18T00:27:53.7798010Z ARTIFACTS_FILE_SUFFIX=test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T00:27:53.7798690Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-12-18T00:27:53.7798980Z CI=true 2024-12-18T00:27:53.7799217Z NV_LIBNCCL_PACKAGE_VERSION=2.21.5-1 2024-12-18T00:27:53.7799538Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T00:27:53.7799816Z JOB_ID=34567098055 2024-12-18T00:27:53.7800065Z INSTALLED_PROTOBUF=yes 2024-12-18T00:27:53.7800325Z GITHUB_HEAD_REF= 2024-12-18T00:27:53.7800568Z GITHUB_ACTION_REF= 2024-12-18T00:27:53.7800873Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T00:27:53.7801222Z TEST_SHOWLOCALS=False 2024-12-18T00:27:53.7801486Z GITHUB_WORKFLOW=inductor 2024-12-18T00:27:53.7801771Z DEBIAN_FRONTEND=noninteractive 2024-12-18T00:27:53.7802387Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c27926df-acc4-4189-9501-91ef764d90e6 2024-12-18T00:27:53.7803083Z NO_TD=False 2024-12-18T00:27:53.7803349Z SKIP_SCCACHE_INITIALIZATION=1 2024-12-18T00:27:53.7803630Z _=/usr/bin/env 2024-12-18T00:27:53.7803871Z + echo 'Testing pytorch' 2024-12-18T00:27:53.7804134Z Testing pytorch 2024-12-18T00:27:53.7804393Z + export LANG=C.UTF-8 2024-12-18T00:27:53.7804636Z + LANG=C.UTF-8 2024-12-18T00:27:53.7804864Z + PR_NUMBER= 2024-12-18T00:27:53.7805127Z + [[ inductor_distributed == \d\e\f\a\u\l\t ]] 2024-12-18T00:27:53.7805501Z + [[ inductor_distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T00:27:53.7805862Z + [[ inductor_distributed == \s\l\o\w ]] 2024-12-18T00:27:53.7806274Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-12-18T00:27:53.7806731Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *cuda* ]] 2024-12-18T00:27:53.7807125Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:27:53.7807569Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T00:27:53.7807910Z + [[ inductor_distributed == *crossref* ]] 2024-12-18T00:27:53.7808281Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *rocm* ]] 2024-12-18T00:27:53.7808693Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *xpu* ]] 2024-12-18T00:27:53.7809120Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-12-18T00:27:53.7809511Z + pip_install --user ninja==1.10.2 2024-12-18T00:27:53.7809905Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:27:53.7810380Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T00:27:56.0338234Z Collecting ninja==1.10.2 2024-12-18T00:27:56.0518585Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T00:27:56.0727337Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T00:27:56.6278345Z Installing collected packages: ninja 2024-12-18T00:27:56.6358304Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T00:27:56.6359826Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T00:27:56.6421540Z Successfully installed ninja-1.10.2 2024-12-18T00:27:56.7266473Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:27:56.7268137Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:27:56.7269082Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-12-18T00:27:56.7269495Z + install_tlparse 2024-12-18T00:27:56.7269763Z + pip_install --user tlparse==0.3.25 2024-12-18T00:27:56.7270175Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T00:27:56.7270663Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T00:27:57.1123128Z Collecting tlparse==0.3.25 2024-12-18T00:27:57.1295391Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T00:27:57.1426996Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T00:27:57.7306099Z Installing collected packages: tlparse 2024-12-18T00:27:57.7652013Z Successfully installed tlparse-0.3.25 2024-12-18T00:27:57.8525777Z ++ python -m site --user-base 2024-12-18T00:27:57.8769133Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T00:27:57.8770853Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *asan* ]] 2024-12-18T00:27:57.8771406Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-debug* ]] 2024-12-18T00:27:57.8771854Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-12-18T00:27:57.8772452Z + echo 'We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-12-18T00:27:57.8773171Z We are not in debug mode: linux-focal-cuda12.4-py3.10-gcc9-sm86. Expect the assertion to pass 2024-12-18T00:27:57.8775809Z + cd test 2024-12-18T00:27:57.8776168Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T00:27:59.5602750Z + [[ inductor_distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T00:27:59.5603287Z + [[ inductor_distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T00:27:59.5608575Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T00:27:59.5609607Z + [[ inductor_distributed == *pr_time_benchmarks* ]] 2024-12-18T00:27:59.5610116Z + [[ inductor_distributed == *dynamo_eager* ]] 2024-12-18T00:27:59.5610585Z + [[ inductor_distributed == *aot_eager* ]] 2024-12-18T00:27:59.5613569Z + [[ inductor_distributed == *aot_inductor* ]] 2024-12-18T00:27:59.5613953Z + [[ inductor_distributed == *inductor* ]] 2024-12-18T00:27:59.5614293Z + [[ inductor_distributed != *perf* ]] 2024-12-18T00:27:59.5614630Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-12-18T00:27:59.5614951Z + [[ inductor_distributed == *dynamic* ]] 2024-12-18T00:27:59.5615289Z + [[ inductor_distributed == *cpu* ]] 2024-12-18T00:27:59.5615618Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T00:27:59.5641423Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-12-18T00:27:59.5641892Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-12-18T00:27:59.5645384Z + cd test 2024-12-18T00:27:59.5645715Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T00:28:01.5033143Z PyTorch built with: 2024-12-18T00:28:01.5033424Z - GCC 9.4 2024-12-18T00:28:01.5033653Z - C++ Version: 201703 2024-12-18T00:28:01.5034232Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:28:01.5034928Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:28:01.5035371Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:28:01.5035710Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T00:28:01.5036046Z - NNPACK is enabled 2024-12-18T00:28:01.5036328Z - CPU capability usage: AVX2 2024-12-18T00:28:01.5036610Z - CUDA Runtime 12.4 2024-12-18T00:28:01.5036962Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-12-18T00:28:01.5037342Z - CuDNN 90.1 2024-12-18T00:28:01.5037571Z - Magma 2.6.1 2024-12-18T00:28:01.5042256Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CUDA_VERSION=12.4, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T00:28:01.5047589Z 2024-12-18T00:28:01.9625423Z + cd test 2024-12-18T00:28:01.9625788Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T00:28:03.3397634Z ATen/Parallel: 2024-12-18T00:28:03.3398212Z at::get_num_threads() : 24 2024-12-18T00:28:03.3398809Z at::get_num_interop_threads() : 48 2024-12-18T00:28:03.3399434Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T00:28:03.3400037Z omp_get_max_threads() : 24 2024-12-18T00:28:03.3401040Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T00:28:03.3401615Z mkl_get_max_threads() : 24 2024-12-18T00:28:03.3402020Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T00:28:03.3402467Z std::thread::hardware_concurrency() : 48 2024-12-18T00:28:03.3402801Z Environment variables: 2024-12-18T00:28:03.3403149Z OMP_NUM_THREADS : [not set] 2024-12-18T00:28:03.3403446Z MKL_NUM_THREADS : [not set] 2024-12-18T00:28:03.3403751Z ATen parallel backend: OpenMP 2024-12-18T00:28:03.3403940Z 2024-12-18T00:28:03.6305596Z + [[ inductor_distributed == *numpy_2* ]] 2024-12-18T00:28:03.6306175Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *aarch64* ]] 2024-12-18T00:28:03.6306600Z + [[ inductor_distributed == *backward* ]] 2024-12-18T00:28:03.6306933Z + [[ inductor_distributed == *xla* ]] 2024-12-18T00:28:03.6307264Z + [[ inductor_distributed == *executorch* ]] 2024-12-18T00:28:03.6307628Z + [[ inductor_distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T00:28:03.6308051Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 == *libtorch* ]] 2024-12-18T00:28:03.6308453Z + [[ inductor_distributed == distributed ]] 2024-12-18T00:28:03.6308846Z + [[ inductor_distributed == *inductor_distributed* ]] 2024-12-18T00:28:03.6309207Z + test_inductor_distributed 2024-12-18T00:28:03.6309553Z + echo 'Testing multi_gpu tests in test_torchinductor' 2024-12-18T00:28:03.6309947Z Testing multi_gpu tests in test_torchinductor 2024-12-18T00:28:03.6310467Z + python test/run_test.py -i inductor/test_torchinductor.py -k test_multi_gpu --verbose 2024-12-18T00:28:03.7308151Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:28:03.7308950Z import pkg_resources 2024-12-18T00:28:07.4547451Z No TD results found 2024-12-18T00:28:07.4548412Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:28:07.5060024Z Ignoring disabled issues: [''] 2024-12-18T00:28:07.5237485Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:07.5838277Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:07.5860755Z Running all tests 2024-12-18T00:28:07.5864071Z Running parallel tests on 3 processes 2024-12-18T00:28:07.5864502Z Name: tests to run (est. time: 0.08min) 2024-12-18T00:28:07.5864899Z Serial tests (1): 2024-12-18T00:28:07.5865171Z inductor/test_torchinductor 1/1 2024-12-18T00:28:07.5865483Z Parallel tests (0): 2024-12-18T00:28:07.5866062Z Name: excluded (est. time: 0.0min) 2024-12-18T00:28:07.5866362Z Serial tests (0): 2024-12-18T00:28:07.5866607Z Parallel tests (0): 2024-12-18T00:28:07.5935640Z Running inductor/test_torchinductor 1/1 ... [2024-12-18 00:28:07.593241] 2024-12-18T00:28:07.5936095Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:07.5939077Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_torchinductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_multi_gpu', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:07.593575] 2024-12-18T00:28:22.1840534Z 2024-12-18T00:28:22.1841455Z inductor/test_torchinductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_torchinductor_1.1_c6d3724247314335_.log 2024-12-18T00:28:22.1843497Z Running 4 items in this shard: test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_device_cpu, test/inductor/test_torchinductor.py::CpuTests::test_multi_gpu_recompile_on_index_cpu, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_device_cuda, test/inductor/test_torchinductor.py::GPUTests::test_multi_gpu_recompile_on_index_cuda 2024-12-18T00:28:22.1844836Z 2024-12-18T00:28:22.9289787Z Running test batch 'tests to run' cost 15.34 seconds 2024-12-18T00:28:23.7689192Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_non_default_cuda_device --verbose 2024-12-18T00:28:23.8675376Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:28:23.8676951Z import pkg_resources 2024-12-18T00:28:27.4808429Z No TD results found 2024-12-18T00:28:27.4809401Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:28:27.4970387Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:27.5595466Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:27.5616650Z Running all tests 2024-12-18T00:28:27.5621597Z Running parallel tests on 3 processes 2024-12-18T00:28:27.5622309Z Name: tests to run (est. time: 0.16min) 2024-12-18T00:28:27.5622933Z Serial tests (1): 2024-12-18T00:28:27.5623487Z inductor/test_aot_inductor 1/1 2024-12-18T00:28:27.5624096Z Parallel tests (0): 2024-12-18T00:28:27.5624628Z Name: excluded (est. time: 0.0min) 2024-12-18T00:28:27.5625212Z Serial tests (0): 2024-12-18T00:28:27.5625706Z Parallel tests (0): 2024-12-18T00:28:27.5683882Z Running inductor/test_aot_inductor 1/1 ... [2024-12-18 00:28:27.568075] 2024-12-18T00:28:27.5684337Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:27.5688070Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_non_default_cuda_device', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:27.568445] 2024-12-18T00:28:35.5473452Z 2024-12-18T00:28:35.5474494Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_345ce8d03071aaf8_.log 2024-12-18T00:28:35.5475287Z Running 0 items in this shard: 2024-12-18T00:28:35.5475489Z 2024-12-18T00:28:36.3181789Z Running test batch 'tests to run' cost 8.76 seconds 2024-12-18T00:28:37.1767821Z + python test/run_test.py -i inductor/test_aot_inductor.py -k test_replicate_on_devices --verbose 2024-12-18T00:28:37.2754417Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:28:37.2755953Z import pkg_resources 2024-12-18T00:28:40.8475708Z No TD results found 2024-12-18T00:28:40.8476712Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:28:40.8636935Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:40.9265848Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:28:40.9289514Z Running all tests 2024-12-18T00:28:40.9292439Z Running parallel tests on 3 processes 2024-12-18T00:28:40.9292944Z Name: tests to run (est. time: 0.16min) 2024-12-18T00:28:40.9293371Z Serial tests (1): 2024-12-18T00:28:40.9293641Z inductor/test_aot_inductor 1/1 2024-12-18T00:28:40.9293938Z Parallel tests (0): 2024-12-18T00:28:40.9294209Z Name: excluded (est. time: 0.0min) 2024-12-18T00:28:40.9294499Z Serial tests (0): 2024-12-18T00:28:40.9294745Z Parallel tests (0): 2024-12-18T00:28:40.9359313Z Running inductor/test_aot_inductor 1/1 ... [2024-12-18 00:28:40.935528] 2024-12-18T00:28:40.9359873Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:28:40.9363298Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'inductor/test_aot_inductor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_replicate_on_devices', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:28:40.935887] 2024-12-18T00:28:58.0302238Z 2024-12-18T00:28:58.0303226Z inductor/test_aot_inductor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/inductor.test_aot_inductor_1.1_e7b1f1431993dccd_.log 2024-12-18T00:28:58.0304902Z Running 2 items in this shard: test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_replicate_on_devices_cpu, test/inductor/test_aot_inductor.py::AOTInductorTestABICompatibleGpu::test_replicate_on_devices_cuda 2024-12-18T00:28:58.0305841Z 2024-12-18T00:28:58.7468534Z Running test batch 'tests to run' cost 17.82 seconds 2024-12-18T00:28:59.5415389Z + python test/run_test.py -i distributed/test_c10d_functional_native.py --verbose 2024-12-18T00:28:59.6400430Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:28:59.6402044Z import pkg_resources 2024-12-18T00:29:03.2215811Z No TD results found 2024-12-18T00:29:03.2216774Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:29:03.2379222Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:29:03.2998440Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:29:03.3022283Z Running all tests 2024-12-18T00:29:03.3025567Z Running parallel tests on 3 processes 2024-12-18T00:29:03.3025938Z Name: tests to run (est. time: 2.69min) 2024-12-18T00:29:03.3026297Z Serial tests (1): 2024-12-18T00:29:03.3026593Z distributed/test_c10d_functional_native 1/1 2024-12-18T00:29:03.3026934Z Parallel tests (0): 2024-12-18T00:29:03.3027208Z Name: excluded (est. time: 0.0min) 2024-12-18T00:29:03.3027509Z Serial tests (0): 2024-12-18T00:29:03.3028045Z Parallel tests (0): 2024-12-18T00:29:03.3088737Z Running distributed/test_c10d_functional_native 1/1 ... [2024-12-18 00:29:03.308567] 2024-12-18T00:29:03.3089217Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:29:03.3092677Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:29:03.308928] 2024-12-18T00:31:35.6797747Z 2024-12-18T00:31:35.6799019Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_afe1081bc25be411_.log 2024-12-18T00:31:35.6809682Z Running 28 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2024-12-18T00:31:35.6819358Z 2024-12-18T00:31:36.4346888Z Running test batch 'tests to run' cost 153.13 seconds 2024-12-18T00:31:37.2574411Z + python test/run_test.py -i distributed/_tensor/test_dtensor_compile.py --verbose 2024-12-18T00:31:37.3560317Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:31:37.3561911Z import pkg_resources 2024-12-18T00:31:40.9321530Z No TD results found 2024-12-18T00:31:40.9323604Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:31:40.9483686Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:31:41.0112713Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:31:41.0136783Z Running all tests 2024-12-18T00:31:41.0140261Z Running parallel tests on 3 processes 2024-12-18T00:31:41.0140642Z Name: tests to run (est. time: 0.98min) 2024-12-18T00:31:41.0140955Z Serial tests (1): 2024-12-18T00:31:41.0141251Z distributed/_tensor/test_dtensor_compile 1/1 2024-12-18T00:31:41.0141596Z Parallel tests (0): 2024-12-18T00:31:41.0141876Z Name: excluded (est. time: 0.0min) 2024-12-18T00:31:41.0142179Z Serial tests (0): 2024-12-18T00:31:41.0142430Z Parallel tests (0): 2024-12-18T00:31:41.0208867Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-12-18 00:31:41.020551] 2024-12-18T00:31:41.0209364Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:31:41.0212648Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:31:41.020915] 2024-12-18T00:32:43.6513984Z 2024-12-18T00:32:43.6515719Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_08bcad7095619d2b_.log 2024-12-18T00:32:43.6528816Z Running 30 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-12-18T00:32:43.6541451Z 2024-12-18T00:32:44.4161970Z Running test batch 'tests to run' cost 63.4 seconds 2024-12-18T00:32:45.2888401Z + python test/run_test.py -i distributed/tensor/parallel/test_micro_pipeline_tp.py --verbose 2024-12-18T00:32:45.3872284Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:32:45.3873096Z import pkg_resources 2024-12-18T00:32:48.9646922Z No TD results found 2024-12-18T00:32:48.9648397Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:32:48.9808692Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:32:49.0428361Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:32:49.0449822Z Running all tests 2024-12-18T00:32:49.0452846Z Running parallel tests on 3 processes 2024-12-18T00:32:49.0453194Z Name: tests to run (est. time: 0.19min) 2024-12-18T00:32:49.0453508Z Serial tests (1): 2024-12-18T00:32:49.0453843Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-12-18T00:32:49.0454234Z Parallel tests (0): 2024-12-18T00:32:49.0454512Z Name: excluded (est. time: 0.0min) 2024-12-18T00:32:49.0454810Z Serial tests (0): 2024-12-18T00:32:49.0455067Z Parallel tests (0): 2024-12-18T00:32:49.0517814Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-12-18 00:32:49.051374] 2024-12-18T00:32:49.0518386Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:32:49.0520703Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:32:49.051709] 2024-12-18T00:33:05.5449837Z 2024-12-18T00:33:05.5451162Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_b2d04a4a17757235_.log 2024-12-18T00:33:05.5467375Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-12-18T00:33:05.5481917Z 2024-12-18T00:33:06.3063313Z Running test batch 'tests to run' cost 17.26 seconds 2024-12-18T00:33:07.1481822Z + python test/run_test.py -i distributed/_composable/test_replicate_with_compiler.py --verbose 2024-12-18T00:33:07.2469087Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:33:07.2469902Z import pkg_resources 2024-12-18T00:33:10.7486350Z No TD results found 2024-12-18T00:33:10.7487320Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:33:10.7649536Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:33:10.8270621Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:33:10.8292477Z Running all tests 2024-12-18T00:33:10.8295957Z Running parallel tests on 3 processes 2024-12-18T00:33:10.8296303Z Name: tests to run (est. time: 2.07min) 2024-12-18T00:33:10.8296614Z Serial tests (1): 2024-12-18T00:33:10.8296954Z distributed/_composable/test_replicate_with_compiler 1/1 2024-12-18T00:33:10.8297345Z Parallel tests (0): 2024-12-18T00:33:10.8297627Z Name: excluded (est. time: 0.0min) 2024-12-18T00:33:10.8297927Z Serial tests (0): 2024-12-18T00:33:10.8298179Z Parallel tests (0): 2024-12-18T00:33:10.8361264Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2024-12-18 00:33:10.835811] 2024-12-18T00:33:10.8361804Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:33:10.8364873Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:33:10.836148] 2024-12-18T00:35:17.0862059Z 2024-12-18T00:35:17.0863155Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_efe28b3877a95fc7_.log 2024-12-18T00:35:17.0868069Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp 2024-12-18T00:35:17.0871838Z 2024-12-18T00:35:17.8358893Z Running test batch 'tests to run' cost 127.01 seconds 2024-12-18T00:35:18.6602366Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_comm.py --verbose 2024-12-18T00:35:18.7589509Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:35:18.7590975Z import pkg_resources 2024-12-18T00:35:22.2885098Z No TD results found 2024-12-18T00:35:22.2886039Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:35:22.3047488Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:35:22.3664438Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:35:22.3687984Z Running all tests 2024-12-18T00:35:22.3692538Z Running parallel tests on 3 processes 2024-12-18T00:35:22.3692899Z Name: tests to run (est. time: 2.99min) 2024-12-18T00:35:22.3693208Z Serial tests (1): 2024-12-18T00:35:22.3697503Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-12-18T00:35:22.3697933Z Parallel tests (0): 2024-12-18T00:35:22.3698216Z Name: excluded (est. time: 0.0min) 2024-12-18T00:35:22.3698515Z Serial tests (0): 2024-12-18T00:35:22.3698772Z Parallel tests (0): 2024-12-18T00:35:22.3756382Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-12-18 00:35:22.375311] 2024-12-18T00:35:22.3756927Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:35:22.3760011Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:35:22.375655] 2024-12-18T00:38:25.1983175Z 2024-12-18T00:38:25.1984272Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_052af212cf500711_.log 2024-12-18T00:38:25.1995259Z Running 15 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init 2024-12-18T00:38:25.2002633Z 2024-12-18T00:38:25.9673169Z Running test batch 'tests to run' cost 183.6 seconds 2024-12-18T00:38:26.8245224Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_multi_group --verbose 2024-12-18T00:38:26.9228515Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:38:26.9229355Z import pkg_resources 2024-12-18T00:38:30.5457784Z No TD results found 2024-12-18T00:38:30.5458739Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:38:30.5619262Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:38:30.6234953Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:38:30.6257582Z Running all tests 2024-12-18T00:38:30.6260914Z Running parallel tests on 3 processes 2024-12-18T00:38:30.6261273Z Name: tests to run (est. time: 3.85min) 2024-12-18T00:38:30.6261581Z Serial tests (1): 2024-12-18T00:38:30.6261928Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:38:30.6262324Z Parallel tests (0): 2024-12-18T00:38:30.6262597Z Name: excluded (est. time: 0.0min) 2024-12-18T00:38:30.6262893Z Serial tests (0): 2024-12-18T00:38:30.6263144Z Parallel tests (0): 2024-12-18T00:38:30.6327091Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 00:38:30.632305] 2024-12-18T00:38:30.6327838Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:38:30.6329910Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_multi_group', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:38:30.632639] 2024-12-18T00:43:31.5723786Z 2024-12-18T00:43:31.5728655Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_a3139a56c0a22b59_.log 2024-12-18T00:43:31.5731458Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_unshard_async_op 2024-12-18T00:43:31.5733244Z 2024-12-18T00:43:32.3676588Z Running test batch 'tests to run' cost 301.74 seconds 2024-12-18T00:43:33.2706879Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_with_activation_checkpointing --verbose 2024-12-18T00:43:33.3694126Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:43:33.3695610Z import pkg_resources 2024-12-18T00:43:36.9531495Z No TD results found 2024-12-18T00:43:36.9533349Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:43:36.9693317Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:43:37.0311456Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:43:37.0335112Z Running all tests 2024-12-18T00:43:37.0338328Z Running parallel tests on 3 processes 2024-12-18T00:43:37.0338712Z Name: tests to run (est. time: 3.85min) 2024-12-18T00:43:37.0339138Z Serial tests (1): 2024-12-18T00:43:37.0339567Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:43:37.0340062Z Parallel tests (0): 2024-12-18T00:43:37.0340382Z Name: excluded (est. time: 0.0min) 2024-12-18T00:43:37.0340981Z Serial tests (0): 2024-12-18T00:43:37.0341237Z Parallel tests (0): 2024-12-18T00:43:37.0401548Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 00:43:37.039680] 2024-12-18T00:43:37.0402637Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:43:37.0405362Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_with_activation_checkpointing', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:43:37.040020] 2024-12-18T00:44:29.5408395Z 2024-12-18T00:44:29.5410484Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_095a1e85c2a8db65_.log 2024-12-18T00:44:29.5412867Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing 2024-12-18T00:44:29.5413715Z 2024-12-18T00:44:30.2998819Z Running test batch 'tests to run' cost 53.27 seconds 2024-12-18T00:44:31.1899361Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_hsdp --verbose 2024-12-18T00:44:31.2881976Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:44:31.2883750Z import pkg_resources 2024-12-18T00:44:34.8789680Z No TD results found 2024-12-18T00:44:34.8790636Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:44:34.8951601Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:44:34.9571432Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:44:34.9594447Z Running all tests 2024-12-18T00:44:34.9598532Z Running parallel tests on 3 processes 2024-12-18T00:44:34.9598891Z Name: tests to run (est. time: 3.85min) 2024-12-18T00:44:34.9599220Z Serial tests (1): 2024-12-18T00:44:34.9599574Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:44:34.9599969Z Parallel tests (0): 2024-12-18T00:44:34.9600238Z Name: excluded (est. time: 0.0min) 2024-12-18T00:44:34.9600539Z Serial tests (0): 2024-12-18T00:44:34.9600791Z Parallel tests (0): 2024-12-18T00:44:34.9664064Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 00:44:34.965985] 2024-12-18T00:44:34.9664630Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:44:34.9667407Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_hsdp', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:44:34.966324] 2024-12-18T00:44:56.0218459Z 2024-12-18T00:44:56.0219774Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_42f80aa0778ee104_.log 2024-12-18T00:44:56.0221256Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp 2024-12-18T00:44:56.0221888Z 2024-12-18T00:44:56.7953120Z Running test batch 'tests to run' cost 21.84 seconds 2024-12-18T00:44:57.6901070Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_train_parity_2d_transformer_checkpoint_resume --verbose 2024-12-18T00:44:57.7892870Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:44:57.7893687Z import pkg_resources 2024-12-18T00:45:01.4131896Z No TD results found 2024-12-18T00:45:01.4133835Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:45:01.4296024Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:01.4916950Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:01.4939766Z Running all tests 2024-12-18T00:45:01.4943213Z Running parallel tests on 3 processes 2024-12-18T00:45:01.4943553Z Name: tests to run (est. time: 3.85min) 2024-12-18T00:45:01.4943866Z Serial tests (1): 2024-12-18T00:45:01.4944211Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:45:01.4944604Z Parallel tests (0): 2024-12-18T00:45:01.4944876Z Name: excluded (est. time: 0.0min) 2024-12-18T00:45:01.4945164Z Serial tests (0): 2024-12-18T00:45:01.4945411Z Parallel tests (0): 2024-12-18T00:45:01.5007325Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 00:45:01.500323] 2024-12-18T00:45:01.5007921Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:45:01.5010714Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_train_parity_2d_transformer_checkpoint_resume', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:45:01.500688] 2024-12-18T00:45:06.1732772Z 2024-12-18T00:45:06.1734046Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_9f93ea92865eb61a_.log 2024-12-18T00:45:06.1735042Z Running 0 items in this shard: 2024-12-18T00:45:06.1735234Z 2024-12-18T00:45:06.9529807Z Running test batch 'tests to run' cost 5.46 seconds 2024-12-18T00:45:07.8468680Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_training.py -k test_gradient_accumulation --verbose 2024-12-18T00:45:07.9450270Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:45:07.9451898Z import pkg_resources 2024-12-18T00:45:11.5179318Z No TD results found 2024-12-18T00:45:11.5180271Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:45:11.5342094Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:11.5960952Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:11.5986523Z Running all tests 2024-12-18T00:45:11.5989892Z Running parallel tests on 3 processes 2024-12-18T00:45:11.5990512Z Name: tests to run (est. time: 3.85min) 2024-12-18T00:45:11.5991047Z Serial tests (1): 2024-12-18T00:45:11.5991674Z distributed/_composable/fsdp/test_fully_shard_training 1/1 2024-12-18T00:45:11.5992073Z Parallel tests (0): 2024-12-18T00:45:11.5992347Z Name: excluded (est. time: 0.0min) 2024-12-18T00:45:11.5992649Z Serial tests (0): 2024-12-18T00:45:11.5992900Z Parallel tests (0): 2024-12-18T00:45:11.6057558Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2024-12-18 00:45:11.605411] 2024-12-18T00:45:11.6058099Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:45:11.6061253Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_gradient_accumulation', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:45:11.605772] 2024-12-18T00:45:53.2405483Z 2024-12-18T00:45:53.2406776Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_19d737b752a6fbd5_.log 2024-12-18T00:45:53.2408291Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation 2024-12-18T00:45:53.2408956Z 2024-12-18T00:45:53.9817022Z Running test batch 'tests to run' cost 42.38 seconds 2024-12-18T00:45:54.9266113Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_state_dict.py -k test_dp_state_dict_save_load --verbose 2024-12-18T00:45:55.0253935Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:45:55.0254769Z import pkg_resources 2024-12-18T00:45:58.6305885Z No TD results found 2024-12-18T00:45:58.6306841Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:45:58.6467552Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:58.7093271Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:45:58.7115407Z Running all tests 2024-12-18T00:45:58.7118766Z Running parallel tests on 3 processes 2024-12-18T00:45:58.7119143Z Name: tests to run (est. time: 0.16min) 2024-12-18T00:45:58.7119461Z Serial tests (1): 2024-12-18T00:45:58.7119823Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-12-18T00:45:58.7120233Z Parallel tests (0): 2024-12-18T00:45:58.7120508Z Name: excluded (est. time: 0.0min) 2024-12-18T00:45:58.7120806Z Serial tests (0): 2024-12-18T00:45:58.7121064Z Parallel tests (0): 2024-12-18T00:45:58.7182380Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-12-18 00:45:58.717826] 2024-12-18T00:45:58.7182965Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:45:58.7185436Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_dp_state_dict_save_load', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:45:58.718166] 2024-12-18T00:46:12.7061778Z 2024-12-18T00:46:12.7062924Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_5ac29da3b0a6b299_.log 2024-12-18T00:46:12.7064641Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load 2024-12-18T00:46:12.7066285Z 2024-12-18T00:46:13.4612248Z Running test batch 'tests to run' cost 14.75 seconds 2024-12-18T00:46:14.3772619Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_frozen.py --verbose 2024-12-18T00:46:14.4762156Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:46:14.4763026Z import pkg_resources 2024-12-18T00:46:18.1012603Z No TD results found 2024-12-18T00:46:18.1013572Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:46:18.1175529Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:46:18.1797674Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:46:18.1819987Z Running all tests 2024-12-18T00:46:18.1824842Z Running parallel tests on 3 processes 2024-12-18T00:46:18.1825527Z Name: tests to run (est. time: 0.48min) 2024-12-18T00:46:18.1826077Z Serial tests (1): 2024-12-18T00:46:18.1826697Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-12-18T00:46:18.1827399Z Parallel tests (0): 2024-12-18T00:46:18.1828305Z Name: excluded (est. time: 0.0min) 2024-12-18T00:46:18.1828612Z Serial tests (0): 2024-12-18T00:46:18.1828852Z Parallel tests (0): 2024-12-18T00:46:18.1886670Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-12-18 00:46:18.188283] 2024-12-18T00:46:18.1887226Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:46:18.1889921Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:46:18.188625] 2024-12-18T00:46:51.7108108Z 2024-12-18T00:46:51.7110023Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_adb022b549e3503a_.log 2024-12-18T00:46:51.7114720Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-12-18T00:46:51.7116748Z 2024-12-18T00:46:52.4762447Z Running test batch 'tests to run' cost 34.29 seconds 2024-12-18T00:46:53.3649877Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_compute_dtype --verbose 2024-12-18T00:46:53.4639133Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:46:53.4639938Z import pkg_resources 2024-12-18T00:46:57.0785686Z No TD results found 2024-12-18T00:46:57.0786626Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:46:57.0948145Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:46:57.1579464Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:46:57.1601840Z Running all tests 2024-12-18T00:46:57.1605258Z Running parallel tests on 3 processes 2024-12-18T00:46:57.1605589Z Name: tests to run (est. time: 0.37min) 2024-12-18T00:46:57.1605906Z Serial tests (1): 2024-12-18T00:46:57.1606273Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-12-18T00:46:57.1606697Z Parallel tests (0): 2024-12-18T00:46:57.1606981Z Name: excluded (est. time: 0.0min) 2024-12-18T00:46:57.1607271Z Serial tests (0): 2024-12-18T00:46:57.1607522Z Parallel tests (0): 2024-12-18T00:46:57.1669008Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-12-18 00:46:57.166474] 2024-12-18T00:46:57.1669619Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:46:57.1671611Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_compute_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:46:57.166811] 2024-12-18T00:47:12.9088981Z 2024-12-18T00:47:12.9091240Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_00a606aa9a3fb8c6_.log 2024-12-18T00:47:12.9093281Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype 2024-12-18T00:47:12.9093953Z 2024-12-18T00:47:13.6753891Z Running test batch 'tests to run' cost 16.51 seconds 2024-12-18T00:47:14.5823415Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_mixed_precision.py -k test_reduce_dtype --verbose 2024-12-18T00:47:14.6807163Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:47:14.6807979Z import pkg_resources 2024-12-18T00:47:18.3084213Z No TD results found 2024-12-18T00:47:18.3086117Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:47:18.3246561Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:47:18.3865564Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:47:18.3889230Z Running all tests 2024-12-18T00:47:18.3892201Z Running parallel tests on 3 processes 2024-12-18T00:47:18.3892576Z Name: tests to run (est. time: 0.37min) 2024-12-18T00:47:18.3892920Z Serial tests (1): 2024-12-18T00:47:18.3893286Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-12-18T00:47:18.3893708Z Parallel tests (0): 2024-12-18T00:47:18.3893981Z Name: excluded (est. time: 0.0min) 2024-12-18T00:47:18.3894278Z Serial tests (0): 2024-12-18T00:47:18.3894527Z Parallel tests (0): 2024-12-18T00:47:18.3957362Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-12-18 00:47:18.395344] 2024-12-18T00:47:18.3957958Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:47:18.3960488Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_reduce_dtype', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:47:18.395682] 2024-12-18T00:47:33.8866714Z 2024-12-18T00:47:33.8868195Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_b9bbfa7d8ded0659_.log 2024-12-18T00:47:33.8869772Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype 2024-12-18T00:47:33.8870429Z 2024-12-18T00:47:34.6397042Z Running test batch 'tests to run' cost 16.25 seconds 2024-12-18T00:47:35.4985495Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py -k test_clip_grad_norm_2d --verbose 2024-12-18T00:47:35.5968057Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:47:35.5968882Z import pkg_resources 2024-12-18T00:47:39.2103128Z No TD results found 2024-12-18T00:47:39.2104052Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:47:39.2266846Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:47:39.2882483Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:47:39.2905673Z Running all tests 2024-12-18T00:47:39.2908822Z Running parallel tests on 3 processes 2024-12-18T00:47:39.2909167Z Name: tests to run (est. time: 0.19min) 2024-12-18T00:47:39.2909488Z Serial tests (1): 2024-12-18T00:47:39.2909847Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-12-18T00:47:39.2910262Z Parallel tests (0): 2024-12-18T00:47:39.2910536Z Name: excluded (est. time: 0.0min) 2024-12-18T00:47:39.2910831Z Serial tests (0): 2024-12-18T00:47:39.2911077Z Parallel tests (0): 2024-12-18T00:47:39.2975214Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-12-18 00:47:39.297170] 2024-12-18T00:47:39.2975779Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:47:39.2978956Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_clip_grad_norm_2d', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:47:39.297505] 2024-12-18T00:47:55.5408890Z 2024-12-18T00:47:55.5410493Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_4934dbbc4babbb28_.log 2024-12-18T00:47:55.5415639Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-12-18T00:47:55.5416274Z 2024-12-18T00:47:56.2921471Z Running test batch 'tests to run' cost 17.0 seconds 2024-12-18T00:47:57.1700951Z + python test/run_test.py -i distributed/_composable/fsdp/test_fully_shard_compile.py --verbose 2024-12-18T00:47:57.2689403Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:47:57.2690355Z import pkg_resources 2024-12-18T00:48:00.8746604Z No TD results found 2024-12-18T00:48:00.8749018Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:48:00.8909316Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:48:00.9527752Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:48:00.9550877Z Running all tests 2024-12-18T00:48:00.9554558Z Running parallel tests on 3 processes 2024-12-18T00:48:00.9555040Z Name: tests to run (est. time: 5.37min) 2024-12-18T00:48:00.9555480Z Serial tests (1): 2024-12-18T00:48:00.9555946Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 2024-12-18T00:48:00.9556372Z Parallel tests (0): 2024-12-18T00:48:00.9556656Z Name: excluded (est. time: 0.0min) 2024-12-18T00:48:00.9556970Z Serial tests (0): 2024-12-18T00:48:00.9557227Z Parallel tests (0): 2024-12-18T00:48:00.9618301Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2024-12-18 00:48:00.961492] 2024-12-18T00:48:00.9619050Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:48:00.9622003Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:48:00.961824] 2024-12-18T00:51:36.5665670Z 2024-12-18T00:51:36.5667246Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_736d1569e7d44a0f_.log 2024-12-18T00:51:36.5677985Z Running 16 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True 2024-12-18T00:51:36.5691263Z 2024-12-18T00:51:37.3259765Z Running test batch 'tests to run' cost 216.37 seconds 2024-12-18T00:51:38.2600354Z + python test/run_test.py -i distributed/fsdp/test_fsdp_tp_integration.py -k test_fsdp_tp_integration --verbose 2024-12-18T00:51:38.3590438Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:51:38.3591250Z import pkg_resources 2024-12-18T00:51:41.9317110Z No TD results found 2024-12-18T00:51:41.9319042Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:51:41.9476930Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:51:42.0099556Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:51:42.0123792Z Running all tests 2024-12-18T00:51:42.0127732Z Running parallel tests on 3 processes 2024-12-18T00:51:42.0128130Z Name: tests to run (est. time: 0.47min) 2024-12-18T00:51:42.0128447Z Serial tests (1): 2024-12-18T00:51:42.0128753Z distributed/fsdp/test_fsdp_tp_integration 1/1 2024-12-18T00:51:42.0129145Z Parallel tests (0): 2024-12-18T00:51:42.0129451Z Name: excluded (est. time: 0.0min) 2024-12-18T00:51:42.0129755Z Serial tests (0): 2024-12-18T00:51:42.0130292Z Parallel tests (0): 2024-12-18T00:51:42.0189718Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2024-12-18 00:51:42.018610] 2024-12-18T00:51:42.0190274Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:51:42.0193694Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_fsdp_tp_integration', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:51:42.018948] 2024-12-18T00:52:14.5915300Z 2024-12-18T00:52:14.5917581Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_680adda8ff934fc7_.log 2024-12-18T00:52:14.5920372Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state 2024-12-18T00:52:14.5922021Z 2024-12-18T00:52:15.3302086Z Running test batch 'tests to run' cost 33.32 seconds 2024-12-18T00:52:16.2252812Z + python test/run_test.py --include distributed/test_dynamo_distributed distributed/test_inductor_collectives distributed/test_compute_comm_reordering --verbose 2024-12-18T00:52:16.3237394Z /var/lib/jenkins/workspace/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T00:52:16.3238217Z import pkg_resources 2024-12-18T00:52:19.9201017Z No TD results found 2024-12-18T00:52:19.9202864Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-12-18T00:52:19.9363119Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:52:19.9982095Z ##[warning] Gathered no stats from artifacts for linux-focal-cuda12.4-py3.10-gcc9-sm86 build env and inductor_distributed test config. Using default build env and inductor_distributed test config instead. 2024-12-18T00:52:20.0004851Z Running all tests 2024-12-18T00:52:20.0008647Z Running parallel tests on 3 processes 2024-12-18T00:52:20.0009106Z Name: tests to run (est. time: 10.38min) 2024-12-18T00:52:20.0009447Z Serial tests (3): 2024-12-18T00:52:20.0009757Z distributed/test_compute_comm_reordering 1/1 2024-12-18T00:52:20.0010130Z distributed/test_dynamo_distributed 1/1 2024-12-18T00:52:20.0010491Z distributed/test_inductor_collectives 1/1 2024-12-18T00:52:20.0010827Z Parallel tests (0): 2024-12-18T00:52:20.0011101Z Name: excluded (est. time: 0.0min) 2024-12-18T00:52:20.0011406Z Serial tests (0): 2024-12-18T00:52:20.0011653Z Parallel tests (0): 2024-12-18T00:52:20.0072130Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-12-18 00:52:20.006862] 2024-12-18T00:52:20.0072643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:52:20.0075250Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:52:20.007209] 2024-12-18T00:53:15.9166449Z 2024-12-18T00:53:15.9167692Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_f5a162412b126d04_.log 2024-12-18T00:53:15.9172237Z Running 7 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-12-18T00:53:15.9175699Z 2024-12-18T00:53:15.9175960Z Running distributed/test_dynamo_distributed 1/1 ... [2024-12-18 00:53:15.916507] 2024-12-18T00:53:15.9176419Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T00:53:15.9177469Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 00:53:15.916849] 2024-12-18T01:00:34.4947847Z 2024-12-18T01:00:34.4949814Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_18fcfa6b2c004458_.log 2024-12-18T01:00:34.4974765Z Running 58 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-12-18T01:00:34.4996345Z 2024-12-18T01:00:34.4996608Z Running distributed/test_inductor_collectives 1/1 ... [2024-12-18 01:00:34.494685] 2024-12-18T01:00:34.4997098Z SCRIBE_GRAPHQL_ACCESS_TOKEN is set 2024-12-18T01:00:34.4998150Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 01:00:34.495028] 2024-12-18T01:02:43.9857376Z 2024-12-18T01:02:43.9858486Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_83956a12a77037c4_.log 2024-12-18T01:02:43.9881630Z Running 48 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta 2024-12-18T01:02:43.9902516Z 2024-12-18T01:02:44.7570843Z Running test batch 'tests to run' cost 624.76 seconds 2024-12-18T01:02:45.6071568Z + assert_git_not_dirty 2024-12-18T01:02:45.6072585Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *rocm* ]] 2024-12-18T01:02:45.6073480Z + [[ linux-focal-cuda12.4-py3.10-gcc9-sm86 != *xla* ]] 2024-12-18T01:02:45.6079903Z ++ git status --porcelain 2024-12-18T01:02:45.6080445Z ++ grep -v '?? third_party' 2024-12-18T01:02:48.2394640Z ++ true 2024-12-18T01:02:48.2395702Z + git_status= 2024-12-18T01:02:48.2396248Z + [[ -n '' ]] 2024-12-18T01:02:48.2399027Z + cleanup_workspace 2024-12-18T01:02:48.2399757Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-12-18T01:02:48.2400839Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-12-18T01:02:48.2401686Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-12-18T01:02:48.2402280Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T01:02:48.2403016Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-12-18T01:02:48.2403861Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-12-18T01:02:48.2404329Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-12-18T01:02:49.0118391Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@release/2.6 2024-12-18T01:02:49.0118895Z with: 2024-12-18T01:02:49.0119157Z benchmark-results-dir: test/test-reports 2024-12-18T01:02:49.0119489Z dry-run: false 2024-12-18T01:02:49.0119730Z schema-version: v3 2024-12-18T01:02:49.0120173Z github-token: *** 2024-12-18T01:02:49.0120416Z env: 2024-12-18T01:02:49.0120642Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:49.0120986Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:49.0121535Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:49.0122026Z ##[endgroup] 2024-12-18T01:02:49.0147861Z ##[group]Run set -eux 2024-12-18T01:02:49.0148132Z set -eux 2024-12-18T01:02:49.0148407Z python3 -mpip install boto3==1.35.33 2024-12-18T01:02:49.0162250Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:49.0162641Z env: 2024-12-18T01:02:49.0162998Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:49.0163344Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:49.0164147Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:49.0164642Z ##[endgroup] 2024-12-18T01:02:49.0198931Z + python3 -mpip install boto3==1.35.33 2024-12-18T01:02:49.2623637Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T01:02:49.2824076Z Requirement already satisfied: boto3==1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (1.35.33) 2024-12-18T01:02:49.2874857Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4) 2024-12-18T01:02:49.2878760Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.83) 2024-12-18T01:02:49.2882644Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2024-12-18T01:02:49.2939925Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2024-12-18T01:02:49.2945345Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2024-12-18T01:02:49.2982920Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2024-12-18T01:02:49.4219699Z ##[group]Run set -eux 2024-12-18T01:02:49.4219995Z set -eux 2024-12-18T01:02:49.4220237Z  2024-12-18T01:02:49.4220502Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2024-12-18T01:02:49.4220866Z  echo "Missing github-token input" 2024-12-18T01:02:49.4221219Z  exit 1 2024-12-18T01:02:49.4221451Z fi 2024-12-18T01:02:49.4230544Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:49.4230933Z env: 2024-12-18T01:02:49.4231163Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:49.4231509Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:49.4232063Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:49.4232981Z GITHUB_TOKEN: *** 2024-12-18T01:02:49.4233238Z ##[endgroup] 2024-12-18T01:02:49.4262755Z + [[ -z *** ]] 2024-12-18T01:02:49.4312419Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2024-12-18T01:02:49.4312844Z with: 2024-12-18T01:02:49.4313193Z github-token: *** 2024-12-18T01:02:49.4313440Z env: 2024-12-18T01:02:49.4313669Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:49.4314014Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:49.4314779Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:49.4315273Z ##[endgroup] 2024-12-18T01:02:49.4336940Z ##[group]Run set -eux 2024-12-18T01:02:49.4337212Z set -eux 2024-12-18T01:02:49.4337472Z  2024-12-18T01:02:49.4337961Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T01:02:49.4346081Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:49.4346457Z env: 2024-12-18T01:02:49.4346685Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:49.4347023Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:49.4347574Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:49.4348361Z GITHUB_TOKEN: *** 2024-12-18T01:02:49.4348609Z ##[endgroup] 2024-12-18T01:02:49.4377935Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 12383255690 i-079e9e3a7e7495371 2024-12-18T01:02:50.7113057Z setting job-id=34567098055 2024-12-18T01:02:50.7113674Z setting job-name=unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T01:02:50.7222087Z ##[group]Run set -eux 2024-12-18T01:02:50.7222418Z set -eux 2024-12-18T01:02:50.7222652Z  2024-12-18T01:02:50.7223042Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2024-12-18T01:02:50.7223552Z  --schema-version "${SCHEMA_VERSION}" \ 2024-12-18T01:02:50.7223902Z  --repo "${REPO}" \ 2024-12-18T01:02:50.7224215Z  --head-branch "${HEAD_BRANCH}" \ 2024-12-18T01:02:50.7224554Z  --head-sha "${HEAD_SHA}" \ 2024-12-18T01:02:50.7224901Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2024-12-18T01:02:50.7225260Z  --run-attempt "${RUN_ATTEMPT}" \ 2024-12-18T01:02:50.7225594Z  --job-id "${JOB_ID}" \ 2024-12-18T01:02:50.7225912Z  --job-name "${JOB_NAME}" 2024-12-18T01:02:50.7235655Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.7236035Z env: 2024-12-18T01:02:50.7236262Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.7236596Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.7237151Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.7237649Z SCHEMA_VERSION: v3 2024-12-18T01:02:50.7237915Z REPO: pytorch/pytorch 2024-12-18T01:02:50.7238196Z HEAD_BRANCH: refs/heads/release/2.6 2024-12-18T01:02:50.7238547Z HEAD_SHA: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T01:02:50.7238900Z WORKFLOW_RUN_ID: 12383255690 2024-12-18T01:02:50.7239183Z RUN_ATTEMPT: 1 2024-12-18T01:02:50.7239429Z JOB_ID: 34567098055 2024-12-18T01:02:50.7239969Z JOB_NAME: unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu) 2024-12-18T01:02:50.7240543Z ##[endgroup] 2024-12-18T01:02:50.7274204Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/release/2.6 --head-sha 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 --workflow-id 12383255690 --run-attempt 1 --job-id 34567098055 --job-name 'unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu)' 2024-12-18T01:02:50.7618027Z ##[group]Run set -eux 2024-12-18T01:02:50.7618296Z set -eux 2024-12-18T01:02:50.7618523Z  2024-12-18T01:02:50.7618778Z # TODO (huydhn): Implement this part 2024-12-18T01:02:50.7619141Z echo "runners=[]" >> "${GITHUB_OUTPUT}" 2024-12-18T01:02:50.7627943Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.7628321Z env: 2024-12-18T01:02:50.7628545Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.7629088Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.7629811Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.7630313Z ##[endgroup] 2024-12-18T01:02:50.7659318Z + echo 'runners=[]' 2024-12-18T01:02:50.7692896Z ##[group]Run set -eux 2024-12-18T01:02:50.7693168Z set -eux 2024-12-18T01:02:50.7693394Z  2024-12-18T01:02:50.7693650Z # TODO (huydhn): Implement this part 2024-12-18T01:02:50.7694035Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2024-12-18T01:02:50.7701936Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.7702318Z env: 2024-12-18T01:02:50.7702546Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.7702889Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.7703435Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.7703926Z ##[endgroup] 2024-12-18T01:02:50.7735998Z + echo 'dependencies={}' 2024-12-18T01:02:50.7766312Z ##[group]Run set -eux 2024-12-18T01:02:50.7766622Z set -eux 2024-12-18T01:02:50.7766867Z  2024-12-18T01:02:50.7767151Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2024-12-18T01:02:50.7767602Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2024-12-18T01:02:50.7768096Z  # We don't want the job to fail if the directory doesn't exist 2024-12-18T01:02:50.7768496Z  exit 0 2024-12-18T01:02:50.7768734Z fi 2024-12-18T01:02:50.7768957Z  2024-12-18T01:02:50.7769210Z if [[ "${DRY_RUN}" == "true" ]]; then 2024-12-18T01:02:50.7769677Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T01:02:50.7770236Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T01:02:50.7770669Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T01:02:50.7771047Z  --runners "${RUNNER_INFO}" \ 2024-12-18T01:02:50.7771414Z  --dependencies "${DEPENDENCIES}" \ 2024-12-18T01:02:50.7771753Z  --dry-run 2024-12-18T01:02:50.7772007Z else 2024-12-18T01:02:50.7772395Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2024-12-18T01:02:50.7772941Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2024-12-18T01:02:50.7773360Z  --metadata "${BENCHMARK_METADATA}" \ 2024-12-18T01:02:50.7773727Z  --runners "${RUNNER_INFO}" \ 2024-12-18T01:02:50.7774084Z  --dependencies "${DEPENDENCIES}" 2024-12-18T01:02:50.7774405Z fi 2024-12-18T01:02:50.7783148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.7783533Z env: 2024-12-18T01:02:50.7783756Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.7784103Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.7784652Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.7785186Z BENCHMARK_RESULTS_DIR: test/test-reports 2024-12-18T01:02:50.7785512Z DRY_RUN: false 2024-12-18T01:02:50.7786844Z BENCHMARK_METADATA: {"timestamp": 1734483770, "schema_version": "v3", "name": "unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567098055} 2024-12-18T01:02:50.7788217Z RUNNER_INFO: [] 2024-12-18T01:02:50.7788462Z DEPENDENCIES: {} 2024-12-18T01:02:50.7788713Z ##[endgroup] 2024-12-18T01:02:50.7819963Z + [[ ! -d test/test-reports ]] 2024-12-18T01:02:50.7820272Z + [[ false == \t\r\u\e ]] 2024-12-18T01:02:50.7822729Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/release/2.6/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1734483770, "schema_version": "v3", "name": "unit-test / cuda12.4-py3.10-gcc9-sm86 / test (inductor_distributed, 1, 1, linux.g5.12xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/release/2.6", "head_sha": "0cdf8b1d09254cfda66191d1bd01e3041c3c76f7", "workflow_id": 12383255690, "run_attempt": 1, "job_id": 34567098055}' --runners '[]' --dependencies '{}' 2024-12-18T01:02:50.9487719Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T01:02:50.9488115Z cat test/**/*_toprint.log || true 2024-12-18T01:02:50.9496004Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.9496386Z env: 2024-12-18T01:02:50.9496618Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.9496966Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.9497523Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.9498076Z ##[endgroup] 2024-12-18T01:02:50.9583737Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T01:02:50.9618664Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-12-18T01:02:50.9619038Z kill "$MONITOR_SCRIPT_PID" 2024-12-18T01:02:50.9626994Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:50.9627381Z env: 2024-12-18T01:02:50.9627622Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:50.9627985Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:50.9628600Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:50.9629106Z MONITOR_SCRIPT_PID: 191290 2024-12-18T01:02:50.9629392Z ##[endgroup] 2024-12-18T01:02:50.9771017Z Prepare all required actions 2024-12-18T01:02:50.9771421Z Getting action download info 2024-12-18T01:02:51.1666603Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T01:02:51.4892863Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T01:02:51.4893239Z with: 2024-12-18T01:02:51.4893661Z file-suffix: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T01:02:51.4894168Z s3-bucket: gha-artifacts 2024-12-18T01:02:51.4894439Z env: 2024-12-18T01:02:51.4894664Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.4895003Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.4895566Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.4896058Z ##[endgroup] 2024-12-18T01:02:51.4928810Z ##[group]Run # Remove any previous test jsons if they exist 2024-12-18T01:02:51.4929297Z # Remove any previous test jsons if they exist 2024-12-18T01:02:51.4929663Z rm -f test-jsons-*.zip 2024-12-18T01:02:51.4930097Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2024-12-18T01:02:51.4939103Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:51.4939491Z env: 2024-12-18T01:02:51.4939721Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.4940076Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.4940624Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.4941306Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T01:02:51.4941799Z ##[endgroup] 2024-12-18T01:02:51.4998893Z adding: test/test-reports/td_exclusions-3e976aa14b2bfcd9f581.json (deflated 25%) 2024-12-18T01:02:51.4999796Z adding: test/test-reports/td_exclusions-e3554f9a3eec1aa90a68.json (deflated 24%) 2024-12-18T01:02:51.5000653Z adding: test/test-reports/td_exclusions-b84b627a082d5ad21a07.json (deflated 24%) 2024-12-18T01:02:51.5001493Z adding: test/test-reports/td_exclusions-daf1ad5d3aa83f24826f.json (deflated 13%) 2024-12-18T01:02:51.5002330Z adding: test/test-reports/td_exclusions-ec3b23e79863cc17234f.json (deflated 19%) 2024-12-18T01:02:51.5003256Z adding: test/test-reports/td_exclusions-3554feb0cf462a110404.json (deflated 14%) 2024-12-18T01:02:51.5004355Z adding: test/test-reports/td_exclusions-51ff192d275f4d98c516.json (deflated 15%) 2024-12-18T01:02:51.5005177Z adding: test/test-reports/td_exclusions-a56790d0753b7571f04a.json (deflated 14%) 2024-12-18T01:02:51.5005797Z adding: test/test-reports/td_exclusions-2037d72958f0f175fec4.json (deflated 15%) 2024-12-18T01:02:51.5006410Z adding: test/test-reports/td_exclusions-0f8a79c204046906c3b3.json (deflated 15%) 2024-12-18T01:02:51.5007019Z adding: test/test-reports/td_exclusions-0b4571e83d0aee8b7518.json (deflated 15%) 2024-12-18T01:02:51.5007645Z adding: test/test-reports/td_exclusions-0c7501d0aadd492fc806.json (deflated 15%) 2024-12-18T01:02:51.5008319Z adding: test/test-reports/td_exclusions-5c88ba03a2cd8ae6dda2.json (deflated 15%) 2024-12-18T01:02:51.5008941Z adding: test/test-reports/td_exclusions-71436c841dd27aa4f25f.json (deflated 16%) 2024-12-18T01:02:51.5009603Z adding: test/test-reports/td_exclusions-fdbe483a54314e410f1f.json (deflated 13%) 2024-12-18T01:02:51.5010260Z adding: test/test-reports/td_exclusions-682c5c1fa33e31ebf6ee.json (deflated 16%) 2024-12-18T01:02:51.5010877Z adding: test/test-reports/td_exclusions-5f06c737b5f9f808940e.json (deflated 16%) 2024-12-18T01:02:51.5011501Z adding: test/test-reports/td_exclusions-b5631f9decc4c30885ff.json (deflated 15%) 2024-12-18T01:02:51.5012128Z adding: test/test-reports/td_exclusions-ef85ba3ec0a53717bd94.json (deflated 17%) 2024-12-18T01:02:51.5012756Z adding: test/test-reports/td_exclusions-895178be622feceed61e.json (deflated 14%) 2024-12-18T01:02:51.5013579Z adding: test/test-reports/td_exclusions-b4d4d4b5a662b563ddb4.json (deflated 46%) 2024-12-18T01:02:51.5040572Z ##[group]Run # Remove any previous test reports if they exist 2024-12-18T01:02:51.5041034Z # Remove any previous test reports if they exist 2024-12-18T01:02:51.5041416Z rm -f test-reports-*.zip 2024-12-18T01:02:51.5041894Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2024-12-18T01:02:51.5050318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:51.5050693Z env: 2024-12-18T01:02:51.5050920Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.5051256Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.5051810Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.5052488Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T01:02:51.5052976Z ##[endgroup] 2024-12-18T01:02:51.5110818Z adding: test/test-reports/python-pytest/inductor.test_torchinductor/inductor.test_torchinductor-f1031565a7ce3a9e.xml (deflated 79%) 2024-12-18T01:02:51.5111898Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-4b575bb352f31636.xml (deflated 28%) 2024-12-18T01:02:51.5112931Z adding: test/test-reports/python-pytest/inductor.test_aot_inductor/inductor.test_aot_inductor-1ce03015e4778ad6.xml (deflated 47%) 2024-12-18T01:02:51.5114077Z adding: test/test-reports/python-pytest/distributed.test_c10d_functional_native/distributed.test_c10d_functional_native-6f4210b69b2e623c.xml (deflated 80%) 2024-12-18T01:02:51.5115336Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-e740978eb2c165e0.xml (deflated 85%) 2024-12-18T01:02:51.5116690Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_micro_pipeline_tp/distributed.tensor.parallel.test_micro_pipeline_tp-8a51e78c1e7eaef9.xml (deflated 87%) 2024-12-18T01:02:51.5118168Z adding: test/test-reports/python-pytest/distributed._composable.test_replicate_with_compiler/distributed._composable.test_replicate_with_compiler-232fe1a641461b9a.xml (deflated 75%) 2024-12-18T01:02:51.5119601Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_comm/distributed._composable.fsdp.test_fully_shard_comm-670239ad9f891a3b.xml (deflated 81%) 2024-12-18T01:02:51.5121224Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-c87bd39928f2d984.xml (deflated 62%) 2024-12-18T01:02:51.5122702Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-0fe45eec70ceb177.xml (deflated 35%) 2024-12-18T01:02:51.5124270Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-265bdf2c0ee9465d.xml (deflated 35%) 2024-12-18T01:02:51.5125762Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-94270efa4ebb4b32.xml (deflated 28%) 2024-12-18T01:02:51.5127240Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_training/distributed._composable.fsdp.test_fully_shard_training-c94bbecbd75e231e.xml (deflated 36%) 2024-12-18T01:02:51.5128717Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state_dict/distributed._composable.fsdp.test_fully_shard_state_dict-d337670625965a71.xml (deflated 36%) 2024-12-18T01:02:51.5130174Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_frozen/distributed._composable.fsdp.test_fully_shard_frozen-282aef864882aa53.xml (deflated 59%) 2024-12-18T01:02:51.5131835Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-98e30d19b6cdc964.xml (deflated 38%) 2024-12-18T01:02:51.5133419Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_mixed_precision/distributed._composable.fsdp.test_fully_shard_mixed_precision-0337de3c003929e5.xml (deflated 38%) 2024-12-18T01:02:51.5135056Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_/distributed._composable.fsdp.test_fully_shard_clip_grad_norm_-58e347a588866716.xml (deflated 38%) 2024-12-18T01:02:51.5136554Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_compile/distributed._composable.fsdp.test_fully_shard_compile-7df3080443bd06a0.xml (deflated 82%) 2024-12-18T01:02:51.5137925Z adding: test/test-reports/python-pytest/distributed.fsdp.test_fsdp_tp_integration/distributed.fsdp.test_fsdp_tp_integration-0221d044a92822a3.xml (deflated 58%) 2024-12-18T01:02:51.5139199Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-0e3ddba58c215f13.xml (deflated 73%) 2024-12-18T01:02:51.5140448Z adding: test/test-reports/python-pytest/distributed.test_dynamo_distributed/distributed.test_dynamo_distributed-22e0ea2840fd382f.xml (deflated 92%) 2024-12-18T01:02:51.5141670Z adding: test/test-reports/python-pytest/distributed.test_inductor_collectives/distributed.test_inductor_collectives-0d3288c624ffb130.xml (deflated 88%) 2024-12-18T01:02:51.5170750Z ##[group]Run # Remove any previous usage logs if they exist 2024-12-18T01:02:51.5171206Z # Remove any previous usage logs if they exist 2024-12-18T01:02:51.5171577Z rm -f logs-*.zip 2024-12-18T01:02:51.5172044Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-12-18T01:02:51.5172569Z # so check to see if the file exists first 2024-12-18T01:02:51.5172928Z if [ -f 'usage_log.txt' ]; then 2024-12-18T01:02:51.5173309Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-12-18T01:02:51.5173650Z fi 2024-12-18T01:02:51.5174019Z if find "test/test-reports" -name "*.log" 2>/dev/null | grep -q .; then 2024-12-18T01:02:51.5174565Z  zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' 2024-12-18T01:02:51.5175083Z fi 2024-12-18T01:02:51.5182911Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:51.5183304Z env: 2024-12-18T01:02:51.5183529Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.5183878Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.5184440Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.5185119Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T01:02:51.5185609Z ##[endgroup] 2024-12-18T01:02:51.5513053Z adding: usage_log.txt (deflated 97%) 2024-12-18T01:02:51.5565712Z adding: test/test-reports/inductor.test_torchinductor_1.1_c6d3724247314335_.log (deflated 62%) 2024-12-18T01:02:51.5567161Z adding: test/test-reports/inductor.test_aot_inductor_1.1_345ce8d03071aaf8_.log (deflated 51%) 2024-12-18T01:02:51.5568209Z adding: test/test-reports/inductor.test_aot_inductor_1.1_e7b1f1431993dccd_.log (deflated 60%) 2024-12-18T01:02:51.5568978Z adding: test/test-reports/distributed.test_c10d_functional_native_1.1_afe1081bc25be411_.log (deflated 79%) 2024-12-18T01:02:51.5570010Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_08bcad7095619d2b_.log (deflated 84%) 2024-12-18T01:02:51.5571087Z adding: test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_b2d04a4a17757235_.log (deflated 88%) 2024-12-18T01:02:51.5572747Z adding: test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_efe28b3877a95fc7_.log (deflated 88%) 2024-12-18T01:02:51.5595278Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_052af212cf500711_.log (deflated 95%) 2024-12-18T01:02:51.5596217Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_a3139a56c0a22b59_.log (deflated 74%) 2024-12-18T01:02:51.5597158Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_095a1e85c2a8db65_.log (deflated 61%) 2024-12-18T01:02:51.5598132Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_42f80aa0778ee104_.log (deflated 57%) 2024-12-18T01:02:51.5599082Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_9f93ea92865eb61a_.log (deflated 54%) 2024-12-18T01:02:51.5600012Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_19d737b752a6fbd5_.log (deflated 58%) 2024-12-18T01:02:51.5600955Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_5ac29da3b0a6b299_.log (deflated 58%) 2024-12-18T01:02:51.5601896Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_adb022b549e3503a_.log (deflated 83%) 2024-12-18T01:02:51.5602911Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_00a606aa9a3fb8c6_.log (deflated 58%) 2024-12-18T01:02:51.5603898Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_b9bbfa7d8ded0659_.log (deflated 58%) 2024-12-18T01:02:51.5604880Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_4934dbbc4babbb28_.log (deflated 57%) 2024-12-18T01:02:51.5606173Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_736d1569e7d44a0f_.log (deflated 94%) 2024-12-18T01:02:51.5607069Z adding: test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_680adda8ff934fc7_.log (deflated 66%) 2024-12-18T01:02:51.5608077Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_f5a162412b126d04_.log (deflated 87%) 2024-12-18T01:02:51.5612760Z adding: test/test-reports/distributed.test_dynamo_distributed_1.1_18fcfa6b2c004458_.log (deflated 92%) 2024-12-18T01:02:51.5615922Z adding: test/test-reports/distributed.test_inductor_collectives_1.1_83956a12a77037c4_.log (deflated 89%) 2024-12-18T01:02:51.5651831Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-12-18T01:02:51.5652546Z # Remove any previous debugging artifacts if they exist 2024-12-18T01:02:51.5652949Z rm -f debug-*.zip 2024-12-18T01:02:51.5653232Z if [ -d 'test/debug' ]; then 2024-12-18T01:02:51.5653602Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-12-18T01:02:51.5653949Z fi 2024-12-18T01:02:51.5662910Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:51.5663292Z env: 2024-12-18T01:02:51.5663876Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.5664216Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.5664799Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.5665486Z FILE_SUFFIX: test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055 2024-12-18T01:02:51.5665975Z ##[endgroup] 2024-12-18T01:02:51.5753563Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:02:51.5753893Z with: 2024-12-18T01:02:51.5754129Z s3-bucket: gha-artifacts 2024-12-18T01:02:51.5754454Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:51.5754808Z retention-days: 14 2024-12-18T01:02:51.5755075Z if-no-files-found: warn 2024-12-18T01:02:51.5755357Z path: test-jsons-*.zip 2024-12-18T01:02:51.5755611Z name: artifact 2024-12-18T01:02:51.5755855Z region: us-east-1 2024-12-18T01:02:51.5756093Z env: 2024-12-18T01:02:51.5756324Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:51.5756668Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:51.5757217Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:51.5757709Z ##[endgroup] 2024-12-18T01:02:51.9374097Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:02:51.9374561Z With the provided path, there will be 1 file uploaded 2024-12-18T01:02:51.9375020Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:51.9432959Z Starting upload of test-jsons-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:52.1399340Z Finished upload of test-jsons-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:52.1690802Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:02:52.1691144Z with: 2024-12-18T01:02:52.1691376Z s3-bucket: gha-artifacts 2024-12-18T01:02:52.1691707Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:52.1692063Z retention-days: 14 2024-12-18T01:02:52.1692334Z if-no-files-found: error 2024-12-18T01:02:52.1692629Z path: test-reports-*.zip 2024-12-18T01:02:52.1692900Z name: artifact 2024-12-18T01:02:52.1693145Z region: us-east-1 2024-12-18T01:02:52.1693401Z env: 2024-12-18T01:02:52.1693622Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:52.1693968Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:52.1694529Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:52.1695029Z ##[endgroup] 2024-12-18T01:02:52.4990539Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:02:52.4991413Z With the provided path, there will be 1 file uploaded 2024-12-18T01:02:52.4992328Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:52.5048621Z Starting upload of test-reports-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:52.6264722Z Finished upload of test-reports-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:52.6555787Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:02:52.6556131Z with: 2024-12-18T01:02:52.6556360Z s3-bucket: gha-artifacts 2024-12-18T01:02:52.6556709Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:52.6557071Z retention-days: 14 2024-12-18T01:02:52.6557338Z if-no-files-found: ignore 2024-12-18T01:02:52.6557620Z path: logs-*.zip 2024-12-18T01:02:52.6557862Z name: artifact 2024-12-18T01:02:52.6558300Z region: us-east-1 2024-12-18T01:02:52.6558545Z env: 2024-12-18T01:02:52.6558781Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:52.6559138Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:52.6559691Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:52.6560183Z ##[endgroup] 2024-12-18T01:02:52.9854995Z NOTE: s3-prefix specified, ignoring name parameter 2024-12-18T01:02:52.9855443Z With the provided path, there will be 1 file uploaded 2024-12-18T01:02:52.9855904Z Uploading to s3 prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:52.9914860Z Starting upload of logs-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:53.2678668Z Finished upload of logs-test-inductor_distributed-1-1-linux.g5.12xlarge.nvidia.gpu_34567098055.zip 2024-12-18T01:02:53.2969205Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-12-18T01:02:53.2969547Z with: 2024-12-18T01:02:53.2969785Z s3-bucket: gha-artifacts 2024-12-18T01:02:53.2970133Z s3-prefix: pytorch/pytorch/12383255690/1/artifact 2024-12-18T01:02:53.2970483Z retention-days: 14 2024-12-18T01:02:53.2970751Z if-no-files-found: ignore 2024-12-18T01:02:53.2971035Z path: debug-*.zip 2024-12-18T01:02:53.2971284Z name: artifact 2024-12-18T01:02:53.2971529Z region: us-east-1 2024-12-18T01:02:53.2971771Z env: 2024-12-18T01:02:53.2971988Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:53.2972336Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:53.2972896Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:53.2973386Z ##[endgroup] 2024-12-18T01:02:53.6205876Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-12-18T01:02:53.6500536Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T01:02:53.6500923Z # shellcheck disable=SC2156 2024-12-18T01:02:53.6501497Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T01:02:53.6510570Z shell: /usr/bin/bash -e {0} 2024-12-18T01:02:53.6510861Z env: 2024-12-18T01:02:53.6511095Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:53.6511448Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:53.6512005Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:53.6512508Z ##[endgroup] 2024-12-18T01:02:53.9167306Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@release/2.6 2024-12-18T01:02:53.9167760Z with: 2024-12-18T01:02:53.9167974Z env: 2024-12-18T01:02:53.9168190Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:53.9168540Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:53.9169101Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:53.9169596Z ##[endgroup] 2024-12-18T01:02:53.9196438Z ##[group]Run set -eou pipefail 2024-12-18T01:02:53.9196775Z set -eou pipefail 2024-12-18T01:02:53.9197050Z  2024-12-18T01:02:53.9197410Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-12-18T01:02:53.9197915Z for _ in $(seq 1440); do 2024-12-18T01:02:53.9198251Z  # Break if no ssh session exists anymore 2024-12-18T01:02:53.9198604Z  if [ "$(who)" = "" ]; then 2024-12-18T01:02:53.9198910Z  break 2024-12-18T01:02:53.9199178Z  fi 2024-12-18T01:02:53.9199437Z  echo "." 2024-12-18T01:02:53.9199708Z  sleep 5 2024-12-18T01:02:53.9199958Z done 2024-12-18T01:02:53.9208752Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:53.9209159Z env: 2024-12-18T01:02:53.9209427Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:53.9209775Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:53.9210331Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:53.9211038Z ##[endgroup] 2024-12-18T01:02:53.9237367Z Holding runner for 2 hours until all ssh sessions have logged out 2024-12-18T01:02:53.9303367Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T01:02:53.9303931Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T01:02:53.9304349Z # shellcheck disable=SC2046 2024-12-18T01:02:53.9304712Z docker stop $(docker ps -q) || true 2024-12-18T01:02:53.9305068Z # Prune all of the docker images 2024-12-18T01:02:53.9305406Z docker system prune -af 2024-12-18T01:02:53.9314062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:02:53.9314438Z env: 2024-12-18T01:02:53.9314672Z GIT_DEFAULT_BRANCH: main 2024-12-18T01:02:53.9315024Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-12-18T01:02:53.9315584Z DOCKER_CONTAINER_ID: 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:53.9316089Z ##[endgroup] 2024-12-18T01:02:54.5300684Z 06c5d3cbe8cd 2024-12-18T01:02:55.6144128Z Deleted Containers: 2024-12-18T01:02:55.6144667Z 06c5d3cbe8cd0a5f784aaf0aab1ed8236d7e86dd33b567416679707d620dc96d 2024-12-18T01:02:55.6145132Z 2024-12-18T01:03:07.8540632Z Deleted Images: 2024-12-18T01:03:07.8541619Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T01:03:07.8543137Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.4-cudnn9-py3-gcc9-inductor-benchmarks@sha256:ff80d4c93d18e8ab7a2501f0e62b7c40cf97aa83fc9702e88b9c2d02cdcbecdb 2024-12-18T01:03:07.8544278Z deleted: sha256:67e93a8badaf799ceb664d7c453077c6c6785d56dc10c720be9358edeb91759b 2024-12-18T01:03:07.8544912Z deleted: sha256:7958cbbed07d0e7e404de5150d4669da383984a17de74e455563b7ecc409b2e8 2024-12-18T01:03:07.8545551Z deleted: sha256:dc6a8b10e3a8452ea47655ddad3b52816f12bcbe35298b3aa96aae8a0da9bc12 2024-12-18T01:03:07.8546216Z deleted: sha256:c92bf6e7cfbdecd0dfe86d452eac5dbe5aa5b2a6aa7ef2eaeffb21af2870cda7 2024-12-18T01:03:07.8546859Z deleted: sha256:f9597cb89d67c0c899e266adc5e5f04817474bad200712b3c256ed33c346b3e8 2024-12-18T01:03:07.8547481Z deleted: sha256:eb062308fcca86ab5a341555bf03af093d21f1d44dbe3dce6f9b7781b650b2f1 2024-12-18T01:03:07.8548438Z deleted: sha256:96f28e0a6369023dd4e75125d2454531ff686a14467f3e06d3b5c6da89893472 2024-12-18T01:03:07.8549054Z deleted: sha256:3e6e64b255956d1bf66c97393ae4325f7fc67414cd710e284c17190cb262c92d 2024-12-18T01:03:07.8549663Z deleted: sha256:a3c0a58b4f23e46b45dd1b385363731800042b3f04ca66f8867b99f3c9311f11 2024-12-18T01:03:07.8550277Z deleted: sha256:435e519973ffb9c85330ae8db1d1fb1fa75d673e2f27640c2ca7277c6352092b 2024-12-18T01:03:07.8550888Z deleted: sha256:69800681a8b6382d520ac4b8b8828bd5540a297961ddeb004e415bc46b33f80d 2024-12-18T01:03:07.8551492Z deleted: sha256:1e837ad662894f46afc56a73aa5ccb04064e4a974c9e3cad86ce9e5546b1d63d 2024-12-18T01:03:07.8552114Z deleted: sha256:688eccf2945022bd75f9f2782e2397ea54f1032c2435aa8f7dffa23cbcf2ae99 2024-12-18T01:03:07.8552731Z deleted: sha256:4855651bf4c3c8acad23029fda7d17f43bf32a69e8d575bb43f4491f0d653c1f 2024-12-18T01:03:07.8553352Z deleted: sha256:ca2d39907b7e10e3c7ae753b8bd230fac77c4e4b15980befd71173da8a36da93 2024-12-18T01:03:07.8553976Z deleted: sha256:d4f04a8342f272455eb816a413dc2dfcc75b9ebb9ef7f6fab32436621412ba6b 2024-12-18T01:03:07.8554614Z deleted: sha256:63ae01c22dc840b763ddc08bf4e145c14a04e0ff8a833b357ed45cb0962a8116 2024-12-18T01:03:07.8555230Z deleted: sha256:cf91e4695bf05d3607b7fa68e3ba3d6cc8762e6fac8773483f8116812f3d52c0 2024-12-18T01:03:07.8555840Z deleted: sha256:71baa7b72633376bb45367512dcb676c81b1e09964085a3151381530df7cdc28 2024-12-18T01:03:07.8556434Z deleted: sha256:68855433b69809d035da2a30cb22864ed1cd2795434571a0c77532de19344497 2024-12-18T01:03:07.8557022Z deleted: sha256:929280209b1f4e29860f386a519df32161cb608338d5338f24735201939dc91f 2024-12-18T01:03:07.8557625Z deleted: sha256:25133b63282efce5897b0e8fbf64d8b2a2625e15792dad95bdada277b78c777d 2024-12-18T01:03:07.8558433Z deleted: sha256:11fddfe713ea9ddd5acd8285319a606609b5c428111f15a019c8286e7164b9b4 2024-12-18T01:03:07.8559039Z deleted: sha256:b1464f3888e3b0a1694aa960a726584d01ae6b62df53fe8743d197d2d5051036 2024-12-18T01:03:07.8559637Z deleted: sha256:9627c2925d7a3936b439a6f519083e650d160bc2522d2a9244d6a1dfda02bc83 2024-12-18T01:03:07.8560245Z deleted: sha256:33880827ded8236cc91531c2f928d6de5c7cfb95a241da2653b7e8a8d1ea4881 2024-12-18T01:03:07.8560854Z deleted: sha256:6851b39610a99eb7ec6939a15e35657e144eb3e628a3856afa0a0adff49b8c2f 2024-12-18T01:03:07.8561464Z deleted: sha256:1a76c7f0af1e05689638a3792e8b9d4418648384aed1f56d3ccde5b0a9681d29 2024-12-18T01:03:07.8562080Z deleted: sha256:072f501c97aae0be55d1457426bc5f8238b0bb86eac9e92889b01924352d79a5 2024-12-18T01:03:07.8562682Z deleted: sha256:8277868848759cc86b200280dee8701d7e8328a811bf75330ed24aeaf490e69e 2024-12-18T01:03:07.8563392Z deleted: sha256:1da994d575d08f659512ea580648bfddd62020acc1aef2f7d4c6008d36a6f83f 2024-12-18T01:03:07.8564254Z deleted: sha256:ddf6d49d71559688ae65aeecccbbc58dd555887d4b3306d02a87d049114dd2f9 2024-12-18T01:03:07.8564875Z deleted: sha256:648d99cd03665c7b88aae7e3291f27daa65f496f7ab2c37f0ee62c72a4459151 2024-12-18T01:03:07.8565489Z deleted: sha256:6e57cd88598cddad4070d39467d8e1d4c387388f89d2970a55391653f9243e1a 2024-12-18T01:03:07.8566112Z deleted: sha256:9b0c5db353ac5d4e0995c2a70ae3c28390676afe60dbd11e4ade07c792eac4ce 2024-12-18T01:03:07.8566727Z deleted: sha256:8fa5311fe41b6caf697456e9678220a152bbf6a7645f4b7ff89ea9317ae04444 2024-12-18T01:03:07.8567335Z deleted: sha256:dc438d6465050b81bff4a8c5925e70c2167712dbab174c38f79b6a5d7af407af 2024-12-18T01:03:07.8567938Z deleted: sha256:29b8f0a42a1f3b944f2c54348348d173d322522b578e3d485428d67996f6adde 2024-12-18T01:03:07.8568535Z deleted: sha256:26231024334c7a704f9acba86430547a013e959014bfd7a31e26c6ed78521479 2024-12-18T01:03:07.8569150Z deleted: sha256:4166e9dd7564ae2f46ea2c3dcb8a8ef6bef9e06f4faf1742f4c6f21861967193 2024-12-18T01:03:07.8569778Z deleted: sha256:80a3113f256f146f0a2528ca69aa9244a1f9ba1ecdf54c3ef8ba0fb21507cbc4 2024-12-18T01:03:07.8570391Z deleted: sha256:e90bbba5d7c08e01c5bedb0a855cc5cdd371688e23994155674704963591296f 2024-12-18T01:03:07.8571011Z deleted: sha256:d3b3df431f230849965cc370c2f1853a81d78dd6eb3f7ac3bc3b4ca64aa9818a 2024-12-18T01:03:07.8571630Z deleted: sha256:7a33f9ca143b08c9f5d8c53974b2d6fe840739b5d2cd30694ba1c86db9393908 2024-12-18T01:03:07.8572433Z deleted: sha256:bad1d7dae968956b4a81369dedfbcf539fd5b61dd3051e2eccb5457174fe788c 2024-12-18T01:03:07.8573064Z deleted: sha256:08a420245ca6eed4aa5662d0fec04e7892165cdb3fd2f5b99428adbb4d6199c0 2024-12-18T01:03:07.8573682Z deleted: sha256:c3fda6b9a4d08a345824508dc5625f52e4d23163843d0caf337c60feda68a830 2024-12-18T01:03:07.8574301Z deleted: sha256:c74a8a3ca5e31670eed41e4bbb4a89f25c7c05dacabdf93cbd03984ead0330e2 2024-12-18T01:03:07.8574925Z deleted: sha256:4171d4c54045bd3035546906a7dd3b63c9ca748e0a34eeba27fe918f52cdcf85 2024-12-18T01:03:07.8575539Z deleted: sha256:f6a1d112657a4aeaca96ac2792a10ea09463dc3d633cb8572314814d5abf1ddb 2024-12-18T01:03:07.8576170Z deleted: sha256:af5d54d84b9c302254498eed8b0bbce1cdd705351bf5cc210d3f11c0deb269b3 2024-12-18T01:03:07.8576793Z deleted: sha256:45ae227353c6db8230b86fc1c4b8c50def31bd64895b879d91ea6e600669045d 2024-12-18T01:03:07.8577401Z deleted: sha256:6032ea27ad90071e3b75290e03fd0763a74155f513f356d21e56663a878b0ed9 2024-12-18T01:03:07.8578009Z deleted: sha256:d5aef8ace96787c48070530c09b68e4bcb23bc92a9fb0978220204f64bfbd97f 2024-12-18T01:03:07.8578626Z deleted: sha256:e3116153021ece2a3d29eccdce1f7e0f092316e889c1d534257789f6b5743d51 2024-12-18T01:03:07.8579244Z deleted: sha256:fb0f11460e5fa2e4895230ae84c97fe0552e9509dfbae60abf94e1b8a6563fb9 2024-12-18T01:03:07.8579868Z deleted: sha256:e262415e0abb361b284df9e05790ae27cc3a1ea7b8d4618e7eb3eca17cf1d18e 2024-12-18T01:03:07.8580487Z deleted: sha256:7f810eab57a6121f1a2aae664f7a5940d12f789661445ce3500ecd3b3688b46b 2024-12-18T01:03:07.8581096Z deleted: sha256:6ed71f55ff78520154c43468df2f05a92eaa28ee2aa2e28b8ea860b9ffb258ef 2024-12-18T01:03:07.8581856Z deleted: sha256:92a10d8d42a56a8516d0e9e14fec53263302fe6e6438ced263a3845680456dd7 2024-12-18T01:03:07.8582476Z deleted: sha256:ec67f912ff53e9b7164f48250da3bb4c1ffc94c3eb2c1c56c534a94fe55e3845 2024-12-18T01:03:07.8583100Z deleted: sha256:c85033e4f60ace56eaad80a8151b1bdd1619865b10a9ccee5638878b0a027c23 2024-12-18T01:03:07.8583721Z deleted: sha256:0e8680489f0a37851beb2cb5e42f0df129247a68b755b08a6367ea42180b3e7a 2024-12-18T01:03:07.8584340Z deleted: sha256:3a767fb38f6b1124e5fe9a230fdd383ccb188c38fe1d6ec252260676efe255c8 2024-12-18T01:03:07.8584956Z deleted: sha256:2cce59d2a43773cb7c6b83d239dab3a53659767f96bc0d47a70ee0d51cfa096f 2024-12-18T01:03:07.8585575Z deleted: sha256:51e50e75789ab5b4e40a66bb2f284e2da947e37c0e757c74d49fdeea5da9cbec 2024-12-18T01:03:07.8586189Z deleted: sha256:29e55e4feccba0660950ed92941d309527fe5f41bea609662d0e703b9c76b3a1 2024-12-18T01:03:07.8586814Z deleted: sha256:d15bee8fc455fcb45d16b0abc67f18c99ce1cf2c9e998ef5a100bf8373f2f8ab 2024-12-18T01:03:07.8587437Z deleted: sha256:363936db6a8869d44b6f0bee97a786b168093848e9c08c8042830fc7fccd956b 2024-12-18T01:03:07.8588425Z deleted: sha256:0f54d1f89ceaf71ded488007635a618fac1528bf3603d9415a593fe4926f8b2a 2024-12-18T01:03:07.8589046Z deleted: sha256:2f5796e9879cab7dac02bdbb178aec5301ac00029b38be8613333ac639c4dc3a 2024-12-18T01:03:07.8589668Z deleted: sha256:e68525d57f9da8f6c74b456d92869c4e60e451b88cb403addbc4ea917d8ab3ac 2024-12-18T01:03:07.8590287Z deleted: sha256:3386e25983944ce3b2ebdde10d840832d581a7dfafd1df66568822a18979e268 2024-12-18T01:03:07.8590904Z deleted: sha256:d3a036e6bd36f014b3a1540d5b848655ecdf72a20dc81c0da5f80430700cfa9a 2024-12-18T01:03:07.8591522Z deleted: sha256:96d0556708ff4ec0d5d727d5947d6cb29a9d6c16e1a1ee76a92dc85aa3ea8f25 2024-12-18T01:03:07.8592199Z deleted: sha256:f3c2fb3abccb3a33dbf204d3f0d8857537348e6c1d3cc87a431afe866435d41b 2024-12-18T01:03:07.8592818Z deleted: sha256:45cd6c9a41f12991ee60a89e775abb604b0892ef3936526c18184bbbf4de57da 2024-12-18T01:03:07.8593443Z deleted: sha256:b3cacd0508bbfd2fd668f30c539033e7c6b606bf669258c8e8a3cbf80bf050c6 2024-12-18T01:03:07.8594063Z deleted: sha256:de72ce930361a96c151a8a4725f8360f33df6c812b6ed6d3dd5b6e6710a5d2c0 2024-12-18T01:03:07.8594684Z deleted: sha256:2e5d223d2bfa0f47e628b631a540d20c284dabb380ad9ee4caec65bf8d0ecde2 2024-12-18T01:03:07.8595302Z deleted: sha256:14a85e604a0401093323d356b3bb7f75a8473abd712b4bf1a2ef01fa432b344c 2024-12-18T01:03:07.8596013Z deleted: sha256:15a95a1e8f47d7c18dcbf9a6e2fc530cddcb161c0093c663181f3e73dc9f599e 2024-12-18T01:03:07.8596631Z deleted: sha256:106e8431b412f51ccd75ea46a2d5cb4343b23273cbcf50188377cb93aa9a6d82 2024-12-18T01:03:07.8596995Z 2024-12-18T01:03:07.8597118Z Total reclaimed space: 43GB 2024-12-18T01:03:07.8660263Z Post job cleanup. 2024-12-18T01:03:07.8705648Z Post job cleanup. 2024-12-18T01:03:07.9480651Z [command]/usr/bin/git version 2024-12-18T01:03:07.9517993Z git version 2.40.1 2024-12-18T01:03:07.9558779Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/6417cfc6-2f29-4897-b497-02c28bd5d4d0' before making global git config changes 2024-12-18T01:03:07.9559713Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T01:03:07.9563177Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-12-18T01:03:07.9596549Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T01:03:07.9628047Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T01:03:08.0017128Z Entering 'android/libs/fbjni' 2024-12-18T01:03:08.0088044Z Entering 'third_party/FP16' 2024-12-18T01:03:08.0157807Z Entering 'third_party/FXdiv' 2024-12-18T01:03:08.0229269Z Entering 'third_party/NNPACK' 2024-12-18T01:03:08.0298028Z Entering 'third_party/NVTX' 2024-12-18T01:03:08.0368306Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T01:03:08.0438222Z Entering 'third_party/XNNPACK' 2024-12-18T01:03:08.0525790Z Entering 'third_party/benchmark' 2024-12-18T01:03:08.0596271Z Entering 'third_party/composable_kernel' 2024-12-18T01:03:08.0676977Z Entering 'third_party/cpp-httplib' 2024-12-18T01:03:08.0747108Z Entering 'third_party/cpuinfo' 2024-12-18T01:03:08.0820343Z Entering 'third_party/cudnn_frontend' 2024-12-18T01:03:08.0890424Z Entering 'third_party/cutlass' 2024-12-18T01:03:08.0973990Z Entering 'third_party/eigen' 2024-12-18T01:03:08.1049359Z Entering 'third_party/fbgemm' 2024-12-18T01:03:08.1118063Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T01:03:08.1190292Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T01:03:08.1257358Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T01:03:08.1334971Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T01:03:08.1407752Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T01:03:08.1477187Z Entering 'third_party/flatbuffers' 2024-12-18T01:03:08.1553133Z Entering 'third_party/fmt' 2024-12-18T01:03:08.1623865Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T01:03:08.1696701Z Entering 'third_party/gloo' 2024-12-18T01:03:08.1775878Z Entering 'third_party/googletest' 2024-12-18T01:03:08.1848129Z Entering 'third_party/ideep' 2024-12-18T01:03:08.1915804Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T01:03:08.1995050Z Entering 'third_party/ittapi' 2024-12-18T01:03:08.2071035Z Entering 'third_party/kineto' 2024-12-18T01:03:08.2144222Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T01:03:08.2212697Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T01:03:08.2287689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T01:03:08.2356802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T01:03:08.2427813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T01:03:08.2494691Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T01:03:08.2574619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T01:03:08.2643893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T01:03:08.2714463Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T01:03:08.2789456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T01:03:08.2864948Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T01:03:08.2938500Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T01:03:08.3011447Z Entering 'third_party/mimalloc' 2024-12-18T01:03:08.3087233Z Entering 'third_party/nccl/nccl' 2024-12-18T01:03:08.3158377Z Entering 'third_party/nlohmann' 2024-12-18T01:03:08.3231798Z Entering 'third_party/onnx' 2024-12-18T01:03:08.3319504Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T01:03:08.3395489Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T01:03:08.3470911Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T01:03:08.3537434Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T01:03:08.3608262Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T01:03:08.3677696Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T01:03:08.3750405Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T01:03:08.3819332Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T01:03:08.3887162Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T01:03:08.3953069Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T01:03:08.4030316Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T01:03:08.4113633Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T01:03:08.4206862Z Entering 'third_party/pocketfft' 2024-12-18T01:03:08.4288305Z Entering 'third_party/protobuf' 2024-12-18T01:03:08.4360097Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T01:03:08.4428048Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T01:03:08.4500358Z Entering 'third_party/psimd' 2024-12-18T01:03:08.4575656Z Entering 'third_party/pthreadpool' 2024-12-18T01:03:08.4646015Z Entering 'third_party/pybind11' 2024-12-18T01:03:08.4717471Z Entering 'third_party/python-peachpy' 2024-12-18T01:03:08.4794687Z Entering 'third_party/sleef' 2024-12-18T01:03:08.4870591Z Entering 'third_party/tensorpipe' 2024-12-18T01:03:08.4939743Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T01:03:08.5017732Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T01:03:08.5088373Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T01:03:08.5164763Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T01:03:08.5240938Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T01:03:08.5338775Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T01:03:08.5361583Z http.https://github.com/.extraheader 2024-12-18T01:03:08.5372335Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T01:03:08.5406382Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T01:03:08.5785907Z Entering 'android/libs/fbjni' 2024-12-18T01:03:08.5832139Z http.https://github.com/.extraheader 2024-12-18T01:03:08.5878930Z Entering 'third_party/FP16' 2024-12-18T01:03:08.5924715Z http.https://github.com/.extraheader 2024-12-18T01:03:08.5969652Z Entering 'third_party/FXdiv' 2024-12-18T01:03:08.6020028Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6071745Z Entering 'third_party/NNPACK' 2024-12-18T01:03:08.6116079Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6166228Z Entering 'third_party/NVTX' 2024-12-18T01:03:08.6212477Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6259675Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T01:03:08.6304376Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6351412Z Entering 'third_party/XNNPACK' 2024-12-18T01:03:08.6401610Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6467726Z Entering 'third_party/benchmark' 2024-12-18T01:03:08.6513066Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6558662Z Entering 'third_party/composable_kernel' 2024-12-18T01:03:08.6604277Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6656033Z Entering 'third_party/cpp-httplib' 2024-12-18T01:03:08.6704801Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6750311Z Entering 'third_party/cpuinfo' 2024-12-18T01:03:08.6796067Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6846300Z Entering 'third_party/cudnn_frontend' 2024-12-18T01:03:08.6893234Z http.https://github.com/.extraheader 2024-12-18T01:03:08.6941828Z Entering 'third_party/cutlass' 2024-12-18T01:03:08.6987271Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7041237Z Entering 'third_party/eigen' 2024-12-18T01:03:08.7085442Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7137774Z Entering 'third_party/fbgemm' 2024-12-18T01:03:08.7183717Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7228258Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T01:03:08.7272763Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7318627Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T01:03:08.7360580Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7410259Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T01:03:08.7454038Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7505281Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T01:03:08.7550924Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7596380Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T01:03:08.7643549Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7692025Z Entering 'third_party/flatbuffers' 2024-12-18T01:03:08.7741851Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7791704Z Entering 'third_party/fmt' 2024-12-18T01:03:08.7836489Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7886257Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T01:03:08.7931058Z http.https://github.com/.extraheader 2024-12-18T01:03:08.7986395Z Entering 'third_party/gloo' 2024-12-18T01:03:08.8031784Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8085502Z Entering 'third_party/googletest' 2024-12-18T01:03:08.8130692Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8176285Z Entering 'third_party/ideep' 2024-12-18T01:03:08.8221089Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8265518Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T01:03:08.8308808Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8367847Z Entering 'third_party/ittapi' 2024-12-18T01:03:08.8419437Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8468810Z Entering 'third_party/kineto' 2024-12-18T01:03:08.8514845Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8558592Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T01:03:08.8605062Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8650720Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T01:03:08.8695013Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8741767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T01:03:08.8786129Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8831086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T01:03:08.8875909Z http.https://github.com/.extraheader 2024-12-18T01:03:08.8922375Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T01:03:08.8966510Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9010878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T01:03:08.9055045Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9105732Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T01:03:08.9152556Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9207205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T01:03:08.9250481Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9298046Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T01:03:08.9342291Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9389089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T01:03:08.9434525Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9484014Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T01:03:08.9528378Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9577646Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T01:03:08.9622451Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9674926Z Entering 'third_party/mimalloc' 2024-12-18T01:03:08.9720013Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9767663Z Entering 'third_party/nccl/nccl' 2024-12-18T01:03:08.9812533Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9864177Z Entering 'third_party/nlohmann' 2024-12-18T01:03:08.9910587Z http.https://github.com/.extraheader 2024-12-18T01:03:08.9961733Z Entering 'third_party/onnx' 2024-12-18T01:03:09.0008045Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0067889Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T01:03:09.0113410Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0167087Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T01:03:09.0212350Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0260133Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T01:03:09.0303475Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0352304Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T01:03:09.0396267Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0440992Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T01:03:09.0483977Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0530310Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T01:03:09.0573532Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0620539Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T01:03:09.0669064Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0714801Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T01:03:09.0761956Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0809059Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T01:03:09.0851738Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0895217Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T01:03:09.0939729Z http.https://github.com/.extraheader 2024-12-18T01:03:09.0989028Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T01:03:09.1034216Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1082842Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T01:03:09.1126684Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1195885Z Entering 'third_party/pocketfft' 2024-12-18T01:03:09.1240724Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1286635Z Entering 'third_party/protobuf' 2024-12-18T01:03:09.1331831Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1380767Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T01:03:09.1429608Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1473968Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T01:03:09.1518255Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1566658Z Entering 'third_party/psimd' 2024-12-18T01:03:09.1611604Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1656634Z Entering 'third_party/pthreadpool' 2024-12-18T01:03:09.1701821Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1749220Z Entering 'third_party/pybind11' 2024-12-18T01:03:09.1795145Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1845175Z Entering 'third_party/python-peachpy' 2024-12-18T01:03:09.1890853Z http.https://github.com/.extraheader 2024-12-18T01:03:09.1935968Z Entering 'third_party/sleef' 2024-12-18T01:03:09.1982744Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2029101Z Entering 'third_party/tensorpipe' 2024-12-18T01:03:09.2073669Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2117832Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T01:03:09.2163319Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2211744Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T01:03:09.2254659Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2299741Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T01:03:09.2345588Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2391009Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T01:03:09.2440459Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2484936Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T01:03:09.2532844Z http.https://github.com/.extraheader 2024-12-18T01:03:09.2704515Z A job completed hook has been configured by the self-hosted runner administrator 2024-12-18T01:03:09.2729182Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-12-18T01:03:09.2737037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T01:03:09.2737628Z ##[endgroup] 2024-12-18T01:03:09.2851147Z [!ALERT!] Swap in detected! [!ALERT!] 2024-12-18T01:03:21.2029890Z [!ALERT!] Swap out detected [!ALERT!] 2024-12-18T01:03:40.0721062Z Cleaning up orphan processes