2024-06-26T06:31:50.3458342Z Current runner version: '2.317.0' 2024-06-26T06:31:50.3467420Z Runner name: 'i-05c3ee73f141ef674' 2024-06-26T06:31:50.3468495Z Runner group name: 'Default' 2024-06-26T06:31:50.3469665Z Machine name: 'ip-10-0-35-8' 2024-06-26T06:31:50.3475825Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T06:31:50.3478950Z Actions: read 2024-06-26T06:31:50.3479672Z Attestations: read 2024-06-26T06:31:50.3480462Z Checks: read 2024-06-26T06:31:50.3481052Z Contents: read 2024-06-26T06:31:50.3481728Z Deployments: read 2024-06-26T06:31:50.3482450Z Discussions: read 2024-06-26T06:31:50.3483112Z Issues: read 2024-06-26T06:31:50.3483754Z Metadata: read 2024-06-26T06:31:50.3484439Z Packages: read 2024-06-26T06:31:50.3485042Z Pages: read 2024-06-26T06:31:50.3485704Z PullRequests: read 2024-06-26T06:31:50.3486504Z RepositoryProjects: read 2024-06-26T06:31:50.3487236Z SecurityEvents: read 2024-06-26T06:31:50.3487942Z Statuses: read 2024-06-26T06:31:50.3488632Z ##[endgroup] 2024-06-26T06:31:50.3493516Z Secret source: Actions 2024-06-26T06:31:50.3494692Z Prepare workflow directory 2024-06-26T06:31:50.8313690Z Prepare all required actions 2024-06-26T06:31:50.8580335Z Getting action download info 2024-06-26T06:31:51.0528863Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T06:31:51.4417239Z Download action repository 'pytorch/pytorch@main' (SHA:000f2d637bf383567bd62289f12c868c3afea02d) 2024-06-26T06:31:55.1786399Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T06:31:55.3458382Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T06:31:55.6481488Z Getting action download info 2024-06-26T06:31:55.7424676Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T06:31:55.9357458Z Getting action download info 2024-06-26T06:31:56.0241345Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T06:31:56.1665475Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/129470/merge (4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d) 2024-06-26T06:31:56.1669240Z ##[group] Inputs 2024-06-26T06:31:56.1669833Z build-environment: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:31:56.1671983Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-06-26T06:31:56.1674760Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:31:56.1676110Z sync-tag: 2024-06-26T06:31:56.1677721Z timeout-minutes: 360 2024-06-26T06:31:56.1678191Z use-gha: 2024-06-26T06:31:56.1678561Z dashboard-tag: 2024-06-26T06:31:56.1678947Z s3-bucket: gha-artifacts 2024-06-26T06:31:56.1679455Z aws-role-to-assume: 2024-06-26T06:31:56.1679887Z ##[endgroup] 2024-06-26T06:31:56.1681188Z Complete job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:31:56.2679522Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T06:31:56.2893980Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:31:56.2906718Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:31:56.2907394Z ##[endgroup] 2024-06-26T06:31:57.1630993Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T06:31:57.1631756Z with: 2024-06-26T06:31:57.1632641Z github-secret: *** 2024-06-26T06:31:57.1633863Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T06:31:57.1635436Z activate-with-label: false 2024-06-26T06:31:57.1636389Z label: with-ssh 2024-06-26T06:31:57.1636895Z remove-existing-keys: true 2024-06-26T06:31:57.1637246Z fail-silently: true 2024-06-26T06:31:57.1637626Z env: 2024-06-26T06:31:57.1637961Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:31:57.1638393Z ##[endgroup] 2024-06-26T06:31:57.2865806Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T06:31:57.5549478Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:31:57.6189260Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:31:57.6190158Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:31:57.6879157Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:31:57.7016614Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T06:31:57.7017328Z with: 2024-06-26T06:31:57.7017671Z submodules: recursive 2024-06-26T06:31:57.7018107Z fetch-depth: 0 2024-06-26T06:31:57.7018445Z env: 2024-06-26T06:31:57.7018776Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:31:57.7019204Z ##[endgroup] 2024-06-26T06:31:57.7282877Z ##[group]Run retry () { 2024-06-26T06:31:57.7283358Z retry () { 2024-06-26T06:31:57.7283997Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T06:31:57.7284705Z } 2024-06-26T06:31:57.7285067Z echo "${GITHUB_WORKSPACE}" 2024-06-26T06:31:57.7285601Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T06:31:57.7286200Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:31:57.7286755Z else 2024-06-26T06:31:57.7287183Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:31:57.7287721Z fi 2024-06-26T06:31:57.7288144Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T06:31:57.7296775Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:31:57.7297418Z env: 2024-06-26T06:31:57.7297773Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:31:57.7298186Z NO_SUDO: 2024-06-26T06:31:57.7298531Z ##[endgroup] 2024-06-26T06:31:57.7326008Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:32:01.4315554Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T06:32:01.4316448Z with: 2024-06-26T06:32:01.4316941Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:32:01.4317506Z fetch-depth: 0 2024-06-26T06:32:01.4317889Z submodules: recursive 2024-06-26T06:32:01.4318290Z quiet-checkout: true 2024-06-26T06:32:01.4318720Z repository: pytorch/pytorch 2024-06-26T06:32:01.4319338Z token: *** 2024-06-26T06:32:01.4319677Z ssh-strict: true 2024-06-26T06:32:01.4320080Z persist-credentials: true 2024-06-26T06:32:01.4320520Z clean: true 2024-06-26T06:32:01.4320911Z sparse-checkout-cone-mode: true 2024-06-26T06:32:01.4321391Z lfs: false 2024-06-26T06:32:01.4321759Z set-safe-directory: true 2024-06-26T06:32:01.4322158Z env: 2024-06-26T06:32:01.4322484Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:32:01.4322926Z ##[endgroup] 2024-06-26T06:32:01.5759450Z Syncing repository: pytorch/pytorch 2024-06-26T06:32:01.5763264Z ##[group]Getting Git version info 2024-06-26T06:32:01.5765132Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:32:01.5767506Z [command]/usr/bin/git version 2024-06-26T06:32:01.5768569Z git version 2.40.1 2024-06-26T06:32:01.5772615Z ##[endgroup] 2024-06-26T06:32:01.5799654Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/825c886e-c1cc-4d88-9f02-c7f632028041' before making global git config changes 2024-06-26T06:32:01.5802998Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:32:01.5810234Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:32:01.5859736Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:32:01.5867638Z ##[group]Initializing the repository 2024-06-26T06:32:01.5873727Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:32:01.5907209Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T06:32:01.5908892Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T06:32:01.5911036Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T06:32:01.5912250Z hint: 2024-06-26T06:32:01.5913479Z hint: git config --global init.defaultBranch 2024-06-26T06:32:01.5914772Z hint: 2024-06-26T06:32:01.5916463Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T06:32:01.5919239Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T06:32:01.5920867Z hint: 2024-06-26T06:32:01.5921874Z hint: git branch -m 2024-06-26T06:32:01.5924211Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T06:32:01.5927821Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T06:32:01.5962536Z ##[endgroup] 2024-06-26T06:32:01.5963839Z ##[group]Disabling automatic garbage collection 2024-06-26T06:32:01.5968004Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T06:32:01.6004249Z ##[endgroup] 2024-06-26T06:32:01.6005448Z ##[group]Setting up auth 2024-06-26T06:32:01.6016145Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:32:01.6055980Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:32:01.6354250Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:32:01.6390452Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:32:01.6686682Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:32:01.6739360Z ##[endgroup] 2024-06-26T06:32:01.6740612Z ##[group]Fetching the repository 2024-06-26T06:32:01.6750957Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T06:32:04.0478581Z remote: Enumerating objects: 987625 2024-06-26T06:32:04.0479795Z remote: Enumerating objects: 990184, done. 2024-06-26T06:32:04.0481644Z remote: Counting objects: 0% (1/2559) 2024-06-26T06:32:04.0482367Z remote: Counting objects: 1% (26/2559) 2024-06-26T06:32:04.0483070Z remote: Counting objects: 2% (52/2559) 2024-06-26T06:32:04.0483829Z remote: Counting objects: 3% (77/2559) 2024-06-26T06:32:04.0485596Z remote: Counting objects: 4% (103/2559) 2024-06-26T06:32:04.0487050Z remote: Counting objects: 5% (128/2559) 2024-06-26T06:32:04.0488416Z remote: Counting objects: 6% (154/2559) 2024-06-26T06:32:04.0489916Z remote: Counting objects: 7% (180/2559) 2024-06-26T06:32:04.0491355Z remote: Counting objects: 8% (205/2559) 2024-06-26T06:32:04.0492674Z remote: Counting objects: 9% (231/2559) 2024-06-26T06:32:04.0494163Z remote: Counting objects: 10% (256/2559) 2024-06-26T06:32:04.0495557Z remote: Counting objects: 11% (282/2559) 2024-06-26T06:32:04.0496955Z remote: Counting objects: 12% (308/2559) 2024-06-26T06:32:04.0498040Z remote: Counting objects: 13% (333/2559) 2024-06-26T06:32:04.0499793Z remote: Counting objects: 14% (359/2559) 2024-06-26T06:32:04.0501068Z remote: Counting objects: 15% (384/2559) 2024-06-26T06:32:04.0502513Z remote: Counting objects: 16% (410/2559) 2024-06-26T06:32:04.0503356Z remote: Counting objects: 17% (436/2559) 2024-06-26T06:32:04.0505262Z remote: Counting objects: 18% (461/2559) 2024-06-26T06:32:04.0506290Z remote: Counting objects: 19% (487/2559) 2024-06-26T06:32:04.0507622Z remote: Counting objects: 20% (512/2559) 2024-06-26T06:32:04.0508746Z remote: Counting objects: 21% (538/2559) 2024-06-26T06:32:04.0509923Z remote: Counting objects: 22% (563/2559) 2024-06-26T06:32:04.0510821Z remote: Counting objects: 23% (589/2559) 2024-06-26T06:32:04.0512201Z remote: Counting objects: 24% (615/2559) 2024-06-26T06:32:04.0513293Z remote: Counting objects: 25% (640/2559) 2024-06-26T06:32:04.0514564Z remote: Counting objects: 26% (666/2559) 2024-06-26T06:32:04.0515921Z remote: Counting objects: 27% (691/2559) 2024-06-26T06:32:04.0518051Z remote: Counting objects: 28% (717/2559) 2024-06-26T06:32:04.0519466Z remote: Counting objects: 29% (743/2559) 2024-06-26T06:32:04.0520904Z remote: Counting objects: 30% (768/2559) 2024-06-26T06:32:04.0522085Z remote: Counting objects: 31% (794/2559) 2024-06-26T06:32:04.0523281Z remote: Counting objects: 32% (819/2559) 2024-06-26T06:32:04.0524251Z remote: Counting objects: 33% (845/2559) 2024-06-26T06:32:04.0525527Z remote: Counting objects: 34% (871/2559) 2024-06-26T06:32:04.0526257Z remote: Counting objects: 35% (896/2559) 2024-06-26T06:32:04.0527122Z remote: Counting objects: 36% (922/2559) 2024-06-26T06:32:04.0528792Z remote: Counting objects: 37% (947/2559) 2024-06-26T06:32:04.0530105Z remote: Counting objects: 38% (973/2559) 2024-06-26T06:32:04.0531300Z remote: Counting objects: 39% (999/2559) 2024-06-26T06:32:04.0532037Z remote: Counting objects: 40% (1024/2559) 2024-06-26T06:32:04.0532788Z remote: Counting objects: 41% (1050/2559) 2024-06-26T06:32:04.0533541Z remote: Counting objects: 42% (1075/2559) 2024-06-26T06:32:04.0534977Z remote: Counting objects: 43% (1101/2559) 2024-06-26T06:32:04.0535879Z remote: Counting objects: 44% (1126/2559) 2024-06-26T06:32:04.0537106Z remote: Counting objects: 45% (1152/2559) 2024-06-26T06:32:04.0538354Z remote: Counting objects: 46% (1178/2559) 2024-06-26T06:32:04.0539103Z remote: Counting objects: 47% (1203/2559) 2024-06-26T06:32:04.0539819Z remote: Counting objects: 48% (1229/2559) 2024-06-26T06:32:04.0540289Z remote: Counting objects: 49% (1254/2559) 2024-06-26T06:32:04.0540946Z remote: Counting objects: 50% (1280/2559) 2024-06-26T06:32:04.0541659Z remote: Counting objects: 51% (1306/2559) 2024-06-26T06:32:04.0542393Z remote: Counting objects: 52% (1331/2559) 2024-06-26T06:32:04.0543140Z remote: Counting objects: 53% (1357/2559) 2024-06-26T06:32:04.0543853Z remote: Counting objects: 54% (1382/2559) 2024-06-26T06:32:04.0544588Z remote: Counting objects: 55% (1408/2559) 2024-06-26T06:32:04.0545373Z remote: Counting objects: 56% (1434/2559) 2024-06-26T06:32:04.0546048Z remote: Counting objects: 57% (1459/2559) 2024-06-26T06:32:04.0546785Z remote: Counting objects: 58% (1485/2559) 2024-06-26T06:32:04.0547518Z remote: Counting objects: 59% (1510/2559) 2024-06-26T06:32:04.0548229Z remote: Counting objects: 60% (1536/2559) 2024-06-26T06:32:04.0548966Z remote: Counting objects: 61% (1561/2559) 2024-06-26T06:32:04.0549699Z remote: Counting objects: 62% (1587/2559) 2024-06-26T06:32:04.0550449Z remote: Counting objects: 63% (1613/2559) 2024-06-26T06:32:04.0551352Z remote: Counting objects: 64% (1638/2559) 2024-06-26T06:32:04.0552099Z remote: Counting objects: 65% (1664/2559) 2024-06-26T06:32:04.0552842Z remote: Counting objects: 66% (1689/2559) 2024-06-26T06:32:04.0553579Z remote: Counting objects: 67% (1715/2559) 2024-06-26T06:32:04.0554294Z remote: Counting objects: 68% (1741/2559) 2024-06-26T06:32:04.0555276Z remote: Counting objects: 69% (1766/2559) 2024-06-26T06:32:04.0556054Z remote: Counting objects: 70% (1792/2559) 2024-06-26T06:32:04.0557009Z remote: Counting objects: 71% (1817/2559) 2024-06-26T06:32:04.0557741Z remote: Counting objects: 72% (1843/2559) 2024-06-26T06:32:04.0558474Z remote: Counting objects: 73% (1869/2559) 2024-06-26T06:32:04.0559185Z remote: Counting objects: 74% (1894/2559) 2024-06-26T06:32:04.0559914Z remote: Counting objects: 75% (1920/2559) 2024-06-26T06:32:04.0560641Z remote: Counting objects: 76% (1945/2559) 2024-06-26T06:32:04.0561351Z remote: Counting objects: 77% (1971/2559) 2024-06-26T06:32:04.0562080Z remote: Counting objects: 78% (1997/2559) 2024-06-26T06:32:04.0562959Z remote: Counting objects: 79% (2022/2559) 2024-06-26T06:32:04.0563704Z remote: Counting objects: 80% (2048/2559) 2024-06-26T06:32:04.0564434Z remote: Counting objects: 81% (2073/2559) 2024-06-26T06:32:04.0565174Z remote: Counting objects: 82% (2099/2559) 2024-06-26T06:32:04.0565883Z remote: Counting objects: 83% (2124/2559) 2024-06-26T06:32:04.0566616Z remote: Counting objects: 84% (2150/2559) 2024-06-26T06:32:04.0567373Z remote: Counting objects: 85% (2176/2559) 2024-06-26T06:32:04.0568102Z remote: Counting objects: 86% (2201/2559) 2024-06-26T06:32:04.0568811Z remote: Counting objects: 87% (2227/2559) 2024-06-26T06:32:04.0569539Z remote: Counting objects: 88% (2252/2559) 2024-06-26T06:32:04.0570268Z remote: Counting objects: 89% (2278/2559) 2024-06-26T06:32:04.0570977Z remote: Counting objects: 90% (2304/2559) 2024-06-26T06:32:04.0571706Z remote: Counting objects: 91% (2329/2559) 2024-06-26T06:32:04.0572446Z remote: Counting objects: 92% (2355/2559) 2024-06-26T06:32:04.0573157Z remote: Counting objects: 93% (2380/2559) 2024-06-26T06:32:04.0573889Z remote: Counting objects: 94% (2406/2559) 2024-06-26T06:32:04.0574624Z remote: Counting objects: 95% (2432/2559) 2024-06-26T06:32:04.0575332Z remote: Counting objects: 96% (2457/2559) 2024-06-26T06:32:04.0576060Z remote: Counting objects: 97% (2483/2559) 2024-06-26T06:32:04.0576792Z remote: Counting objects: 98% (2508/2559) 2024-06-26T06:32:04.0577501Z remote: Counting objects: 99% (2534/2559) 2024-06-26T06:32:04.0578233Z remote: Counting objects: 100% (2559/2559) 2024-06-26T06:32:04.0579007Z remote: Counting objects: 100% (2559/2559), done. 2024-06-26T06:32:04.0713544Z remote: Compressing objects: 0% (1/1092) 2024-06-26T06:32:04.1113363Z remote: Compressing objects: 1% (11/1092) 2024-06-26T06:32:04.1430363Z remote: Compressing objects: 2% (22/1092) 2024-06-26T06:32:04.1831949Z remote: Compressing objects: 3% (33/1092) 2024-06-26T06:32:04.2014009Z remote: Compressing objects: 4% (44/1092) 2024-06-26T06:32:04.2260939Z remote: Compressing objects: 5% (55/1092) 2024-06-26T06:32:04.2435512Z remote: Compressing objects: 6% (66/1092) 2024-06-26T06:32:04.2723176Z remote: Compressing objects: 7% (77/1092) 2024-06-26T06:32:04.3158568Z remote: Compressing objects: 8% (88/1092) 2024-06-26T06:32:04.3807737Z remote: Compressing objects: 9% (99/1092) 2024-06-26T06:32:04.4722237Z remote: Compressing objects: 10% (110/1092) 2024-06-26T06:32:04.5829254Z remote: Compressing objects: 11% (121/1092) 2024-06-26T06:32:04.6212273Z remote: Compressing objects: 12% (132/1092) 2024-06-26T06:32:04.6586906Z remote: Compressing objects: 13% (142/1092) 2024-06-26T06:32:04.6869945Z remote: Compressing objects: 14% (153/1092) 2024-06-26T06:32:04.7005106Z remote: Compressing objects: 15% (164/1092) 2024-06-26T06:32:04.7184319Z remote: Compressing objects: 16% (175/1092) 2024-06-26T06:32:04.7252419Z remote: Compressing objects: 17% (186/1092) 2024-06-26T06:32:04.7308463Z remote: Compressing objects: 18% (197/1092) 2024-06-26T06:32:04.7322568Z remote: Compressing objects: 19% (208/1092) 2024-06-26T06:32:04.7345481Z remote: Compressing objects: 20% (219/1092) 2024-06-26T06:32:04.7358569Z remote: Compressing objects: 21% (230/1092) 2024-06-26T06:32:04.7377692Z remote: Compressing objects: 22% (241/1092) 2024-06-26T06:32:04.7388009Z remote: Compressing objects: 23% (252/1092) 2024-06-26T06:32:04.7411744Z remote: Compressing objects: 24% (263/1092) 2024-06-26T06:32:04.7413315Z remote: Compressing objects: 25% (273/1092) 2024-06-26T06:32:04.7425529Z remote: Compressing objects: 26% (284/1092) 2024-06-26T06:32:04.7431370Z remote: Compressing objects: 27% (295/1092) 2024-06-26T06:32:04.7436953Z remote: Compressing objects: 28% (306/1092) 2024-06-26T06:32:04.7441132Z remote: Compressing objects: 29% (317/1092) 2024-06-26T06:32:04.7448371Z remote: Compressing objects: 30% (328/1092) 2024-06-26T06:32:04.7449787Z remote: Compressing objects: 31% (339/1092) 2024-06-26T06:32:04.7456861Z remote: Compressing objects: 32% (350/1092) 2024-06-26T06:32:04.7466854Z remote: Compressing objects: 33% (361/1092) 2024-06-26T06:32:04.7468429Z remote: Compressing objects: 34% (372/1092) 2024-06-26T06:32:04.7480224Z remote: Compressing objects: 35% (383/1092) 2024-06-26T06:32:04.7492941Z remote: Compressing objects: 36% (394/1092) 2024-06-26T06:32:04.7502455Z remote: Compressing objects: 37% (405/1092) 2024-06-26T06:32:04.7521705Z remote: Compressing objects: 38% (415/1092) 2024-06-26T06:32:04.7535061Z remote: Compressing objects: 39% (426/1092) 2024-06-26T06:32:04.7551262Z remote: Compressing objects: 40% (437/1092) 2024-06-26T06:32:04.7571282Z remote: Compressing objects: 41% (448/1092) 2024-06-26T06:32:04.7582973Z remote: Compressing objects: 42% (459/1092) 2024-06-26T06:32:04.7599528Z remote: Compressing objects: 43% (470/1092) 2024-06-26T06:32:04.7613628Z remote: Compressing objects: 44% (481/1092) 2024-06-26T06:32:04.7617719Z remote: Compressing objects: 45% (492/1092) 2024-06-26T06:32:04.7620441Z remote: Compressing objects: 46% (503/1092) 2024-06-26T06:32:04.7625277Z remote: Compressing objects: 47% (514/1092) 2024-06-26T06:32:04.7631456Z remote: Compressing objects: 48% (525/1092) 2024-06-26T06:32:04.7647731Z remote: Compressing objects: 49% (536/1092) 2024-06-26T06:32:04.7668419Z remote: Compressing objects: 50% (546/1092) 2024-06-26T06:32:04.7671831Z remote: Compressing objects: 51% (557/1092) 2024-06-26T06:32:04.7674009Z remote: Compressing objects: 52% (568/1092) 2024-06-26T06:32:04.7680332Z remote: Compressing objects: 53% (579/1092) 2024-06-26T06:32:04.7693558Z remote: Compressing objects: 54% (590/1092) 2024-06-26T06:32:04.7705403Z remote: Compressing objects: 55% (601/1092) 2024-06-26T06:32:04.7721901Z remote: Compressing objects: 56% (612/1092) 2024-06-26T06:32:04.7736081Z remote: Compressing objects: 57% (623/1092) 2024-06-26T06:32:04.7742717Z remote: Compressing objects: 58% (634/1092) 2024-06-26T06:32:04.7770919Z remote: Compressing objects: 59% (645/1092) 2024-06-26T06:32:04.7774931Z remote: Compressing objects: 60% (656/1092) 2024-06-26T06:32:04.7781409Z remote: Compressing objects: 61% (667/1092) 2024-06-26T06:32:04.7786571Z remote: Compressing objects: 62% (678/1092) 2024-06-26T06:32:04.7797293Z remote: Compressing objects: 63% (688/1092) 2024-06-26T06:32:04.7808971Z remote: Compressing objects: 64% (699/1092) 2024-06-26T06:32:04.7813021Z remote: Compressing objects: 65% (710/1092) 2024-06-26T06:32:04.7823055Z remote: Compressing objects: 66% (721/1092) 2024-06-26T06:32:04.7833080Z remote: Compressing objects: 67% (732/1092) 2024-06-26T06:32:04.7836826Z remote: Compressing objects: 68% (743/1092) 2024-06-26T06:32:04.7840702Z remote: Compressing objects: 69% (754/1092) 2024-06-26T06:32:04.7850803Z remote: Compressing objects: 70% (765/1092) 2024-06-26T06:32:04.7854363Z remote: Compressing objects: 71% (776/1092) 2024-06-26T06:32:04.7860250Z remote: Compressing objects: 72% (787/1092) 2024-06-26T06:32:04.7864685Z remote: Compressing objects: 73% (798/1092) 2024-06-26T06:32:04.7868485Z remote: Compressing objects: 74% (809/1092) 2024-06-26T06:32:04.7872773Z remote: Compressing objects: 75% (819/1092) 2024-06-26T06:32:04.7873554Z remote: Compressing objects: 76% (830/1092) 2024-06-26T06:32:04.7874519Z remote: Compressing objects: 77% (841/1092) 2024-06-26T06:32:04.7876452Z remote: Compressing objects: 78% (852/1092) 2024-06-26T06:32:04.7878094Z remote: Compressing objects: 79% (863/1092) 2024-06-26T06:32:04.7878913Z remote: Compressing objects: 80% (874/1092) 2024-06-26T06:32:04.7885013Z remote: Compressing objects: 81% (885/1092) 2024-06-26T06:32:04.7890875Z remote: Compressing objects: 82% (896/1092) 2024-06-26T06:32:04.7895249Z remote: Compressing objects: 83% (907/1092) 2024-06-26T06:32:04.7898735Z remote: Compressing objects: 84% (918/1092) 2024-06-26T06:32:04.7908620Z remote: Compressing objects: 85% (929/1092) 2024-06-26T06:32:04.7909795Z remote: Compressing objects: 86% (940/1092) 2024-06-26T06:32:04.7913255Z remote: Compressing objects: 87% (951/1092) 2024-06-26T06:32:04.7914011Z remote: Compressing objects: 88% (961/1092) 2024-06-26T06:32:04.7914773Z remote: Compressing objects: 89% (972/1092) 2024-06-26T06:32:04.7917200Z remote: Compressing objects: 90% (983/1092) 2024-06-26T06:32:04.7917993Z remote: Compressing objects: 91% (994/1092) 2024-06-26T06:32:04.7918473Z remote: Compressing objects: 92% (1005/1092) 2024-06-26T06:32:04.7920790Z remote: Compressing objects: 93% (1016/1092) 2024-06-26T06:32:04.7923770Z remote: Compressing objects: 94% (1027/1092) 2024-06-26T06:32:04.7924545Z remote: Compressing objects: 95% (1038/1092) 2024-06-26T06:32:04.7927460Z remote: Compressing objects: 96% (1049/1092) 2024-06-26T06:32:04.7928254Z remote: Compressing objects: 97% (1060/1092) 2024-06-26T06:32:04.7934497Z remote: Compressing objects: 98% (1071/1092) 2024-06-26T06:32:04.7935280Z remote: Compressing objects: 99% (1082/1092) 2024-06-26T06:32:04.7936033Z remote: Compressing objects: 100% (1092/1092) 2024-06-26T06:32:04.7936493Z remote: Compressing objects: 100% (1092/1092), done. 2024-06-26T06:32:31.6527002Z remote: Total 990184 (delta 2041), reused 1828 (delta 1466), pack-reused 987625 2024-06-26T06:32:53.9820781Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T06:32:53.9849311Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:32:53.9856726Z ##[endgroup] 2024-06-26T06:32:53.9858582Z ##[group]Determining the checkout info 2024-06-26T06:32:53.9862427Z ##[endgroup] 2024-06-26T06:32:53.9863987Z ##[group]Checking out the ref 2024-06-26T06:32:53.9865948Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:32:55.7792339Z ##[endgroup] 2024-06-26T06:32:55.7794141Z ##[group]Setting up auth for fetching submodules 2024-06-26T06:32:55.7801264Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:32:55.7860291Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T06:32:55.7895699Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T06:32:55.7932439Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T06:32:55.7964367Z ##[endgroup] 2024-06-26T06:32:55.7965859Z ##[group]Fetching submodules 2024-06-26T06:32:55.7971085Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T06:32:55.8292391Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T06:32:55.8595739Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T06:32:55.8598754Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T06:32:55.8601865Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T06:32:55.8605260Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T06:32:55.8608891Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T06:32:55.8611838Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T06:32:55.8615644Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T06:32:55.8619509Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T06:32:55.8623327Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T06:32:55.8627390Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T06:32:55.8631306Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T06:32:55.8635402Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T06:32:55.8640221Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T06:32:55.8644961Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T06:32:55.8649365Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T06:32:55.8654190Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T06:32:55.8659324Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:32:55.8664204Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T06:32:55.8669464Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T06:32:55.8674601Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T06:32:55.8680641Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T06:32:55.8686374Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T06:32:55.8692344Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T06:32:55.8698141Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T06:32:55.8704116Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T06:32:55.8710142Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T06:32:55.8717276Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T06:32:55.8723536Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T06:32:55.8730010Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T06:32:55.8736585Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T06:32:55.8743522Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T06:32:55.8750517Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T06:32:55.8757923Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T06:32:55.8765231Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T06:32:55.8772678Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T06:32:55.8799866Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T06:32:56.1626185Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T06:32:56.3580679Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T06:32:56.5139610Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T06:32:56.7819444Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T06:32:59.0271900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T06:33:09.7424887Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T06:33:10.1454335Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T06:33:10.6864790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T06:33:11.3421605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T06:33:12.7850117Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T06:33:14.9632002Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T06:33:22.1512360Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T06:33:23.6827423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T06:33:25.5231014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T06:33:26.8779079Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T06:33:27.0289763Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T06:33:27.4514573Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T06:33:27.8010773Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T06:33:28.9660341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T06:33:29.3108231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T06:33:29.5635641Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T06:33:31.5137770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T06:33:32.3321672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T06:33:33.0770454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T06:33:39.8828788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T06:33:42.4643806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T06:33:49.0534460Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T06:33:49.2696192Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T06:33:58.9517173Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T06:33:59.1270785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T06:33:59.3299225Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T06:34:00.4897038Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T06:34:00.8040605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T06:34:01.4605281Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T06:34:01.9678593Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T06:34:01.9807352Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T06:34:01.9901798Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T06:34:02.0186907Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T06:34:02.0651421Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T06:34:03.2511862Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T06:34:03.2776222Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T06:34:03.3254165Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T06:34:03.4455752Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T06:34:03.4841196Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T06:34:04.1322144Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T06:34:04.4384027Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T06:34:04.5295109Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T06:34:04.5313071Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:34:04.5317020Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:34:04.5320744Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:34:04.5324397Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:34:04.5328479Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:34:04.5354168Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T06:34:05.5872656Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T06:34:06.2273210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T06:34:08.4287191Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T06:34:09.4818025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T06:34:09.8555523Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T06:34:09.9745808Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T06:34:10.4995006Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T06:34:10.5752381Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T06:34:10.5886702Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T06:34:10.7263966Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T06:34:10.7667429Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T06:34:10.7767797Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T06:34:10.8248587Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T06:34:10.8539826Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T06:34:10.9077819Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T06:34:10.9218333Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T06:34:10.9235258Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T06:34:10.9262031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T06:34:24.9601834Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T06:34:24.9799949Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T06:34:25.0939193Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T06:34:25.0958020Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:34:25.0960689Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:34:25.0964467Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:34:25.0990352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T06:34:25.6805325Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T06:34:27.0403361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T06:34:28.2166095Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T06:34:28.2183168Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:34:28.2186055Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:34:28.2189749Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:34:28.2193012Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:34:28.2197020Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:34:28.2201345Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:34:28.2204600Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:34:28.2208962Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:34:28.2236400Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T06:34:29.3862680Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T06:34:29.7575041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T06:34:31.1285752Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T06:34:31.4508484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T06:34:31.9792068Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T06:34:33.0370841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T06:34:40.8768757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T06:34:41.2946319Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T06:34:41.3155493Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T06:34:41.3593248Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T06:34:41.3736694Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T06:34:41.3753578Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:34:41.3780657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T06:34:41.6931906Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T06:34:41.7144300Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T06:34:41.7624544Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T06:34:41.8910318Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T06:34:41.9096316Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T06:34:41.9526438Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T06:34:42.0229922Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T06:34:42.0681766Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T06:34:42.0969793Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T06:34:42.2295135Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T06:34:42.6580739Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T06:34:42.6618161Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T06:34:42.6620818Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T06:34:42.6648562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T06:34:43.0654809Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T06:34:44.4728056Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T06:34:44.5139159Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T06:34:44.5967564Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T06:34:44.5988791Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:34:44.5991204Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:34:44.5994833Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:34:44.5998047Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:34:44.6002651Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:34:44.6005974Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:34:44.6009903Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:34:44.6013637Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:34:44.6041916Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T06:34:45.0709341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T06:34:46.1822009Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T06:34:46.4964014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T06:34:53.4061297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T06:34:53.6784869Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T06:34:53.8622114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T06:34:54.2511813Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T06:35:02.5234872Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T06:35:02.5718680Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T06:35:02.5892333Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T06:35:02.7152121Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T06:35:02.7292490Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T06:35:02.7452628Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T06:35:02.7622575Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T06:35:02.7638504Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:02.7641702Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:02.7667058Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T06:35:04.7377527Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T06:35:06.1528070Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T06:35:06.2072585Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T06:35:06.7751569Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T06:35:06.7873807Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T06:35:07.1116174Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T06:35:07.1139059Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:07.1141908Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:07.1169938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T06:35:07.5247517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T06:35:08.6732773Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T06:35:08.7600064Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T06:35:08.7694811Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T06:35:08.7829562Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T06:35:08.8243558Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T06:35:08.8576626Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T06:35:08.9072560Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T06:35:08.9370120Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T06:35:08.9388708Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:08.9392100Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:08.9395876Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:08.9400627Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:08.9427805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T06:35:10.0529883Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T06:35:10.3053131Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T06:35:11.5137103Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T06:35:12.7056694Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T06:35:12.7223407Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T06:35:12.8054972Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T06:35:12.8384926Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T06:35:12.8402098Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:12.8430286Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T06:35:13.0558656Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T06:35:13.0596953Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T06:35:13.0921705Z Entering 'android/libs/fbjni' 2024-06-26T06:35:13.0963307Z Entering 'third_party/FP16' 2024-06-26T06:35:13.1004572Z Entering 'third_party/FXdiv' 2024-06-26T06:35:13.1044699Z Entering 'third_party/NNPACK' 2024-06-26T06:35:13.1085542Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:35:13.1126866Z Entering 'third_party/XNNPACK' 2024-06-26T06:35:13.1186161Z Entering 'third_party/benchmark' 2024-06-26T06:35:13.1227465Z Entering 'third_party/cpp-httplib' 2024-06-26T06:35:13.1268578Z Entering 'third_party/cpuinfo' 2024-06-26T06:35:13.1309637Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:35:13.1351016Z Entering 'third_party/cutlass' 2024-06-26T06:35:13.1400466Z Entering 'third_party/eigen' 2024-06-26T06:35:13.1443609Z Entering 'third_party/fbgemm' 2024-06-26T06:35:13.1485866Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:35:13.1525653Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:35:13.1565598Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:35:13.1612409Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:35:13.1653147Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:35:13.1693202Z Entering 'third_party/flatbuffers' 2024-06-26T06:35:13.1737161Z Entering 'third_party/fmt' 2024-06-26T06:35:13.1777695Z Entering 'third_party/foxi' 2024-06-26T06:35:13.1820382Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:35:13.1860655Z Entering 'third_party/gloo' 2024-06-26T06:35:13.1901563Z Entering 'third_party/googletest' 2024-06-26T06:35:13.1943135Z Entering 'third_party/ideep' 2024-06-26T06:35:13.1982521Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:35:13.2030479Z Entering 'third_party/ittapi' 2024-06-26T06:35:13.2070606Z Entering 'third_party/kineto' 2024-06-26T06:35:13.2111496Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:35:13.2152017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:35:13.2195300Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:35:13.2237102Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:35:13.2278475Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:35:13.2319063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:35:13.2362219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:35:13.2403314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:35:13.2443535Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:35:13.2486076Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:35:13.2528301Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:35:13.2568603Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:35:13.2609152Z Entering 'third_party/mimalloc' 2024-06-26T06:35:13.2650051Z Entering 'third_party/nccl/nccl' 2024-06-26T06:35:13.2690840Z Entering 'third_party/nlohmann' 2024-06-26T06:35:13.2732382Z Entering 'third_party/onnx' 2024-06-26T06:35:13.2789565Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:35:13.2831572Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:35:13.2878940Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:35:13.2922202Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:35:13.2962919Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:35:13.3004302Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:35:13.3044143Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:35:13.3085316Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:35:13.3125149Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:35:13.3164683Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:35:13.3203796Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:13.3245635Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:13.3286866Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:35:13.3347263Z Entering 'third_party/pocketfft' 2024-06-26T06:35:13.3388147Z Entering 'third_party/protobuf' 2024-06-26T06:35:13.3433122Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:13.3473689Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:13.3517327Z Entering 'third_party/psimd' 2024-06-26T06:35:13.3558759Z Entering 'third_party/pthreadpool' 2024-06-26T06:35:13.3600183Z Entering 'third_party/pybind11' 2024-06-26T06:35:13.3640797Z Entering 'third_party/python-peachpy' 2024-06-26T06:35:13.3681771Z Entering 'third_party/sleef' 2024-06-26T06:35:13.3722797Z Entering 'third_party/tensorpipe' 2024-06-26T06:35:13.3763343Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:13.3803552Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:13.3843502Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:13.3884785Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:13.3925578Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:13.3980544Z ##[endgroup] 2024-06-26T06:35:13.3983304Z ##[group]Persisting credentials for submodules 2024-06-26T06:35:13.3987818Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T06:35:13.4302828Z Entering 'android/libs/fbjni' 2024-06-26T06:35:13.4353261Z Entering 'third_party/FP16' 2024-06-26T06:35:13.4404985Z Entering 'third_party/FXdiv' 2024-06-26T06:35:13.4456880Z Entering 'third_party/NNPACK' 2024-06-26T06:35:13.4507667Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:35:13.4558704Z Entering 'third_party/XNNPACK' 2024-06-26T06:35:13.4627509Z Entering 'third_party/benchmark' 2024-06-26T06:35:13.4678904Z Entering 'third_party/cpp-httplib' 2024-06-26T06:35:13.4729391Z Entering 'third_party/cpuinfo' 2024-06-26T06:35:13.4780464Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:35:13.4830615Z Entering 'third_party/cutlass' 2024-06-26T06:35:13.4890339Z Entering 'third_party/eigen' 2024-06-26T06:35:13.4943721Z Entering 'third_party/fbgemm' 2024-06-26T06:35:13.4993973Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:35:13.5045032Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:35:13.5094921Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:35:13.5152501Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:35:13.5203242Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:35:13.5253754Z Entering 'third_party/flatbuffers' 2024-06-26T06:35:13.5308282Z Entering 'third_party/fmt' 2024-06-26T06:35:13.5359708Z Entering 'third_party/foxi' 2024-06-26T06:35:13.5410291Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:35:13.5460299Z Entering 'third_party/gloo' 2024-06-26T06:35:13.5511636Z Entering 'third_party/googletest' 2024-06-26T06:35:13.5563992Z Entering 'third_party/ideep' 2024-06-26T06:35:13.5613005Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:35:13.5670218Z Entering 'third_party/ittapi' 2024-06-26T06:35:13.5722420Z Entering 'third_party/kineto' 2024-06-26T06:35:13.5773557Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:35:13.5823595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:35:13.5875699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:35:13.5928405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:35:13.5979719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:35:13.6029730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:35:13.6083897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:35:13.6133000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:35:13.6183443Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:35:13.6235294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:35:13.6287145Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:35:13.6338924Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:35:13.6389441Z Entering 'third_party/mimalloc' 2024-06-26T06:35:13.6441851Z Entering 'third_party/nccl/nccl' 2024-06-26T06:35:13.6492548Z Entering 'third_party/nlohmann' 2024-06-26T06:35:13.6545082Z Entering 'third_party/onnx' 2024-06-26T06:35:13.6614683Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:35:13.6666429Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:35:13.6720280Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:35:13.6772921Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:35:13.6822606Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:35:13.6872504Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:35:13.6923556Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:35:13.6973591Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:35:13.7023444Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:35:13.7073017Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:35:13.7123669Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:13.7175250Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:13.7226896Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:35:13.7298888Z Entering 'third_party/pocketfft' 2024-06-26T06:35:13.7349675Z Entering 'third_party/protobuf' 2024-06-26T06:35:13.7405328Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:13.7454548Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:13.7506680Z Entering 'third_party/psimd' 2024-06-26T06:35:13.7557033Z Entering 'third_party/pthreadpool' 2024-06-26T06:35:13.7607969Z Entering 'third_party/pybind11' 2024-06-26T06:35:13.7659271Z Entering 'third_party/python-peachpy' 2024-06-26T06:35:13.7710376Z Entering 'third_party/sleef' 2024-06-26T06:35:13.7762678Z Entering 'third_party/tensorpipe' 2024-06-26T06:35:13.7812620Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:13.7862758Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:13.7913198Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:13.7963836Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:13.8013001Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:13.8081885Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T06:35:13.8388725Z Entering 'android/libs/fbjni' 2024-06-26T06:35:13.8436924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T06:35:13.8456525Z Entering 'third_party/FP16' 2024-06-26T06:35:13.8503660Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T06:35:13.8522398Z Entering 'third_party/FXdiv' 2024-06-26T06:35:13.8569787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T06:35:13.8588652Z Entering 'third_party/NNPACK' 2024-06-26T06:35:13.8635659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T06:35:13.8654610Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:35:13.8702453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T06:35:13.8721435Z Entering 'third_party/XNNPACK' 2024-06-26T06:35:13.8768537Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T06:35:13.8805511Z Entering 'third_party/benchmark' 2024-06-26T06:35:13.8853530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:35:13.8871691Z Entering 'third_party/cpp-httplib' 2024-06-26T06:35:13.8919629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T06:35:13.8937997Z Entering 'third_party/cpuinfo' 2024-06-26T06:35:13.8985127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T06:35:13.9004380Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:35:13.9051216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T06:35:13.9070204Z Entering 'third_party/cutlass' 2024-06-26T06:35:13.9118146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T06:35:13.9144890Z Entering 'third_party/eigen' 2024-06-26T06:35:13.9191752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T06:35:13.9212847Z Entering 'third_party/fbgemm' 2024-06-26T06:35:13.9260246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T06:35:13.9278642Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:35:13.9326094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T06:35:13.9344124Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:35:13.9390438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T06:35:13.9408972Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:35:13.9455203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T06:35:13.9481406Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:35:13.9527787Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:13.9545350Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:35:13.9592308Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T06:35:13.9611213Z Entering 'third_party/flatbuffers' 2024-06-26T06:35:13.9658419Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T06:35:13.9680245Z Entering 'third_party/fmt' 2024-06-26T06:35:13.9727370Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T06:35:13.9745737Z Entering 'third_party/foxi' 2024-06-26T06:35:13.9792113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T06:35:13.9810336Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:35:13.9857555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T06:35:13.9875703Z Entering 'third_party/gloo' 2024-06-26T06:35:13.9923094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T06:35:13.9942003Z Entering 'third_party/googletest' 2024-06-26T06:35:13.9990167Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.0008812Z Entering 'third_party/ideep' 2024-06-26T06:35:14.0057116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T06:35:14.0074427Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:35:14.0122347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T06:35:14.0148120Z Entering 'third_party/ittapi' 2024-06-26T06:35:14.0195699Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T06:35:14.0214137Z Entering 'third_party/kineto' 2024-06-26T06:35:14.0273289Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T06:35:14.0281125Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:35:14.0329377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T06:35:14.0346624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:35:14.0394275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T06:35:14.0414291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:35:14.0463585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T06:35:14.0481824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:35:14.0529175Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T06:35:14.0547027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:35:14.0595241Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T06:35:14.0611921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:35:14.0660026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T06:35:14.0679921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:35:14.0727842Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T06:35:14.0745352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:35:14.0793585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.0812113Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:35:14.0858984Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T06:35:14.0878222Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:35:14.0926020Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T06:35:14.0944803Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:35:14.0991972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T06:35:14.1010066Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:35:14.1057711Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.1076778Z Entering 'third_party/mimalloc' 2024-06-26T06:35:14.1124185Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T06:35:14.1142776Z Entering 'third_party/nccl/nccl' 2024-06-26T06:35:14.1190173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T06:35:14.1209307Z Entering 'third_party/nlohmann' 2024-06-26T06:35:14.1256342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T06:35:14.1275977Z Entering 'third_party/onnx' 2024-06-26T06:35:14.1324372Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T06:35:14.1360608Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:35:14.1408353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:35:14.1426790Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:35:14.1473094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:35:14.1493616Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:35:14.1542068Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T06:35:14.1562557Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:35:14.1608824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:35:14.1626468Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:35:14.1674213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.1691640Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:35:14.1738804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T06:35:14.1756566Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:35:14.1803629Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T06:35:14.1822782Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:35:14.1869268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T06:35:14.1887117Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:35:14.1934337Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T06:35:14.1952099Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:35:14.2000193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T06:35:14.2017037Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:14.2064929Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T06:35:14.2084980Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:14.2132378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T06:35:14.2151991Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:35:14.2199525Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T06:35:14.2238721Z Entering 'third_party/pocketfft' 2024-06-26T06:35:14.2285304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T06:35:14.2303421Z Entering 'third_party/protobuf' 2024-06-26T06:35:14.2350672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T06:35:14.2372096Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:14.2418992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T06:35:14.2437278Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:14.2484823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.2504547Z Entering 'third_party/psimd' 2024-06-26T06:35:14.2552173Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T06:35:14.2570503Z Entering 'third_party/pthreadpool' 2024-06-26T06:35:14.2619156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T06:35:14.2636929Z Entering 'third_party/pybind11' 2024-06-26T06:35:14.2684465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:35:14.2703077Z Entering 'third_party/python-peachpy' 2024-06-26T06:35:14.2751102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T06:35:14.2769621Z Entering 'third_party/sleef' 2024-06-26T06:35:14.2817097Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T06:35:14.2835031Z Entering 'third_party/tensorpipe' 2024-06-26T06:35:14.2882838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T06:35:14.2900673Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:14.2948112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T06:35:14.2965780Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:14.3012526Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T06:35:14.3029954Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:14.3076646Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T06:35:14.3094872Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:14.3141281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T06:35:14.3158702Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:14.3206399Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T06:35:14.4352328Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T06:35:14.4661399Z Entering 'android/libs/fbjni' 2024-06-26T06:35:14.4702643Z Entering 'third_party/FP16' 2024-06-26T06:35:14.4744807Z Entering 'third_party/FXdiv' 2024-06-26T06:35:14.4786128Z Entering 'third_party/NNPACK' 2024-06-26T06:35:14.4827945Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:35:14.4869700Z Entering 'third_party/XNNPACK' 2024-06-26T06:35:14.4929262Z Entering 'third_party/benchmark' 2024-06-26T06:35:14.4971274Z Entering 'third_party/cpp-httplib' 2024-06-26T06:35:14.5011376Z Entering 'third_party/cpuinfo' 2024-06-26T06:35:14.5053862Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:35:14.5095778Z Entering 'third_party/cutlass' 2024-06-26T06:35:14.5146097Z Entering 'third_party/eigen' 2024-06-26T06:35:14.5189172Z Entering 'third_party/fbgemm' 2024-06-26T06:35:14.5231434Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:35:14.5272080Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:35:14.5313200Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:35:14.5360895Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:35:14.5402088Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:35:14.5443111Z Entering 'third_party/flatbuffers' 2024-06-26T06:35:14.5487393Z Entering 'third_party/fmt' 2024-06-26T06:35:14.5528531Z Entering 'third_party/foxi' 2024-06-26T06:35:14.5569402Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:35:14.5610372Z Entering 'third_party/gloo' 2024-06-26T06:35:14.5651146Z Entering 'third_party/googletest' 2024-06-26T06:35:14.5691685Z Entering 'third_party/ideep' 2024-06-26T06:35:14.5733097Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:35:14.5781225Z Entering 'third_party/ittapi' 2024-06-26T06:35:14.5824007Z Entering 'third_party/kineto' 2024-06-26T06:35:14.5865140Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:35:14.5906874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:35:14.5951206Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:35:14.5991825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:35:14.6033225Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:35:14.6072518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:35:14.6117587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:35:14.6158073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:35:14.6198895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:35:14.6242184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:35:14.6284011Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:35:14.6324394Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:35:14.6366068Z Entering 'third_party/mimalloc' 2024-06-26T06:35:14.6408754Z Entering 'third_party/nccl/nccl' 2024-06-26T06:35:14.6450394Z Entering 'third_party/nlohmann' 2024-06-26T06:35:14.6493781Z Entering 'third_party/onnx' 2024-06-26T06:35:14.6553454Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:35:14.6596433Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:35:14.6661318Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:35:14.6704531Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:35:14.6746804Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:35:14.6787388Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:35:14.6828443Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:35:14.6870468Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:35:14.6911512Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:35:14.6951249Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:35:14.6992774Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:14.7037845Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:14.7080032Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:35:14.7144824Z Entering 'third_party/pocketfft' 2024-06-26T06:35:14.7187801Z Entering 'third_party/protobuf' 2024-06-26T06:35:14.7234528Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:14.7275293Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:14.7318590Z Entering 'third_party/psimd' 2024-06-26T06:35:14.7359698Z Entering 'third_party/pthreadpool' 2024-06-26T06:35:14.7401220Z Entering 'third_party/pybind11' 2024-06-26T06:35:14.7443920Z Entering 'third_party/python-peachpy' 2024-06-26T06:35:14.7487032Z Entering 'third_party/sleef' 2024-06-26T06:35:14.7529431Z Entering 'third_party/tensorpipe' 2024-06-26T06:35:14.7571828Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:14.7613497Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:14.7654189Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:14.7696590Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:14.7736911Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:14.7795530Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T06:35:14.8110847Z Entering 'android/libs/fbjni' 2024-06-26T06:35:14.8153856Z Entering 'third_party/FP16' 2024-06-26T06:35:14.8194913Z Entering 'third_party/FXdiv' 2024-06-26T06:35:14.8236956Z Entering 'third_party/NNPACK' 2024-06-26T06:35:14.8278673Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:35:14.8321619Z Entering 'third_party/XNNPACK' 2024-06-26T06:35:14.8382074Z Entering 'third_party/benchmark' 2024-06-26T06:35:14.8424269Z Entering 'third_party/cpp-httplib' 2024-06-26T06:35:14.8466258Z Entering 'third_party/cpuinfo' 2024-06-26T06:35:14.8509644Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:35:14.8552912Z Entering 'third_party/cutlass' 2024-06-26T06:35:14.8605203Z Entering 'third_party/eigen' 2024-06-26T06:35:14.8649828Z Entering 'third_party/fbgemm' 2024-06-26T06:35:14.8692059Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:35:14.8734372Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:35:14.8776030Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:35:14.8824435Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:35:14.8865458Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:35:14.8907982Z Entering 'third_party/flatbuffers' 2024-06-26T06:35:14.8952803Z Entering 'third_party/fmt' 2024-06-26T06:35:14.8994927Z Entering 'third_party/foxi' 2024-06-26T06:35:14.9037056Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:35:14.9078911Z Entering 'third_party/gloo' 2024-06-26T06:35:14.9122177Z Entering 'third_party/googletest' 2024-06-26T06:35:14.9164701Z Entering 'third_party/ideep' 2024-06-26T06:35:14.9205500Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:35:14.9253788Z Entering 'third_party/ittapi' 2024-06-26T06:35:14.9295483Z Entering 'third_party/kineto' 2024-06-26T06:35:14.9337729Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:35:14.9378648Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:35:14.9421594Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:35:14.9462364Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:35:14.9503566Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:35:14.9543593Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:35:14.9585917Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:35:14.9626273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:35:14.9667249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:35:14.9708678Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:35:14.9751451Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:35:14.9793710Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:35:14.9835670Z Entering 'third_party/mimalloc' 2024-06-26T06:35:14.9877423Z Entering 'third_party/nccl/nccl' 2024-06-26T06:35:14.9918640Z Entering 'third_party/nlohmann' 2024-06-26T06:35:14.9962092Z Entering 'third_party/onnx' 2024-06-26T06:35:15.0022137Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:35:15.0063897Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:35:15.0106586Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:35:15.0150609Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:35:15.0191046Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:35:15.0231333Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:35:15.0271195Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:35:15.0313576Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:35:15.0353542Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:35:15.0394285Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:35:15.0434553Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:35:15.0477077Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:35:15.0519031Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:35:15.0581099Z Entering 'third_party/pocketfft' 2024-06-26T06:35:15.0621939Z Entering 'third_party/protobuf' 2024-06-26T06:35:15.0667535Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:35:15.0707924Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:35:15.0749767Z Entering 'third_party/psimd' 2024-06-26T06:35:15.0791616Z Entering 'third_party/pthreadpool' 2024-06-26T06:35:15.0832882Z Entering 'third_party/pybind11' 2024-06-26T06:35:15.0874739Z Entering 'third_party/python-peachpy' 2024-06-26T06:35:15.0915711Z Entering 'third_party/sleef' 2024-06-26T06:35:15.0956789Z Entering 'third_party/tensorpipe' 2024-06-26T06:35:15.0998417Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:35:15.1039413Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:35:15.1080199Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:35:15.1120999Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:35:15.1160230Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:35:15.1214994Z ##[endgroup] 2024-06-26T06:35:15.1256695Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T06:35:15.1282842Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T06:35:15.1515647Z Prepare all required actions 2024-06-26T06:35:15.1516508Z Getting action download info 2024-06-26T06:35:15.2937562Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T06:35:15.2938099Z env: 2024-06-26T06:35:15.2938442Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:15.2938855Z ##[endgroup] 2024-06-26T06:35:15.3021573Z ##[group]Run set -euo pipefail 2024-06-26T06:35:15.3022112Z set -euo pipefail 2024-06-26T06:35:15.3022580Z function get_ec2_metadata() { 2024-06-26T06:35:15.3023248Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T06:35:15.3024574Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T06:35:15.3025547Z  category=$1 2024-06-26T06:35:15.3026211Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T06:35:15.3027008Z  runner_name_str=i-05c3ee73f141ef674 2024-06-26T06:35:15.3027629Z  if [[ -f /.inarc ]]; then 2024-06-26T06:35:15.3028250Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T06:35:15.3028945Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T06:35:15.3029823Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T06:35:15.3030597Z  else 2024-06-26T06:35:15.3031197Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T06:35:15.3031922Z  fi 2024-06-26T06:35:15.3032277Z } 2024-06-26T06:35:15.3032706Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T06:35:15.3033452Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T06:35:15.3034283Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T06:35:15.3034978Z echo "system info $(uname -a)" 2024-06-26T06:35:15.3043930Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:15.3044560Z env: 2024-06-26T06:35:15.3044894Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:15.3045335Z ##[endgroup] 2024-06-26T06:35:15.3145738Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T06:35:15.3209335Z instance-id: i-05c3ee73f141ef674 2024-06-26T06:35:15.3272320Z instance-type: g3.8xlarge 2024-06-26T06:35:15.3281293Z system info Linux ip-10-0-35-8.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T06:35:15.3329439Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T06:35:15.3330597Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T06:35:15.3338700Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:15.3339322Z env: 2024-06-26T06:35:15.3339659Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:15.3340072Z ##[endgroup] 2024-06-26T06:35:15.3421184Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T06:35:15.3421946Z if systemctl is-active --quiet docker; then 2024-06-26T06:35:15.3422625Z  echo "Docker daemon is running..."; 2024-06-26T06:35:15.3423187Z else 2024-06-26T06:35:15.3423790Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T06:35:15.3424524Z fi 2024-06-26T06:35:15.3432072Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:15.3432680Z env: 2024-06-26T06:35:15.3433016Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:15.3433447Z ##[endgroup] 2024-06-26T06:35:15.3481063Z Docker daemon is running... 2024-06-26T06:35:15.3540100Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:35:15.3540814Z with: 2024-06-26T06:35:15.3541141Z shell: bash 2024-06-26T06:35:15.3541498Z timeout_minutes: 5 2024-06-26T06:35:15.3541900Z max_attempts: 3 2024-06-26T06:35:15.3542293Z retry_wait_seconds: 30 2024-06-26T06:35:15.3544264Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T06:35:15.3546166Z polling_interval_seconds: 1 2024-06-26T06:35:15.3546638Z warning_on_retry: true 2024-06-26T06:35:15.3547077Z continue_on_error: false 2024-06-26T06:35:15.3547482Z env: 2024-06-26T06:35:15.3547816Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:15.3548267Z AWS_RETRY_MODE: standard 2024-06-26T06:35:15.3548689Z AWS_MAX_ATTEMPTS: 5 2024-06-26T06:35:15.3549113Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T06:35:15.3549732Z ##[endgroup] 2024-06-26T06:35:16.3638081Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:35:16.3639102Z Configure a credential helper to remove this warning. See 2024-06-26T06:35:16.3640168Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:35:16.3640843Z 2024-06-26T06:35:16.3641036Z Login Succeeded 2024-06-26T06:35:16.4260470Z Command completed after 1 attempt(s). 2024-06-26T06:35:16.4318128Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:35:16.4319037Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:35:16.4319863Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T06:35:16.4328351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:16.4328962Z env: 2024-06-26T06:35:16.4329302Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.4329739Z ##[endgroup] 2024-06-26T06:35:16.4410833Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:35:16.4411804Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:35:16.4412546Z # shellcheck disable=SC2046 2024-06-26T06:35:16.4413114Z docker stop $(docker ps -q) || true 2024-06-26T06:35:16.4413704Z # Prune all of the docker images 2024-06-26T06:35:16.4414313Z docker system prune -af 2024-06-26T06:35:16.4422188Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:16.4422788Z env: 2024-06-26T06:35:16.4423124Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.4423554Z ##[endgroup] 2024-06-26T06:35:16.4760605Z "docker stop" requires at least 1 argument. 2024-06-26T06:35:16.4761368Z See 'docker stop --help'. 2024-06-26T06:35:16.4761681Z 2024-06-26T06:35:16.4761854Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T06:35:16.4764096Z 2024-06-26T06:35:16.4764647Z Stop one or more running containers 2024-06-26T06:35:16.4949290Z Total reclaimed space: 0B 2024-06-26T06:35:16.4985069Z ##[group]Run set +e 2024-06-26T06:35:16.4985514Z set +e 2024-06-26T06:35:16.4985864Z set -x 2024-06-26T06:35:16.4986222Z  2024-06-26T06:35:16.4986617Z PT_DOMAIN=download.pytorch.org 2024-06-26T06:35:16.4987614Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T06:35:16.4988990Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T06:35:16.4989943Z # one is returned at random 2024-06-26T06:35:16.4990602Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T06:35:16.4991243Z  2024-06-26T06:35:16.4991626Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T06:35:16.4992379Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T06:35:16.4993331Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T06:35:16.4994041Z  2024-06-26T06:35:16.4994445Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T06:35:16.4995170Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T06:35:16.4995806Z  exit 1 2024-06-26T06:35:16.4996655Z  fi 2024-06-26T06:35:16.4997255Z fi 2024-06-26T06:35:16.4997627Z  2024-06-26T06:35:16.4998098Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T06:35:16.4998722Z  # Clean up any old records first 2024-06-26T06:35:16.4999352Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T06:35:16.4999911Z fi 2024-06-26T06:35:16.5000229Z  2024-06-26T06:35:16.5000752Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T06:35:16.5001435Z cat /etc/hosts 2024-06-26T06:35:16.5009048Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:16.5009833Z env: 2024-06-26T06:35:16.5010167Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.5010577Z ##[endgroup] 2024-06-26T06:35:16.5034366Z + PT_DOMAIN=download.pytorch.org 2024-06-26T06:35:16.5040048Z ++ dig -4 +short download.pytorch.org 2024-06-26T06:35:16.5040700Z ++ tail -n1 2024-06-26T06:35:16.5486300Z + RESOLVED_IP=18.160.18.20 2024-06-26T06:35:16.5486850Z + '[' -z 18.160.18.20 ']' 2024-06-26T06:35:16.5487342Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T06:35:16.5495390Z 18.67.76.42 download.pytorch.org 2024-06-26T06:35:16.5496476Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T06:35:16.5639566Z + echo '18.160.18.20 download.pytorch.org' 2024-06-26T06:35:16.5640620Z + sudo tee -a /etc/hosts 2024-06-26T06:35:16.5892026Z 18.160.18.20 download.pytorch.org 2024-06-26T06:35:16.5908266Z + cat /etc/hosts 2024-06-26T06:35:16.5914346Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T06:35:16.5929060Z ::1 localhost6 localhost6.localdomain6 2024-06-26T06:35:16.5929763Z 18.160.18.20 download.pytorch.org 2024-06-26T06:35:16.6134678Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T06:35:16.6135470Z with: 2024-06-26T06:35:16.6136678Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6138075Z docker-build-dir: .ci/docker 2024-06-26T06:35:16.6138541Z working-directory: . 2024-06-26T06:35:16.6139098Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:16.6139827Z force-push: false 2024-06-26T06:35:16.6140238Z env: 2024-06-26T06:35:16.6140569Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.6141030Z ##[endgroup] 2024-06-26T06:35:16.6168518Z ##[group]Run set -ex 2024-06-26T06:35:16.6168982Z set -ex 2024-06-26T06:35:16.6169346Z  2024-06-26T06:35:16.6170026Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T06:35:16.6171328Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T06:35:16.6172344Z # job could then download the pre-built image as usual 2024-06-26T06:35:16.6173275Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T06:35:16.6174125Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6174919Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6175604Z  2024-06-26T06:35:16.6176243Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T06:35:16.6177029Z  exit 0 2024-06-26T06:35:16.6177378Z else 2024-06-26T06:35:16.6177829Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6178392Z fi 2024-06-26T06:35:16.6178713Z  2024-06-26T06:35:16.6179295Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T06:35:16.6180397Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T06:35:16.6181365Z  # use it as it is, but first let's extract the tag 2024-06-26T06:35:16.6182252Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T06:35:16.6183169Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6184047Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6184735Z else 2024-06-26T06:35:16.6185277Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T06:35:16.6186112Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6187225Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6188180Z fi 2024-06-26T06:35:16.6196535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:16.6197192Z env: 2024-06-26T06:35:16.6197526Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.6197946Z REPO_NAME: pytorch 2024-06-26T06:35:16.6199204Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6200567Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T06:35:16.6201183Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:16.6201808Z ##[endgroup] 2024-06-26T06:35:16.6227528Z + [[ ! -d .ci/docker ]] 2024-06-26T06:35:16.6228018Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T06:35:16.6228492Z + echo skip=false 2024-06-26T06:35:16.6230624Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T06:35:16.6235799Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6237521Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T06:35:16.6245447Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6246286Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6248538Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6283620Z ##[group]Run set +e 2024-06-26T06:35:16.6284067Z set +e 2024-06-26T06:35:16.6284446Z set -x 2024-06-26T06:35:16.6284785Z  2024-06-26T06:35:16.6285123Z login() { 2024-06-26T06:35:16.6285964Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T06:35:16.6286904Z } 2024-06-26T06:35:16.6287224Z  2024-06-26T06:35:16.6287570Z retry () { 2024-06-26T06:35:16.6288039Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T06:35:16.6288596Z } 2024-06-26T06:35:16.6288928Z  2024-06-26T06:35:16.6289296Z retry login "${DOCKER_REGISTRY}" 2024-06-26T06:35:16.6289818Z  2024-06-26T06:35:16.6290410Z # Check if image already exists, if it does then skip building it 2024-06-26T06:35:16.6291313Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T06:35:16.6291934Z  exit 0 2024-06-26T06:35:16.6292304Z fi 2024-06-26T06:35:16.6292642Z  2024-06-26T06:35:16.6293257Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T06:35:16.6294326Z # be empty. The default action would be to continue rebuild the image 2024-06-26T06:35:16.6295339Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T06:35:16.6296196Z  # if we're on the base branch then use the parent commit 2024-06-26T06:35:16.6296947Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T06:35:16.6297488Z else 2024-06-26T06:35:16.6298058Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T06:35:16.6298938Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T06:35:16.6299576Z fi 2024-06-26T06:35:16.6299900Z  2024-06-26T06:35:16.6300287Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T06:35:16.6300927Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6301496Z  2024-06-26T06:35:16.6302313Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T06:35:16.6303294Z  exit 0 2024-06-26T06:35:16.6303662Z fi 2024-06-26T06:35:16.6303980Z  2024-06-26T06:35:16.6304704Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T06:35:16.6305963Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T06:35:16.6306985Z  exit 1 2024-06-26T06:35:16.6307357Z fi 2024-06-26T06:35:16.6307694Z  2024-06-26T06:35:16.6308288Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T06:35:16.6309515Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T06:35:16.6310633Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T06:35:16.6311881Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T06:35:16.6313288Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T06:35:16.6314130Z fi 2024-06-26T06:35:16.6314471Z  2024-06-26T06:35:16.6315123Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T06:35:16.6323146Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:16.6323775Z env: 2024-06-26T06:35:16.6324093Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:16.6324550Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T06:35:16.6325325Z BASE_REVISION: 4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:35:16.6326578Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6328017Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:16.6328768Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:16.6329443Z ##[endgroup] 2024-06-26T06:35:16.6354364Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:16.6355174Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:16.6357092Z + aws ecr get-login-password --region us-east-1 2024-06-26T06:35:16.6358563Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.1033311Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:35:17.1034214Z Configure a credential helper to remove this warning. See 2024-06-26T06:35:17.1035459Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:35:17.1036443Z 2024-06-26T06:35:17.1036592Z Login Succeeded 2024-06-26T06:35:17.1047859Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.3106945Z { 2024-06-26T06:35:17.3107338Z "schemaVersion": 2, 2024-06-26T06:35:17.3107798Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T06:35:17.3108386Z "config": { 2024-06-26T06:35:17.3110385Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T06:35:17.3111439Z "size": 47470, 2024-06-26T06:35:17.3112139Z "digest": "sha256:1fde65d68e1ba467b7c461d097349da1965328db5e6e5252ccb25702fcc3b83e" 2024-06-26T06:35:17.3112590Z }, 2024-06-26T06:35:17.3113039Z "layers": [ 2024-06-26T06:35:17.3113382Z { 2024-06-26T06:35:17.3114032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3114708Z "size": 28580681, 2024-06-26T06:35:17.3115412Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T06:35:17.3116508Z }, 2024-06-26T06:35:17.3117187Z { 2024-06-26T06:35:17.3117609Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3118199Z "size": 7943353, 2024-06-26T06:35:17.3119105Z "digest": "sha256:09d415c238d76b32a7ea4a6e6add9542db9a5641f7f183af70aae185d0709e58" 2024-06-26T06:35:17.3119914Z }, 2024-06-26T06:35:17.3120289Z { 2024-06-26T06:35:17.3121329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3122001Z "size": 56226389, 2024-06-26T06:35:17.3122774Z "digest": "sha256:9fe6e2e61518cba6844870c03b285737daec35e62baf25ae7744629ed3a7b470" 2024-06-26T06:35:17.3123601Z }, 2024-06-26T06:35:17.3123821Z + exit 0 2024-06-26T06:35:17.3124062Z { 2024-06-26T06:35:17.3124539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3125523Z "size": 187, 2024-06-26T06:35:17.3126100Z "digest": "sha256:41f16248e682693ff20b3032c1d5e5541cc87c5af898ae2ff9b24d2940e59100" 2024-06-26T06:35:17.3127097Z }, 2024-06-26T06:35:17.3127696Z { 2024-06-26T06:35:17.3128753Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3130252Z "size": 6885, 2024-06-26T06:35:17.3131564Z "digest": "sha256:95d7b781703928cf3c4eece39d800cccb76728c375fedf51ecd83833fb25e458" 2024-06-26T06:35:17.3132633Z }, 2024-06-26T06:35:17.3132966Z { 2024-06-26T06:35:17.3134000Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3134479Z "size": 1383662475, 2024-06-26T06:35:17.3135208Z "digest": "sha256:8f6c9048534734f4c873935293b7296225846ceb31c1a158400a67ea170dde7f" 2024-06-26T06:35:17.3136694Z }, 2024-06-26T06:35:17.3137090Z { 2024-06-26T06:35:17.3137669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3139056Z "size": 62282, 2024-06-26T06:35:17.3140333Z "digest": "sha256:ab17245097e491b9368790714f9d90ed447bf0973bd677cfe6f2456d62b72a13" 2024-06-26T06:35:17.3141487Z }, 2024-06-26T06:35:17.3141810Z { 2024-06-26T06:35:17.3142468Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3143701Z "size": 1684, 2024-06-26T06:35:17.3145408Z "digest": "sha256:dfecd7e9912b76ed460b8edd5a85f1943666e38a973ab5458177cf2c7c3110e3" 2024-06-26T06:35:17.3146843Z }, 2024-06-26T06:35:17.3147343Z { 2024-06-26T06:35:17.3148292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3149582Z "size": 1522, 2024-06-26T06:35:17.3150985Z "digest": "sha256:464a8f74544589bf7b57f9a4cadcb6681e5ed00758f6c35025e691df4e88e890" 2024-06-26T06:35:17.3152071Z }, 2024-06-26T06:35:17.3152178Z { 2024-06-26T06:35:17.3152560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3153296Z "size": 2445884909, 2024-06-26T06:35:17.3154014Z "digest": "sha256:d67c111fa58886d8904335b5122039816c8e51af5844cb6fe4e14154d9a7c016" 2024-06-26T06:35:17.3154787Z }, 2024-06-26T06:35:17.3155108Z { 2024-06-26T06:35:17.3155669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3156778Z "size": 85882, 2024-06-26T06:35:17.3157615Z "digest": "sha256:0723abde19c09431db80b5d0f0e003541f6232c2da5c4f35c635c861875937ae" 2024-06-26T06:35:17.3158430Z }, 2024-06-26T06:35:17.3158728Z { 2024-06-26T06:35:17.3159287Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3160029Z "size": 1822, 2024-06-26T06:35:17.3160736Z "digest": "sha256:bb93d41d4cb4616eabdd81918bbf6c8673164e995dd0f0cc6d05989fdce3df21" 2024-06-26T06:35:17.3161561Z }, 2024-06-26T06:35:17.3161884Z { 2024-06-26T06:35:17.3162421Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3163151Z "size": 654858224, 2024-06-26T06:35:17.3163885Z "digest": "sha256:4b89839dd06631d80ca192bae8be4ea835acfde539e7e775339edcce9714c418" 2024-06-26T06:35:17.3164678Z }, 2024-06-26T06:35:17.3164997Z { 2024-06-26T06:35:17.3165553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3166261Z "size": 546, 2024-06-26T06:35:17.3166971Z "digest": "sha256:caa9986c765baa228b63ef2e4e24be796d1aa0f41e20f1a9ccc44423d8a9a3f4" 2024-06-26T06:35:17.3167794Z }, 2024-06-26T06:35:17.3168089Z { 2024-06-26T06:35:17.3168645Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3169633Z "size": 1284, 2024-06-26T06:35:17.3170321Z "digest": "sha256:e17c09bdf243963c1185f01eca2a5ed4ae8e627181e77073452ea8a2811b661d" 2024-06-26T06:35:17.3171128Z }, 2024-06-26T06:35:17.3171444Z { 2024-06-26T06:35:17.3171984Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3172712Z "size": 484, 2024-06-26T06:35:17.3173423Z "digest": "sha256:bd2e32de75c90c1ed985c413bb08813271eeb40a7cb9c926e261d381c6fb4d82" 2024-06-26T06:35:17.3174211Z }, 2024-06-26T06:35:17.3174530Z { 2024-06-26T06:35:17.3175078Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3226144Z "size": 91710828, 2024-06-26T06:35:17.3226890Z "digest": "sha256:559a1daa0f9885fe07609ca2641d3e9e3d4d6aedca6c4d36ad90c1c765582bf0" 2024-06-26T06:35:17.3227683Z }, 2024-06-26T06:35:17.3228093Z { 2024-06-26T06:35:17.3228558Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3229247Z "size": 3235, 2024-06-26T06:35:17.3230208Z "digest": "sha256:d96cec7a425ccddeae1efd3e0f73355515e86e146cbaab2da9739cd651741068" 2024-06-26T06:35:17.3231077Z }, 2024-06-26T06:35:17.3231368Z { 2024-06-26T06:35:17.3231918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3232636Z "size": 1905, 2024-06-26T06:35:17.3233315Z "digest": "sha256:1db6fc6c25902dd64b1dad171454f5b492ecb8895b4f8b6270de2a65750cfe12" 2024-06-26T06:35:17.3234119Z }, 2024-06-26T06:35:17.3234433Z { 2024-06-26T06:35:17.3234957Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3235670Z "size": 700, 2024-06-26T06:35:17.3236766Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:35:17.3237643Z }, 2024-06-26T06:35:17.3237947Z { 2024-06-26T06:35:17.3238493Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3239189Z "size": 2812517131, 2024-06-26T06:35:17.3239914Z "digest": "sha256:a05e0234ee85b8358a1b4c4bf7d9d7c4bc38e991a22895c0f9ffcbc709645fd9" 2024-06-26T06:35:17.3240734Z }, 2024-06-26T06:35:17.3241029Z { 2024-06-26T06:35:17.3241583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3242300Z "size": 381, 2024-06-26T06:35:17.3242960Z "digest": "sha256:44f27f91d8fa036ce2af809752cba0d79d7da4ffe91d4981827319d759d6a230" 2024-06-26T06:35:17.3243753Z }, 2024-06-26T06:35:17.3244068Z { 2024-06-26T06:35:17.3244596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3245315Z "size": 11891, 2024-06-26T06:35:17.3246008Z "digest": "sha256:0a0ded10616759d17568ed6ae0f22ebfe503413d2d4710223b5852c8eeb131b3" 2024-06-26T06:35:17.3246793Z }, 2024-06-26T06:35:17.3247087Z { 2024-06-26T06:35:17.3247637Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3248328Z "size": 803, 2024-06-26T06:35:17.3249025Z "digest": "sha256:0c62bfb80b7cdf32a713040cc54d11d45eec4105a51976da8b88f6e8dd79bb49" 2024-06-26T06:35:17.3249844Z }, 2024-06-26T06:35:17.3250158Z { 2024-06-26T06:35:17.3250716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3251403Z "size": 106, 2024-06-26T06:35:17.3252096Z "digest": "sha256:0c8d1e7f32823458cc5bf0e51b63e2abb70878dea21687edffcf738af41051b6" 2024-06-26T06:35:17.3252898Z }, 2024-06-26T06:35:17.3253188Z { 2024-06-26T06:35:17.3253722Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3254432Z "size": 503, 2024-06-26T06:35:17.3255065Z "digest": "sha256:846366e78885c05844e15926128f2d3113c617522f700e485df868fbbba1c876" 2024-06-26T06:35:17.3255825Z }, 2024-06-26T06:35:17.3256140Z { 2024-06-26T06:35:17.3256664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3257378Z "size": 121477247, 2024-06-26T06:35:17.3258101Z "digest": "sha256:efa11c713d18f2a9dcf7901ba63df5b9766293ca7c7824e5f3c089c960f45162" 2024-06-26T06:35:17.3259112Z }, 2024-06-26T06:35:17.3259427Z { 2024-06-26T06:35:17.3259994Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3260677Z "size": 109, 2024-06-26T06:35:17.3261640Z "digest": "sha256:32f0221befebf94bb7f0f81bdaef0f566e32695e7a98cb9020b555956fc9b89b" 2024-06-26T06:35:17.3262186Z }, 2024-06-26T06:35:17.3262482Z { 2024-06-26T06:35:17.3263033Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3263840Z "size": 490, 2024-06-26T06:35:17.3264524Z "digest": "sha256:a1d57c94ba33612a149ca2fea27363c0f960fd8516723c166d5b2783aabdaf16" 2024-06-26T06:35:17.3265311Z }, 2024-06-26T06:35:17.3265602Z { 2024-06-26T06:35:17.3266150Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3266863Z "size": 295, 2024-06-26T06:35:17.3267521Z "digest": "sha256:3dc15441e6d000a89a95092afb061c4c3c36b263a5159e68b4385e453b793faf" 2024-06-26T06:35:17.3268309Z }, 2024-06-26T06:35:17.3268627Z { 2024-06-26T06:35:17.3269292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3270047Z "size": 103, 2024-06-26T06:35:17.3270731Z "digest": "sha256:668eadf990e86d6ac337335a56269ee7d21c3b7ab9d73a08b6234234070fcd70" 2024-06-26T06:35:17.3271498Z }, 2024-06-26T06:35:17.3271813Z { 2024-06-26T06:35:17.3272367Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3273059Z "size": 1473, 2024-06-26T06:35:17.3273753Z "digest": "sha256:fd5a832fe26e3c2c26c20395f500fc681ab28e9f9812e4b6b3e95aa15442103d" 2024-06-26T06:35:17.3274555Z }, 2024-06-26T06:35:17.3274846Z { 2024-06-26T06:35:17.3275394Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3276496Z "size": 446711131, 2024-06-26T06:35:17.3277306Z "digest": "sha256:b581c3782fd627bac2e6cc292144b92a21b499a07e2b598eaa2a48252e57a537" 2024-06-26T06:35:17.3278110Z }, 2024-06-26T06:35:17.3278434Z { 2024-06-26T06:35:17.3278959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3279685Z "size": 160, 2024-06-26T06:35:17.3280360Z "digest": "sha256:3f42611c56360ba670f886b3c62442f9f556652382512f0153dbd888798fe690" 2024-06-26T06:35:17.3281107Z }, 2024-06-26T06:35:17.3281421Z { 2024-06-26T06:35:17.3281974Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3282661Z "size": 564, 2024-06-26T06:35:17.3283363Z "digest": "sha256:fe8dd930eda117742ffbb011f0c821bd08f9d183b46a81a9dc5fc3fca741b8bf" 2024-06-26T06:35:17.3284174Z }, 2024-06-26T06:35:17.3284468Z { 2024-06-26T06:35:17.3285023Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3285747Z "size": 35807552, 2024-06-26T06:35:17.3286434Z "digest": "sha256:6a9711a8a50dc2f9871121744f60deeb06e436ea2282ab2ed3d153ed4f2360d7" 2024-06-26T06:35:17.3287236Z }, 2024-06-26T06:35:17.3287556Z { 2024-06-26T06:35:17.3288090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3288813Z "size": 104, 2024-06-26T06:35:17.3289517Z "digest": "sha256:76cd931663c9f6f95baad24c47a3f0125db05e116cc221dd313fb9f089af72d6" 2024-06-26T06:35:17.3290292Z }, 2024-06-26T06:35:17.3290608Z { 2024-06-26T06:35:17.3291161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3291850Z "size": 426, 2024-06-26T06:35:17.3292531Z "digest": "sha256:c9ec95615433cc9fb674bbd2820228a5977f981d70ff677927b0caf42d8c56b8" 2024-06-26T06:35:17.3293326Z }, 2024-06-26T06:35:17.3293622Z { 2024-06-26T06:35:17.3294174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3294960Z "size": 20262060, 2024-06-26T06:35:17.3295642Z "digest": "sha256:7da33ab0319397119e259c5ee2c37d7c6cb854983c8a320c59746a27d4826ed2" 2024-06-26T06:35:17.3296423Z }, 2024-06-26T06:35:17.3296743Z { 2024-06-26T06:35:17.3297271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3298206Z "size": 439, 2024-06-26T06:35:17.3298910Z "digest": "sha256:da956bbe6180c03f03d667edd26b2f9a43df083200e13847da28ad27a9f83489" 2024-06-26T06:35:17.3299686Z }, 2024-06-26T06:35:17.3300001Z { 2024-06-26T06:35:17.3300560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3301250Z "size": 700, 2024-06-26T06:35:17.3301951Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:35:17.3302756Z }, 2024-06-26T06:35:17.3303048Z { 2024-06-26T06:35:17.3303813Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3304309Z "size": 143, 2024-06-26T06:35:17.3304966Z "digest": "sha256:73e6b5984544e73e642e8a11fd1dddf14a05913257132ee47f7782b6fb519ee6" 2024-06-26T06:35:17.3305752Z }, 2024-06-26T06:35:17.3306079Z { 2024-06-26T06:35:17.3306603Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3307317Z "size": 135, 2024-06-26T06:35:17.3308174Z "digest": "sha256:e5a6ec0a41d77fcb1af98a4aa079b9219c47c2e5eb696b52883c390373b1af8e" 2024-06-26T06:35:17.3308984Z }, 2024-06-26T06:35:17.3309300Z { 2024-06-26T06:35:17.3309855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3310544Z "size": 32, 2024-06-26T06:35:17.3311238Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3312045Z }, 2024-06-26T06:35:17.3312336Z { 2024-06-26T06:35:17.3312884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3313608Z "size": 191, 2024-06-26T06:35:17.3314280Z "digest": "sha256:4b25a744b402ec8a8cdfd55f95c346bcf9178c450cd08e8bb2834efa41bb621e" 2024-06-26T06:35:17.3315078Z }, 2024-06-26T06:35:17.3315397Z { 2024-06-26T06:35:17.3316524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3317392Z "size": 564, 2024-06-26T06:35:17.3318089Z "digest": "sha256:21f74857bd1744f0b55351469eec060cd60d78fac461e695f550de53cec8edc0" 2024-06-26T06:35:17.3318864Z }, 2024-06-26T06:35:17.3319185Z { 2024-06-26T06:35:17.3319832Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3320526Z "size": 43163151, 2024-06-26T06:35:17.3321236Z "digest": "sha256:6c12f547d202783d251e0813abaeaad2b30ba464d28f5902171b637f829ab84a" 2024-06-26T06:35:17.3322032Z }, 2024-06-26T06:35:17.3322325Z { 2024-06-26T06:35:17.3322877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3323592Z "size": 106, 2024-06-26T06:35:17.3324261Z "digest": "sha256:ed4e5a4f8526fca89a9272c39328cb84dbec72a12e74d4abde50a706f2983b55" 2024-06-26T06:35:17.3325065Z }, 2024-06-26T06:35:17.3325387Z { 2024-06-26T06:35:17.3325915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3326630Z "size": 1212, 2024-06-26T06:35:17.3327328Z "digest": "sha256:6f09f741ead60ea209087bf9071ef1a6fcc299c42ce4b3a297c6d96a6393f67f" 2024-06-26T06:35:17.3328111Z }, 2024-06-26T06:35:17.3328424Z { 2024-06-26T06:35:17.3328983Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3329675Z "size": 700, 2024-06-26T06:35:17.3330375Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:35:17.3331189Z }, 2024-06-26T06:35:17.3331482Z { 2024-06-26T06:35:17.3332032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3332749Z "size": 138, 2024-06-26T06:35:17.3333429Z "digest": "sha256:411fce2db725b6d9a9d7b7f953cdd354ceef707e1dec7a2ef5115beba5a6788e" 2024-06-26T06:35:17.3334245Z }, 2024-06-26T06:35:17.3334565Z { 2024-06-26T06:35:17.3335090Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3335800Z "size": 120, 2024-06-26T06:35:17.3336487Z "digest": "sha256:87f7902d12a4c612e9be6b4cceae10e66baf25332f924496d3b70f1967b83360" 2024-06-26T06:35:17.3337255Z }, 2024-06-26T06:35:17.3337748Z { 2024-06-26T06:35:17.3338314Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3339013Z "size": 1905185822, 2024-06-26T06:35:17.3339715Z "digest": "sha256:9777c07909ada379d44a538999c6492011749a4f0fcbd5dfee534795401df9d3" 2024-06-26T06:35:17.3340485Z }, 2024-06-26T06:35:17.3340780Z { 2024-06-26T06:35:17.3341329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3342036Z "size": 172, 2024-06-26T06:35:17.3342693Z "digest": "sha256:9755a952d3ecbc817b3290445710170b8c63a35f902a9bddec1837eee9a4db57" 2024-06-26T06:35:17.3343476Z }, 2024-06-26T06:35:17.3343790Z { 2024-06-26T06:35:17.3344318Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3345027Z "size": 908, 2024-06-26T06:35:17.3345727Z "digest": "sha256:5ca5d2e8bebe5a5641fd35fc6585f4c83ae8ae4528b84682b276a04b9006faa2" 2024-06-26T06:35:17.3346503Z }, 2024-06-26T06:35:17.3346817Z { 2024-06-26T06:35:17.3347368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3348191Z "size": 700, 2024-06-26T06:35:17.3348924Z "digest": "sha256:ac522bc703457eff4f1ceef81fd329f630b0feb067578ffebb5f721f5dd7a120" 2024-06-26T06:35:17.3349729Z }, 2024-06-26T06:35:17.3350019Z { 2024-06-26T06:35:17.3350570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3351280Z "size": 135, 2024-06-26T06:35:17.3351954Z "digest": "sha256:d07af843db47bac709de490bf12e9f993080a666ddda8ee54210dd4b6526f97d" 2024-06-26T06:35:17.3352757Z }, 2024-06-26T06:35:17.3353082Z { 2024-06-26T06:35:17.3353606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3354310Z "size": 32, 2024-06-26T06:35:17.3354998Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3355801Z }, 2024-06-26T06:35:17.3356463Z { 2024-06-26T06:35:17.3357106Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3357837Z "size": 157, 2024-06-26T06:35:17.3358555Z "digest": "sha256:dbbd7a4ca8469b1fc21ae5f549fc832e79c4dae36515cc0b594a4f8d6531832e" 2024-06-26T06:35:17.3359364Z }, 2024-06-26T06:35:17.3359654Z { 2024-06-26T06:35:17.3360205Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3360918Z "size": 1841, 2024-06-26T06:35:17.3361610Z "digest": "sha256:6a18c61a96dd8f2f6cd96d48d62262059c9b024ffad4cf0df6a7d4c02f58d9e2" 2024-06-26T06:35:17.3362431Z }, 2024-06-26T06:35:17.3362706Z { 2024-06-26T06:35:17.3363231Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3363941Z "size": 7529784, 2024-06-26T06:35:17.3364638Z "digest": "sha256:bd12222c02373373bcdd92c079d4d6e405dee01eb986422d608659c4bb0cd809" 2024-06-26T06:35:17.3365428Z }, 2024-06-26T06:35:17.3365715Z { 2024-06-26T06:35:17.3366261Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3366975Z "size": 163, 2024-06-26T06:35:17.3367659Z "digest": "sha256:a5dfe6ba36fd2223ebbc9e888593b6341d5bd3b56b1674cbd9064f55c26de604" 2024-06-26T06:35:17.3368465Z }, 2024-06-26T06:35:17.3368777Z { 2024-06-26T06:35:17.3369304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3370017Z "size": 7944, 2024-06-26T06:35:17.3370714Z "digest": "sha256:4b990eb76e8f2de191d0c2e639846adb8df24ce3380ec150ead1baa95b26d7ec" 2024-06-26T06:35:17.3371493Z }, 2024-06-26T06:35:17.3371804Z { 2024-06-26T06:35:17.3372353Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3373043Z "size": 8069, 2024-06-26T06:35:17.3373736Z "digest": "sha256:d9df85e8d42263f8089be7b63aaf1e2bfc3207f8d81b1d79f9f86e96c17298ee" 2024-06-26T06:35:17.3374533Z }, 2024-06-26T06:35:17.3374824Z { 2024-06-26T06:35:17.3375370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3376078Z "size": 304, 2024-06-26T06:35:17.3376946Z "digest": "sha256:47d89094b1a7e9cc21b8db8ab9c66fefcc9bf7544c170508f41efb5d26e3ccd2" 2024-06-26T06:35:17.3377737Z }, 2024-06-26T06:35:17.3378007Z { 2024-06-26T06:35:17.3378507Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3379173Z "size": 7628836, 2024-06-26T06:35:17.3379845Z "digest": "sha256:94beee2c356a83536890159dc698da22baca90a4c8289622ed556f5a47947d79" 2024-06-26T06:35:17.3380612Z }, 2024-06-26T06:35:17.3380925Z { 2024-06-26T06:35:17.3381470Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3382160Z "size": 108, 2024-06-26T06:35:17.3382832Z "digest": "sha256:322c644984f492ea6b5b3d0b9eaab0bdd8d29e156e25a60d475a7dbe0c08621f" 2024-06-26T06:35:17.3383609Z }, 2024-06-26T06:35:17.3383905Z { 2024-06-26T06:35:17.3384456Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3385163Z "size": 54145774, 2024-06-26T06:35:17.3385846Z "digest": "sha256:b43ee04d974d5ef5236d381dd8e9d87677715edc695f235bb1e954b54a8f1143" 2024-06-26T06:35:17.3386792Z }, 2024-06-26T06:35:17.3387119Z { 2024-06-26T06:35:17.3387651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3388363Z "size": 474, 2024-06-26T06:35:17.3389033Z "digest": "sha256:40f95cbeec2c62a99f4548e3f7139c89986c010bce3742c85d203fdd5a120c4e" 2024-06-26T06:35:17.3389788Z }, 2024-06-26T06:35:17.3390078Z { 2024-06-26T06:35:17.3390626Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3391316Z "size": 1373229352, 2024-06-26T06:35:17.3392048Z "digest": "sha256:4b2a3c41d6ae93889d0aed7b8b3e87eb334b8fb8aedb20bab9d7fea2759b7055" 2024-06-26T06:35:17.3392857Z }, 2024-06-26T06:35:17.3393150Z { 2024-06-26T06:35:17.3393695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3394408Z "size": 106, 2024-06-26T06:35:17.3395122Z "digest": "sha256:b2689866a0b9bc7b622498695ae245e7f4996dfa2db1311cff0b6fae0ff7e6e8" 2024-06-26T06:35:17.3395915Z }, 2024-06-26T06:35:17.3396420Z { 2024-06-26T06:35:17.3396923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3397604Z "size": 556, 2024-06-26T06:35:17.3398270Z "digest": "sha256:9b36f9d4ace6c70a0fddeca47d17eb19639fd636a6084e02555f7d862bf44c40" 2024-06-26T06:35:17.3399044Z }, 2024-06-26T06:35:17.3399362Z { 2024-06-26T06:35:17.3399917Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3400614Z "size": 36532084, 2024-06-26T06:35:17.3401320Z "digest": "sha256:a863a1c01885d8f3506984d7a7d8179273b3d29ed4b4e2dee41471a6437155f4" 2024-06-26T06:35:17.3402084Z }, 2024-06-26T06:35:17.3402378Z { 2024-06-26T06:35:17.3402889Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3403558Z "size": 111, 2024-06-26T06:35:17.3404238Z "digest": "sha256:b9b8edd214be18381ccf301a0b92a7853f7ecba0b219699cd74c9b2e064183fd" 2024-06-26T06:35:17.3405052Z }, 2024-06-26T06:35:17.3405367Z { 2024-06-26T06:35:17.3405908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3406626Z "size": 32, 2024-06-26T06:35:17.3407322Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3408098Z }, 2024-06-26T06:35:17.3408456Z { 2024-06-26T06:35:17.3409079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3409775Z "size": 32, 2024-06-26T06:35:17.3410462Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3411257Z }, 2024-06-26T06:35:17.3411552Z { 2024-06-26T06:35:17.3412095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3412801Z "size": 32, 2024-06-26T06:35:17.3413470Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3414267Z }, 2024-06-26T06:35:17.3414761Z { 2024-06-26T06:35:17.3415485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T06:35:17.3416200Z "size": 32, 2024-06-26T06:35:17.3416891Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T06:35:17.3417668Z } 2024-06-26T06:35:17.3417982Z ] 2024-06-26T06:35:17.3418292Z } 2024-06-26T06:35:17.3559857Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T06:35:17.3560504Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T06:35:17.3561192Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T06:35:17.3569756Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:17.3570393Z env: 2024-06-26T06:35:17.3570738Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:17.3572026Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.3573376Z ##[endgroup] 2024-06-26T06:35:17.3603157Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.3669863Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T06:35:17.3670603Z with: 2024-06-26T06:35:17.3671788Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.3673309Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.3673934Z env: 2024-06-26T06:35:17.3674271Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:17.3674702Z ##[endgroup] 2024-06-26T06:35:17.3698682Z ##[group]Run set -x 2024-06-26T06:35:17.3699111Z set -x 2024-06-26T06:35:17.3699484Z set +e 2024-06-26T06:35:17.3699845Z  2024-06-26T06:35:17.3700166Z login() { 2024-06-26T06:35:17.3701005Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T06:35:17.3701949Z } 2024-06-26T06:35:17.3702269Z  2024-06-26T06:35:17.3702673Z retry () { 2024-06-26T06:35:17.3703147Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T06:35:17.3703713Z } 2024-06-26T06:35:17.3704054Z  2024-06-26T06:35:17.3704427Z retry login "${DOCKER_REGISTRY}" 2024-06-26T06:35:17.3704949Z  2024-06-26T06:35:17.3705286Z set -e 2024-06-26T06:35:17.3705863Z # ignore output since only exit code is used for conditional 2024-06-26T06:35:17.3706765Z # only pull docker image if it's not available locally 2024-06-26T06:35:17.3707741Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T06:35:17.3708660Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T06:35:17.3709197Z fi 2024-06-26T06:35:17.3717880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:35:17.3718531Z env: 2024-06-26T06:35:17.3718877Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:35:17.3720151Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.3721659Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.3722319Z ##[endgroup] 2024-06-26T06:35:17.3747797Z + set +e 2024-06-26T06:35:17.3748275Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.3749086Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.3752136Z + aws ecr get-login-password --region us-east-1 2024-06-26T06:35:17.3754042Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T06:35:17.8448225Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T06:35:17.8449262Z Configure a credential helper to remove this warning. See 2024-06-26T06:35:17.8451157Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T06:35:17.8451874Z 2024-06-26T06:35:17.8452046Z Login Succeeded 2024-06-26T06:35:17.8461295Z + set -e 2024-06-26T06:35:17.8462774Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.8637502Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:17.8639892Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:35:18.0772467Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9 2024-06-26T06:35:18.0774531Z 7a2c55901189: Pulling fs layer 2024-06-26T06:35:18.0775315Z 09d415c238d7: Pulling fs layer 2024-06-26T06:35:18.0776265Z 9fe6e2e61518: Pulling fs layer 2024-06-26T06:35:18.0777161Z 41f16248e682: Pulling fs layer 2024-06-26T06:35:18.0777970Z 95d7b7817039: Pulling fs layer 2024-06-26T06:35:18.0778474Z 8f6c90485347: Pulling fs layer 2024-06-26T06:35:18.0778939Z ab17245097e4: Pulling fs layer 2024-06-26T06:35:18.0779388Z dfecd7e9912b: Pulling fs layer 2024-06-26T06:35:18.0779858Z 464a8f745445: Pulling fs layer 2024-06-26T06:35:18.0780324Z d67c111fa588: Pulling fs layer 2024-06-26T06:35:18.0780770Z 0723abde19c0: Pulling fs layer 2024-06-26T06:35:18.0781246Z bb93d41d4cb4: Pulling fs layer 2024-06-26T06:35:18.0781707Z 4b89839dd066: Pulling fs layer 2024-06-26T06:35:18.0782153Z caa9986c765b: Pulling fs layer 2024-06-26T06:35:18.0782616Z e17c09bdf243: Pulling fs layer 2024-06-26T06:35:18.0783080Z bd2e32de75c9: Pulling fs layer 2024-06-26T06:35:18.0783936Z 559a1daa0f98: Pulling fs layer 2024-06-26T06:35:18.0784524Z d96cec7a425c: Pulling fs layer 2024-06-26T06:35:18.0785052Z 1db6fc6c2590: Pulling fs layer 2024-06-26T06:35:18.0785531Z ac522bc70345: Pulling fs layer 2024-06-26T06:35:18.0785974Z a05e0234ee85: Pulling fs layer 2024-06-26T06:35:18.0786443Z 44f27f91d8fa: Pulling fs layer 2024-06-26T06:35:18.0786912Z 0a0ded106167: Pulling fs layer 2024-06-26T06:35:18.0787355Z 0c62bfb80b7c: Pulling fs layer 2024-06-26T06:35:18.0787821Z 0c8d1e7f3282: Pulling fs layer 2024-06-26T06:35:18.0788276Z 846366e78885: Pulling fs layer 2024-06-26T06:35:18.0788706Z dfecd7e9912b: Waiting 2024-06-26T06:35:18.0789121Z efa11c713d18: Pulling fs layer 2024-06-26T06:35:18.0789587Z 32f0221befeb: Pulling fs layer 2024-06-26T06:35:18.0790014Z 4b89839dd066: Waiting 2024-06-26T06:35:18.0790425Z a1d57c94ba33: Pulling fs layer 2024-06-26T06:35:18.0790872Z 464a8f745445: Waiting 2024-06-26T06:35:18.0791261Z 3dc15441e6d0: Pulling fs layer 2024-06-26T06:35:18.0791706Z d67c111fa588: Waiting 2024-06-26T06:35:18.0792329Z 0723abde19c0: Waiting 2024-06-26T06:35:18.0792911Z e17c09bdf243: Waiting 2024-06-26T06:35:18.0793314Z bb93d41d4cb4: Waiting 2024-06-26T06:35:18.0793979Z bd2e32de75c9: Waiting 2024-06-26T06:35:18.0794352Z 1db6fc6c2590: Waiting 2024-06-26T06:35:18.0794775Z 559a1daa0f98: Waiting 2024-06-26T06:35:18.0795248Z caa9986c765b: Waiting 2024-06-26T06:35:18.0795619Z 95d7b7817039: Waiting 2024-06-26T06:35:18.0796025Z 668eadf990e8: Pulling fs layer 2024-06-26T06:35:18.0796721Z d96cec7a425c: Waiting 2024-06-26T06:35:18.0797097Z ac522bc70345: Waiting 2024-06-26T06:35:18.0797481Z 8f6c90485347: Waiting 2024-06-26T06:35:18.0798067Z fd5a832fe26e: Pulling fs layer 2024-06-26T06:35:18.0798521Z 32f0221befeb: Waiting 2024-06-26T06:35:18.0798909Z ab17245097e4: Waiting 2024-06-26T06:35:18.0799294Z a05e0234ee85: Waiting 2024-06-26T06:35:18.0799851Z b581c3782fd6: Pulling fs layer 2024-06-26T06:35:18.0800325Z 3f42611c5636: Pulling fs layer 2024-06-26T06:35:18.0800767Z 41f16248e682: Waiting 2024-06-26T06:35:18.0801375Z fe8dd930eda1: Pulling fs layer 2024-06-26T06:35:18.0801864Z efa11c713d18: Waiting 2024-06-26T06:35:18.0802255Z 44f27f91d8fa: Waiting 2024-06-26T06:35:18.0802846Z 6a9711a8a50d: Pulling fs layer 2024-06-26T06:35:18.0803499Z 0c8d1e7f3282: Waiting 2024-06-26T06:35:18.0804238Z 76cd931663c9: Pulling fs layer 2024-06-26T06:35:18.0804676Z 0c62bfb80b7c: Waiting 2024-06-26T06:35:18.0805083Z c9ec95615433: Pulling fs layer 2024-06-26T06:35:18.0805551Z 7da33ab03193: Pulling fs layer 2024-06-26T06:35:18.0805977Z 3f42611c5636: Waiting 2024-06-26T06:35:18.0806385Z da956bbe6180: Pulling fs layer 2024-06-26T06:35:18.0806834Z 3dc15441e6d0: Waiting 2024-06-26T06:35:18.0807200Z 846366e78885: Waiting 2024-06-26T06:35:18.0807587Z fd5a832fe26e: Waiting 2024-06-26T06:35:18.0807976Z b581c3782fd6: Waiting 2024-06-26T06:35:18.0808364Z 73e6b5984544: Pulling fs layer 2024-06-26T06:35:18.0808805Z 76cd931663c9: Waiting 2024-06-26T06:35:18.0809172Z a1d57c94ba33: Waiting 2024-06-26T06:35:18.0809559Z 7da33ab03193: Waiting 2024-06-26T06:35:18.0809964Z e5a6ec0a41d7: Pulling fs layer 2024-06-26T06:35:18.0810604Z da956bbe6180: Waiting 2024-06-26T06:35:18.0811131Z 4f4fb700ef54: Pulling fs layer 2024-06-26T06:35:18.0811588Z e5a6ec0a41d7: Waiting 2024-06-26T06:35:18.0811979Z 4b25a744b402: Pulling fs layer 2024-06-26T06:35:18.0812589Z 6a9711a8a50d: Waiting 2024-06-26T06:35:18.0813020Z 21f74857bd17: Pulling fs layer 2024-06-26T06:35:18.0813463Z 6c12f547d202: Pulling fs layer 2024-06-26T06:35:18.0813903Z 4b25a744b402: Waiting 2024-06-26T06:35:18.0814309Z ed4e5a4f8526: Pulling fs layer 2024-06-26T06:35:18.0814740Z 4f4fb700ef54: Waiting 2024-06-26T06:35:18.0815146Z 6f09f741ead6: Pulling fs layer 2024-06-26T06:35:18.0815611Z 411fce2db725: Pulling fs layer 2024-06-26T06:35:18.0816037Z 73e6b5984544: Waiting 2024-06-26T06:35:18.0816446Z 87f7902d12a4: Pulling fs layer 2024-06-26T06:35:18.0816907Z 9777c07909ad: Pulling fs layer 2024-06-26T06:35:18.0817348Z 9755a952d3ec: Pulling fs layer 2024-06-26T06:35:18.0817814Z 5ca5d2e8bebe: Pulling fs layer 2024-06-26T06:35:18.0818264Z 411fce2db725: Waiting 2024-06-26T06:35:18.0818630Z 87f7902d12a4: Waiting 2024-06-26T06:35:18.0819014Z 6c12f547d202: Waiting 2024-06-26T06:35:18.0819411Z 9777c07909ad: Waiting 2024-06-26T06:35:18.0819796Z d07af843db47: Pulling fs layer 2024-06-26T06:35:18.0820243Z 5ca5d2e8bebe: Waiting 2024-06-26T06:35:18.0820641Z 9755a952d3ec: Waiting 2024-06-26T06:35:18.0821036Z dbbd7a4ca846: Pulling fs layer 2024-06-26T06:35:18.0821509Z 6a18c61a96dd: Pulling fs layer 2024-06-26T06:35:18.0821964Z dbbd7a4ca846: Waiting 2024-06-26T06:35:18.0822340Z 6a18c61a96dd: Waiting 2024-06-26T06:35:18.0822745Z bd12222c0237: Pulling fs layer 2024-06-26T06:35:18.0823198Z d07af843db47: Waiting 2024-06-26T06:35:18.0823588Z a5dfe6ba36fd: Pulling fs layer 2024-06-26T06:35:18.0824066Z 4b990eb76e8f: Pulling fs layer 2024-06-26T06:35:18.0824534Z d9df85e8d422: Pulling fs layer 2024-06-26T06:35:18.0824965Z a5dfe6ba36fd: Waiting 2024-06-26T06:35:18.0825373Z 47d89094b1a7: Pulling fs layer 2024-06-26T06:35:18.0825835Z 94beee2c356a: Pulling fs layer 2024-06-26T06:35:18.0826279Z 322c644984f4: Pulling fs layer 2024-06-26T06:35:18.0826743Z b43ee04d974d: Pulling fs layer 2024-06-26T06:35:18.0827189Z d9df85e8d422: Waiting 2024-06-26T06:35:18.0827591Z 40f95cbeec2c: Pulling fs layer 2024-06-26T06:35:18.0828043Z 94beee2c356a: Waiting 2024-06-26T06:35:18.0828465Z 4b2a3c41d6ae: Pulling fs layer 2024-06-26T06:35:18.0828910Z b2689866a0b9: Pulling fs layer 2024-06-26T06:35:18.0829357Z 4b990eb76e8f: Waiting 2024-06-26T06:35:18.0829746Z b43ee04d974d: Waiting 2024-06-26T06:35:18.0830135Z 9b36f9d4ace6: Pulling fs layer 2024-06-26T06:35:18.0830601Z a863a1c01885: Pulling fs layer 2024-06-26T06:35:18.0831043Z 9b36f9d4ace6: Waiting 2024-06-26T06:35:18.0831436Z b9b8edd214be: Pulling fs layer 2024-06-26T06:35:18.0831885Z 4b2a3c41d6ae: Waiting 2024-06-26T06:35:18.0832274Z a863a1c01885: Waiting 2024-06-26T06:35:18.0832638Z b2689866a0b9: Waiting 2024-06-26T06:35:18.0833024Z b9b8edd214be: Waiting 2024-06-26T06:35:18.2423321Z 09d415c238d7: Verifying Checksum 2024-06-26T06:35:18.2423846Z 09d415c238d7: Download complete 2024-06-26T06:35:18.3117803Z 41f16248e682: Verifying Checksum 2024-06-26T06:35:18.3118339Z 41f16248e682: Download complete 2024-06-26T06:35:18.4003854Z 95d7b7817039: Download complete 2024-06-26T06:35:18.4426053Z 7a2c55901189: Download complete 2024-06-26T06:35:18.5449497Z ab17245097e4: Download complete 2024-06-26T06:35:18.6257634Z dfecd7e9912b: Verifying Checksum 2024-06-26T06:35:18.6258214Z dfecd7e9912b: Download complete 2024-06-26T06:35:18.7081239Z 464a8f745445: Verifying Checksum 2024-06-26T06:35:18.7081822Z 464a8f745445: Download complete 2024-06-26T06:35:18.7755512Z 9fe6e2e61518: Verifying Checksum 2024-06-26T06:35:18.7756067Z 9fe6e2e61518: Download complete 2024-06-26T06:35:18.8603909Z 0723abde19c0: Download complete 2024-06-26T06:35:18.9408209Z bb93d41d4cb4: Verifying Checksum 2024-06-26T06:35:18.9408779Z bb93d41d4cb4: Download complete 2024-06-26T06:35:19.7303070Z 7a2c55901189: Pull complete 2024-06-26T06:35:20.0819395Z 09d415c238d7: Pull complete 2024-06-26T06:35:21.2528782Z 9fe6e2e61518: Pull complete 2024-06-26T06:35:21.3171693Z 41f16248e682: Pull complete 2024-06-26T06:35:21.3654677Z 95d7b7817039: Pull complete 2024-06-26T06:35:25.5766341Z 4b89839dd066: Verifying Checksum 2024-06-26T06:35:25.5767176Z 4b89839dd066: Download complete 2024-06-26T06:35:25.6656387Z caa9986c765b: Download complete 2024-06-26T06:35:25.7413333Z e17c09bdf243: Download complete 2024-06-26T06:35:25.8283328Z bd2e32de75c9: Verifying Checksum 2024-06-26T06:35:25.8283902Z bd2e32de75c9: Download complete 2024-06-26T06:35:27.1207521Z 559a1daa0f98: Verifying Checksum 2024-06-26T06:35:27.1208522Z 559a1daa0f98: Download complete 2024-06-26T06:35:27.2073886Z d96cec7a425c: Verifying Checksum 2024-06-26T06:35:27.2075166Z d96cec7a425c: Download complete 2024-06-26T06:35:27.2813971Z 1db6fc6c2590: Download complete 2024-06-26T06:35:27.3964235Z ac522bc70345: Verifying Checksum 2024-06-26T06:35:27.3965037Z ac522bc70345: Download complete 2024-06-26T06:35:32.3413660Z 8f6c90485347: Verifying Checksum 2024-06-26T06:35:32.3414253Z 8f6c90485347: Download complete 2024-06-26T06:35:32.4126216Z 44f27f91d8fa: Verifying Checksum 2024-06-26T06:35:32.4127088Z 44f27f91d8fa: Download complete 2024-06-26T06:35:32.5137033Z 0a0ded106167: Verifying Checksum 2024-06-26T06:35:32.5137559Z 0a0ded106167: Download complete 2024-06-26T06:35:32.5932456Z 0c62bfb80b7c: Verifying Checksum 2024-06-26T06:35:32.5932958Z 0c62bfb80b7c: Download complete 2024-06-26T06:35:32.7566420Z 0c8d1e7f3282: Verifying Checksum 2024-06-26T06:35:32.7567479Z 0c8d1e7f3282: Download complete 2024-06-26T06:35:32.8900412Z 846366e78885: Download complete 2024-06-26T06:35:34.2465394Z efa11c713d18: Verifying Checksum 2024-06-26T06:35:34.2466157Z efa11c713d18: Download complete 2024-06-26T06:35:34.3475476Z 32f0221befeb: Verifying Checksum 2024-06-26T06:35:34.3476361Z 32f0221befeb: Download complete 2024-06-26T06:35:34.4418291Z a1d57c94ba33: Verifying Checksum 2024-06-26T06:35:34.4419125Z a1d57c94ba33: Download complete 2024-06-26T06:35:34.5189633Z 3dc15441e6d0: Verifying Checksum 2024-06-26T06:35:34.5190690Z 3dc15441e6d0: Download complete 2024-06-26T06:35:34.6008798Z 668eadf990e8: Verifying Checksum 2024-06-26T06:35:34.6009745Z 668eadf990e8: Download complete 2024-06-26T06:35:34.6755325Z fd5a832fe26e: Verifying Checksum 2024-06-26T06:35:34.6756459Z fd5a832fe26e: Download complete 2024-06-26T06:35:39.4250516Z b581c3782fd6: Verifying Checksum 2024-06-26T06:35:39.4251463Z b581c3782fd6: Download complete 2024-06-26T06:35:39.5292319Z 3f42611c5636: Verifying Checksum 2024-06-26T06:35:39.5293521Z 3f42611c5636: Download complete 2024-06-26T06:35:39.5940368Z fe8dd930eda1: Download complete 2024-06-26T06:35:40.0168766Z 6a9711a8a50d: Verifying Checksum 2024-06-26T06:35:40.0169490Z 6a9711a8a50d: Download complete 2024-06-26T06:35:40.1153717Z 76cd931663c9: Verifying Checksum 2024-06-26T06:35:40.1154488Z 76cd931663c9: Download complete 2024-06-26T06:35:40.1989048Z c9ec95615433: Verifying Checksum 2024-06-26T06:35:40.1989991Z c9ec95615433: Download complete 2024-06-26T06:35:40.5710579Z 7da33ab03193: Verifying Checksum 2024-06-26T06:35:40.5711054Z 7da33ab03193: Download complete 2024-06-26T06:35:40.6602537Z da956bbe6180: Verifying Checksum 2024-06-26T06:35:40.6603551Z da956bbe6180: Download complete 2024-06-26T06:35:40.7595532Z 73e6b5984544: Verifying Checksum 2024-06-26T06:35:40.7596219Z 73e6b5984544: Download complete 2024-06-26T06:35:40.8409498Z e5a6ec0a41d7: Download complete 2024-06-26T06:35:40.8515767Z 4f4fb700ef54: Download complete 2024-06-26T06:35:40.9379339Z 4b25a744b402: Verifying Checksum 2024-06-26T06:35:40.9379898Z 4b25a744b402: Download complete 2024-06-26T06:35:41.0582926Z 21f74857bd17: Verifying Checksum 2024-06-26T06:35:41.0583870Z 21f74857bd17: Download complete 2024-06-26T06:35:41.5822311Z 6c12f547d202: Verifying Checksum 2024-06-26T06:35:41.5822755Z 6c12f547d202: Download complete 2024-06-26T06:35:41.6660685Z ed4e5a4f8526: Download complete 2024-06-26T06:35:41.7633855Z 6f09f741ead6: Verifying Checksum 2024-06-26T06:35:41.7634521Z 6f09f741ead6: Download complete 2024-06-26T06:35:41.8442103Z 411fce2db725: Verifying Checksum 2024-06-26T06:35:41.8442694Z 411fce2db725: Download complete 2024-06-26T06:35:41.9177321Z 87f7902d12a4: Verifying Checksum 2024-06-26T06:35:41.9178045Z 87f7902d12a4: Download complete 2024-06-26T06:35:43.2563710Z d67c111fa588: Verifying Checksum 2024-06-26T06:35:43.2564852Z d67c111fa588: Download complete 2024-06-26T06:35:43.3435733Z 9755a952d3ec: Verifying Checksum 2024-06-26T06:35:43.3436924Z 9755a952d3ec: Download complete 2024-06-26T06:35:43.4208386Z 5ca5d2e8bebe: Verifying Checksum 2024-06-26T06:35:43.4208989Z 5ca5d2e8bebe: Download complete 2024-06-26T06:35:43.5046901Z d07af843db47: Verifying Checksum 2024-06-26T06:35:43.5047952Z d07af843db47: Download complete 2024-06-26T06:35:43.6014604Z dbbd7a4ca846: Verifying Checksum 2024-06-26T06:35:43.6015707Z dbbd7a4ca846: Download complete 2024-06-26T06:35:43.6810772Z 6a18c61a96dd: Verifying Checksum 2024-06-26T06:35:43.6811459Z 6a18c61a96dd: Download complete 2024-06-26T06:35:43.8439001Z bd12222c0237: Verifying Checksum 2024-06-26T06:35:43.8440158Z bd12222c0237: Download complete 2024-06-26T06:35:43.9238119Z a5dfe6ba36fd: Verifying Checksum 2024-06-26T06:35:43.9239248Z a5dfe6ba36fd: Download complete 2024-06-26T06:35:43.9915634Z 4b990eb76e8f: Download complete 2024-06-26T06:35:44.0783154Z d9df85e8d422: Download complete 2024-06-26T06:35:44.1608769Z 47d89094b1a7: Verifying Checksum 2024-06-26T06:35:44.1609332Z 47d89094b1a7: Download complete 2024-06-26T06:35:44.3254870Z 94beee2c356a: Verifying Checksum 2024-06-26T06:35:44.3255375Z 94beee2c356a: Download complete 2024-06-26T06:35:44.3911486Z 322c644984f4: Verifying Checksum 2024-06-26T06:35:45.0131191Z b43ee04d974d: Verifying Checksum 2024-06-26T06:35:45.0132063Z b43ee04d974d: Download complete 2024-06-26T06:35:45.0923091Z 40f95cbeec2c: Verifying Checksum 2024-06-26T06:35:45.0924187Z 40f95cbeec2c: Download complete 2024-06-26T06:35:50.8978295Z 8f6c90485347: Pull complete 2024-06-26T06:35:50.9488192Z ab17245097e4: Pull complete 2024-06-26T06:35:51.0024950Z dfecd7e9912b: Pull complete 2024-06-26T06:35:51.0617368Z 464a8f745445: Pull complete 2024-06-26T06:35:55.6566223Z a05e0234ee85: Verifying Checksum 2024-06-26T06:35:55.6566785Z a05e0234ee85: Download complete 2024-06-26T06:35:55.7470360Z b2689866a0b9: Download complete 2024-06-26T06:35:55.8234995Z 9b36f9d4ace6: Verifying Checksum 2024-06-26T06:35:55.8235687Z 9b36f9d4ace6: Download complete 2024-06-26T06:35:56.3977436Z a863a1c01885: Download complete 2024-06-26T06:35:56.4810475Z b9b8edd214be: Verifying Checksum 2024-06-26T06:35:56.4811196Z b9b8edd214be: Download complete 2024-06-26T06:35:58.9190469Z 4b2a3c41d6ae: Verifying Checksum 2024-06-26T06:35:58.9191148Z 4b2a3c41d6ae: Download complete 2024-06-26T06:36:01.0867444Z 9777c07909ad: Verifying Checksum 2024-06-26T06:36:01.0867998Z 9777c07909ad: Download complete 2024-06-26T06:36:29.5891833Z d67c111fa588: Pull complete 2024-06-26T06:36:32.3100590Z 0723abde19c0: Pull complete 2024-06-26T06:36:34.7843265Z bb93d41d4cb4: Pull complete 2024-06-26T06:36:51.3687176Z 4b89839dd066: Pull complete 2024-06-26T06:36:53.8057513Z caa9986c765b: Pull complete 2024-06-26T06:36:56.3109146Z e17c09bdf243: Pull complete 2024-06-26T06:36:58.7829691Z bd2e32de75c9: Pull complete 2024-06-26T06:37:04.7118287Z 559a1daa0f98: Pull complete 2024-06-26T06:37:07.1200212Z d96cec7a425c: Pull complete 2024-06-26T06:37:09.8677950Z 1db6fc6c2590: Pull complete 2024-06-26T06:37:12.3112682Z ac522bc70345: Pull complete 2024-06-26T06:38:18.9866862Z a05e0234ee85: Pull complete 2024-06-26T06:38:22.6327339Z 44f27f91d8fa: Pull complete 2024-06-26T06:38:26.8908930Z 0a0ded106167: Pull complete 2024-06-26T06:38:29.2061806Z 0c62bfb80b7c: Pull complete 2024-06-26T06:38:29.2529276Z 0c8d1e7f3282: Pull complete 2024-06-26T06:38:29.2908038Z 846366e78885: Pull complete 2024-06-26T06:38:33.5142525Z efa11c713d18: Pull complete 2024-06-26T06:38:33.5530152Z 32f0221befeb: Pull complete 2024-06-26T06:38:33.5981362Z a1d57c94ba33: Pull complete 2024-06-26T06:38:33.6347730Z 3dc15441e6d0: Pull complete 2024-06-26T06:38:33.6702807Z 668eadf990e8: Pull complete 2024-06-26T06:38:33.7195604Z fd5a832fe26e: Pull complete 2024-06-26T06:38:46.1691352Z b581c3782fd6: Pull complete 2024-06-26T06:38:49.2797977Z 3f42611c5636: Pull complete 2024-06-26T06:38:51.1505334Z fe8dd930eda1: Pull complete 2024-06-26T06:38:52.1033624Z 6a9711a8a50d: Pull complete 2024-06-26T06:38:52.1377159Z 76cd931663c9: Pull complete 2024-06-26T06:38:52.1825401Z c9ec95615433: Pull complete 2024-06-26T06:38:52.5511516Z 7da33ab03193: Pull complete 2024-06-26T06:38:52.6030312Z da956bbe6180: Pull complete 2024-06-26T06:38:52.7023848Z 73e6b5984544: Pull complete 2024-06-26T06:38:52.7382405Z e5a6ec0a41d7: Pull complete 2024-06-26T06:38:52.7868762Z 4f4fb700ef54: Pull complete 2024-06-26T06:38:52.8361773Z 4b25a744b402: Pull complete 2024-06-26T06:38:52.8757184Z 21f74857bd17: Pull complete 2024-06-26T06:38:55.3611715Z 6c12f547d202: Pull complete 2024-06-26T06:38:55.4127116Z ed4e5a4f8526: Pull complete 2024-06-26T06:38:55.4457469Z 6f09f741ead6: Pull complete 2024-06-26T06:38:55.5538384Z 411fce2db725: Pull complete 2024-06-26T06:38:55.6029137Z 87f7902d12a4: Pull complete 2024-06-26T06:39:40.4474791Z 9777c07909ad: Pull complete 2024-06-26T06:39:42.8842684Z 9755a952d3ec: Pull complete 2024-06-26T06:39:45.3252019Z 5ca5d2e8bebe: Pull complete 2024-06-26T06:39:50.2281373Z d07af843db47: Pull complete 2024-06-26T06:39:55.0792907Z dbbd7a4ca846: Pull complete 2024-06-26T06:39:57.3329202Z 6a18c61a96dd: Pull complete 2024-06-26T06:40:00.6357641Z bd12222c0237: Pull complete 2024-06-26T06:40:03.1704356Z a5dfe6ba36fd: Pull complete 2024-06-26T06:40:06.0265717Z 4b990eb76e8f: Pull complete 2024-06-26T06:40:08.7914202Z d9df85e8d422: Pull complete 2024-06-26T06:40:09.1276548Z 47d89094b1a7: Pull complete 2024-06-26T06:40:10.3343478Z 94beee2c356a: Pull complete 2024-06-26T06:40:10.3944527Z 322c644984f4: Pull complete 2024-06-26T06:40:12.6944492Z b43ee04d974d: Pull complete 2024-06-26T06:40:12.7343037Z 40f95cbeec2c: Pull complete 2024-06-26T06:40:29.5509442Z 4b2a3c41d6ae: Pull complete 2024-06-26T06:40:29.6059963Z b2689866a0b9: Pull complete 2024-06-26T06:40:29.6554034Z 9b36f9d4ace6: Pull complete 2024-06-26T06:40:30.3293810Z a863a1c01885: Pull complete 2024-06-26T06:40:30.3792234Z b9b8edd214be: Pull complete 2024-06-26T06:40:30.6530106Z Digest: sha256:493391c3b293fbc39c8ca4208a5b70ab17031329cb9c855a641ab5a208c52fe6 2024-06-26T06:40:30.6564390Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:40:30.6600184Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:40:30.6659848Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T06:40:30.6661062Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T06:40:30.6670243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:40:30.6670883Z env: 2024-06-26T06:40:30.6671222Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:40:30.6671662Z ##[endgroup] 2024-06-26T06:40:30.6878146Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T06:40:30.6878855Z with: 2024-06-26T06:40:30.6879213Z driver-version: 550.54.15 2024-06-26T06:40:30.6879642Z env: 2024-06-26T06:40:30.6879961Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:40:30.6880394Z ##[endgroup] 2024-06-26T06:40:30.6935705Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:40:30.6936439Z with: 2024-06-26T06:40:30.6936774Z timeout_minutes: 10 2024-06-26T06:40:30.6937196Z max_attempts: 3 2024-06-26T06:40:30.6979167Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T06:40:30.7021851Z retry_wait_seconds: 10 2024-06-26T06:40:30.7022321Z polling_interval_seconds: 1 2024-06-26T06:40:30.7022781Z warning_on_retry: true 2024-06-26T06:40:30.7023230Z continue_on_error: false 2024-06-26T06:40:30.7023657Z env: 2024-06-26T06:40:30.7023986Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:40:30.7024451Z DRIVER_VERSION: 550.54.15 2024-06-26T06:40:30.7024905Z ##[endgroup] 2024-06-26T06:40:30.7816354Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T06:40:30.7818553Z + pre_install_nvidia_driver_amzn2 2024-06-26T06:40:30.7819316Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T06:40:31.2940140Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:40:31.3513100Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T06:40:31.3891688Z No Packages marked for removal 2024-06-26T06:40:31.4048975Z + install_nvidia_driver_common 2024-06-26T06:40:31.4052787Z + echo 'Before installing NVIDIA driver' 2024-06-26T06:40:31.4053511Z + lspci 2024-06-26T06:40:31.4055745Z Before installing NVIDIA driver 2024-06-26T06:40:31.4273126Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-06-26T06:40:31.4275058Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T06:40:31.4276900Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-06-26T06:40:31.4278472Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-06-26T06:40:31.4279280Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-06-26T06:40:31.4280189Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T06:40:31.4281210Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:40:31.4282278Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:40:31.4283335Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-06-26T06:40:31.4284083Z + lsmod 2024-06-26T06:40:31.4296575Z Module Size Used by 2024-06-26T06:40:31.4297136Z ib_core 266240 0 2024-06-26T06:40:31.4297767Z nvidia_modeset 1339392 0 2024-06-26T06:40:31.4298774Z veth 16384 0 2024-06-26T06:40:31.4299542Z nvidia_uvm 4599808 0 2024-06-26T06:40:31.4300341Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T06:40:31.4300950Z drm 421888 1 nvidia 2024-06-26T06:40:31.4301492Z i2c_core 77824 2 nvidia,drm 2024-06-26T06:40:31.4302053Z backlight 16384 1 nvidia_modeset 2024-06-26T06:40:31.4302617Z xt_conntrack 16384 1 2024-06-26T06:40:31.4303419Z ipt_MASQUERADE 16384 1 2024-06-26T06:40:31.4304169Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T06:40:31.4304737Z nf_conntrack_netlink 49152 0 2024-06-26T06:40:31.4305291Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T06:40:31.4305845Z xfrm_user 45056 1 2024-06-26T06:40:31.4306330Z xfrm_algo 16384 1 xfrm_user 2024-06-26T06:40:31.4306843Z iptable_nat 16384 1 2024-06-26T06:40:31.4307295Z nf_conntrack_ipv4 16384 3 2024-06-26T06:40:31.4307827Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T06:40:31.4308436Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T06:40:31.4309078Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T06:40:31.4310473Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T06:40:31.4312031Z xt_addrtype 16384 2 2024-06-26T06:40:31.4312494Z iptable_filter 16384 1 2024-06-26T06:40:31.4312965Z br_netfilter 24576 0 2024-06-26T06:40:31.4313476Z bridge 172032 1 br_netfilter 2024-06-26T06:40:31.4314006Z stp 16384 1 bridge 2024-06-26T06:40:31.4314528Z llc 16384 2 bridge,stp 2024-06-26T06:40:31.4315048Z overlay 86016 0 2024-06-26T06:40:31.4315490Z sunrpc 393216 1 2024-06-26T06:40:31.4315945Z dm_mirror 28672 0 2024-06-26T06:40:31.4317146Z dm_region_hash 20480 1 dm_mirror 2024-06-26T06:40:31.4318364Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T06:40:31.4319663Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T06:40:31.4320734Z dax 69632 1 dm_mod 2024-06-26T06:40:31.4321686Z sb_edac 24576 0 2024-06-26T06:40:31.4322358Z crc32_pclmul 16384 0 2024-06-26T06:40:31.4322834Z ghash_clmulni_intel 16384 0 2024-06-26T06:40:31.4323306Z pcbc 16384 0 2024-06-26T06:40:31.4323743Z aesni_intel 188416 0 2024-06-26T06:40:31.4324234Z aes_x86_64 20480 1 aesni_intel 2024-06-26T06:40:31.4324797Z crypto_simd 16384 1 aesni_intel 2024-06-26T06:40:31.4325336Z glue_helper 16384 1 aesni_intel 2024-06-26T06:40:31.4326021Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T06:40:31.4326696Z mousedev 24576 0 2024-06-26T06:40:31.4327135Z evdev 20480 3 2024-06-26T06:40:31.4327591Z psmouse 32768 0 2024-06-26T06:40:31.4328069Z pcc_cpufreq 16384 0 2024-06-26T06:40:31.4328519Z button 16384 0 2024-06-26T06:40:31.4328967Z ena 139264 0 2024-06-26T06:40:31.4329423Z ptp 20480 1 ena 2024-06-26T06:40:31.4329892Z pps_core 20480 1 ptp 2024-06-26T06:40:31.4330377Z ata_piix 36864 0 2024-06-26T06:40:31.4331017Z libata 266240 1 ata_piix 2024-06-26T06:40:31.4331514Z xen_blkfront 49152 2 2024-06-26T06:40:31.4331976Z crc32c_intel 24576 0 2024-06-26T06:40:31.4332563Z scsi_mod 245760 1 libata 2024-06-26T06:40:31.4333043Z autofs4 49152 2 2024-06-26T06:40:31.4333492Z + modinfo nvidia 2024-06-26T06:40:31.4334332Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T06:40:31.4335195Z alias: char-major-195-* 2024-06-26T06:40:31.4335670Z version: 550.54.15 2024-06-26T06:40:31.4336095Z supported: external 2024-06-26T06:40:31.4336707Z license: NVIDIA 2024-06-26T06:40:31.4337321Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T06:40:31.4337924Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T06:40:31.4338476Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T06:40:31.4339058Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T06:40:31.4339680Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T06:40:31.4340275Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T06:40:31.4340890Z depends: i2c-core,drm 2024-06-26T06:40:31.4341339Z retpoline: Y 2024-06-26T06:40:31.4341697Z name: nvidia 2024-06-26T06:40:31.4342393Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T06:40:31.4343216Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T06:40:31.4344025Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T06:40:31.4344756Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T06:40:31.4345319Z parm: NVreg_RmLogonRC:int 2024-06-26T06:40:31.4345870Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T06:40:31.4346422Z parm: NVreg_DeviceFileUID:int 2024-06-26T06:40:31.4346964Z parm: NVreg_DeviceFileGID:int 2024-06-26T06:40:31.4347509Z parm: NVreg_DeviceFileMode:int 2024-06-26T06:40:31.4348148Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T06:40:31.4348856Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T06:40:31.4349453Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T06:40:31.4349977Z parm: NVreg_EnableMSI:int 2024-06-26T06:40:31.4350492Z parm: NVreg_TCEBypassMode:int 2024-06-26T06:40:31.4351062Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T06:40:31.4351715Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T06:40:31.4352448Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T06:40:31.4353157Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T06:40:31.4353919Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T06:40:31.4354671Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T06:40:31.4355441Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T06:40:31.4356177Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T06:40:31.4357026Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T06:40:31.4357697Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T06:40:31.4358363Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T06:40:31.4358987Z parm: NVreg_MemoryPoolSize:int 2024-06-26T06:40:31.4359563Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T06:40:31.4360147Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T06:40:31.4360739Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T06:40:31.4361296Z parm: NVreg_NvLinkDisable:int 2024-06-26T06:40:31.4361918Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T06:40:31.4362587Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T06:40:31.4363183Z parm: NVreg_EnableResizableBar:int 2024-06-26T06:40:31.4363775Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T06:40:31.4364402Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T06:40:31.4365008Z parm: NVreg_RegistryDwords:charp 2024-06-26T06:40:31.4365630Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T06:40:31.4366212Z parm: NVreg_RmMsg:charp 2024-06-26T06:40:31.4366720Z parm: NVreg_GpuBlacklist:charp 2024-06-26T06:40:31.4367307Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T06:40:31.4367876Z parm: NVreg_ExcludedGpus:charp 2024-06-26T06:40:31.4368439Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T06:40:31.4369035Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T06:40:31.4369608Z parm: NVreg_ImexChannelCount:int 2024-06-26T06:40:31.4370173Z parm: rm_firmware_active:charp 2024-06-26T06:40:31.4370984Z + HAS_NVIDIA_DRIVER=0 2024-06-26T06:40:31.4371458Z ++ command -v nvidia-smi 2024-06-26T06:40:31.4371945Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T06:40:31.4372398Z + set +e 2024-06-26T06:40:31.4373019Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T06:40:33.5862411Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T06:40:33.5863024Z + NVIDIA_SMI_STATUS=0 2024-06-26T06:40:33.5863573Z + '[' 0 -ne 0 ']' 2024-06-26T06:40:33.5863996Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T06:40:33.5864476Z + HAS_NVIDIA_DRIVER=1 2024-06-26T06:40:33.5865384Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T06:40:33.5866264Z + set -e 2024-06-26T06:40:33.5866638Z + '[' 1 -eq 0 ']' 2024-06-26T06:40:33.5867362Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T06:40:33.5868216Z + post_install_nvidia_driver_common 2024-06-26T06:40:33.5868749Z + sudo modprobe nvidia 2024-06-26T06:40:33.6012095Z + echo 'After installing NVIDIA driver' 2024-06-26T06:40:33.6012623Z + lspci 2024-06-26T06:40:33.6014258Z After installing NVIDIA driver 2024-06-26T06:40:33.6227664Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 2024-06-26T06:40:33.6229032Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T06:40:33.6230011Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] 2024-06-26T06:40:33.6230956Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01) 2024-06-26T06:40:33.6231771Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446 2024-06-26T06:40:33.6232617Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T06:40:33.6233654Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:40:33.6234723Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1) 2024-06-26T06:40:33.6235787Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01) 2024-06-26T06:40:33.6236809Z + lsmod 2024-06-26T06:40:33.6250184Z Module Size Used by 2024-06-26T06:40:33.6250708Z ib_core 266240 0 2024-06-26T06:40:33.6251635Z nvidia_modeset 1339392 0 2024-06-26T06:40:33.6252187Z veth 16384 0 2024-06-26T06:40:33.6252654Z nvidia_uvm 4599808 0 2024-06-26T06:40:33.6253195Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T06:40:33.6253800Z drm 421888 1 nvidia 2024-06-26T06:40:33.6254319Z i2c_core 77824 2 nvidia,drm 2024-06-26T06:40:33.6254886Z backlight 16384 1 nvidia_modeset 2024-06-26T06:40:33.6255440Z xt_conntrack 16384 1 2024-06-26T06:40:33.6255920Z ipt_MASQUERADE 16384 1 2024-06-26T06:40:33.6256449Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T06:40:33.6257046Z nf_conntrack_netlink 49152 0 2024-06-26T06:40:33.6257874Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T06:40:33.6258452Z xfrm_user 45056 1 2024-06-26T06:40:33.6258941Z xfrm_algo 16384 1 xfrm_user 2024-06-26T06:40:33.6259459Z iptable_nat 16384 1 2024-06-26T06:40:33.6259910Z nf_conntrack_ipv4 16384 3 2024-06-26T06:40:33.6260447Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T06:40:33.6261047Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T06:40:33.6261685Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T06:40:33.6263761Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T06:40:33.6265094Z xt_addrtype 16384 2 2024-06-26T06:40:33.6265571Z iptable_filter 16384 1 2024-06-26T06:40:33.6266032Z br_netfilter 24576 0 2024-06-26T06:40:33.6266539Z bridge 172032 1 br_netfilter 2024-06-26T06:40:33.6267336Z stp 16384 1 bridge 2024-06-26T06:40:33.6268112Z llc 16384 2 bridge,stp 2024-06-26T06:40:33.6268617Z overlay 86016 0 2024-06-26T06:40:33.6269074Z sunrpc 393216 1 2024-06-26T06:40:33.6269536Z dm_mirror 28672 0 2024-06-26T06:40:33.6270006Z dm_region_hash 20480 1 dm_mirror 2024-06-26T06:40:33.6270612Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T06:40:33.6271248Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T06:40:33.6271787Z dax 69632 1 dm_mod 2024-06-26T06:40:33.6272280Z sb_edac 24576 0 2024-06-26T06:40:33.6272742Z crc32_pclmul 16384 0 2024-06-26T06:40:33.6273192Z ghash_clmulni_intel 16384 0 2024-06-26T06:40:33.6273662Z pcbc 16384 0 2024-06-26T06:40:33.6274119Z aesni_intel 188416 0 2024-06-26T06:40:33.6274588Z aes_x86_64 20480 1 aesni_intel 2024-06-26T06:40:33.6275151Z crypto_simd 16384 1 aesni_intel 2024-06-26T06:40:33.6275721Z glue_helper 16384 1 aesni_intel 2024-06-26T06:40:33.6276848Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T06:40:33.6277551Z mousedev 24576 0 2024-06-26T06:40:33.6278007Z evdev 20480 3 2024-06-26T06:40:33.6278555Z psmouse 32768 0 2024-06-26T06:40:33.6279031Z pcc_cpufreq 16384 0 2024-06-26T06:40:33.6279492Z button 16384 0 2024-06-26T06:40:33.6279923Z ena 139264 0 2024-06-26T06:40:33.6280382Z ptp 20480 1 ena 2024-06-26T06:40:33.6280867Z pps_core 20480 1 ptp 2024-06-26T06:40:33.6281333Z ata_piix 36864 0 2024-06-26T06:40:33.6281814Z libata 266240 1 ata_piix 2024-06-26T06:40:33.6282322Z xen_blkfront 49152 2 2024-06-26T06:40:33.6282767Z crc32c_intel 24576 0 2024-06-26T06:40:33.6283239Z scsi_mod 245760 1 libata 2024-06-26T06:40:33.6283741Z autofs4 49152 2 2024-06-26T06:40:33.6284187Z + modinfo nvidia 2024-06-26T06:40:33.6285017Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T06:40:33.6285901Z alias: char-major-195-* 2024-06-26T06:40:33.6286360Z version: 550.54.15 2024-06-26T06:40:33.6286793Z supported: external 2024-06-26T06:40:33.6287225Z license: NVIDIA 2024-06-26T06:40:33.6287676Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T06:40:33.6288282Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T06:40:33.6288913Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T06:40:33.6289540Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T06:40:33.6290178Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T06:40:33.6290776Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T06:40:33.6291399Z depends: i2c-core,drm 2024-06-26T06:40:33.6291851Z retpoline: Y 2024-06-26T06:40:33.6292222Z name: nvidia 2024-06-26T06:40:33.6292924Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T06:40:33.6293751Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T06:40:33.6294546Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T06:40:33.6295298Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T06:40:33.6295853Z parm: NVreg_RmLogonRC:int 2024-06-26T06:40:33.6296376Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T06:40:33.6296944Z parm: NVreg_DeviceFileUID:int 2024-06-26T06:40:33.6297491Z parm: NVreg_DeviceFileGID:int 2024-06-26T06:40:33.6298019Z parm: NVreg_DeviceFileMode:int 2024-06-26T06:40:33.6298680Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T06:40:33.6299389Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T06:40:33.6299969Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T06:40:33.6300691Z parm: NVreg_EnableMSI:int 2024-06-26T06:40:33.6301357Z parm: NVreg_TCEBypassMode:int 2024-06-26T06:40:33.6301910Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T06:40:33.6302576Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T06:40:33.6303308Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T06:40:33.6303997Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T06:40:33.6304763Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T06:40:33.6305526Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T06:40:33.6306281Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T06:40:33.6307035Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T06:40:33.6307652Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T06:40:33.6308308Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T06:40:33.6308997Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T06:40:33.6309635Z parm: NVreg_MemoryPoolSize:int 2024-06-26T06:40:33.6310208Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T06:40:33.6310817Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T06:40:33.6311408Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T06:40:33.6311969Z parm: NVreg_NvLinkDisable:int 2024-06-26T06:40:33.6312588Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T06:40:33.6313253Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T06:40:33.6313852Z parm: NVreg_EnableResizableBar:int 2024-06-26T06:40:33.6314458Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T06:40:33.6315091Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T06:40:33.6315703Z parm: NVreg_RegistryDwords:charp 2024-06-26T06:40:33.6316509Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T06:40:33.6317133Z parm: NVreg_RmMsg:charp 2024-06-26T06:40:33.6317646Z parm: NVreg_GpuBlacklist:charp 2024-06-26T06:40:33.6318222Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T06:40:33.6318822Z parm: NVreg_ExcludedGpus:charp 2024-06-26T06:40:33.6319434Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T06:40:33.6320018Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T06:40:33.6320611Z parm: NVreg_ImexChannelCount:int 2024-06-26T06:40:33.6321182Z parm: rm_firmware_active:charp 2024-06-26T06:40:33.6321666Z + set +e 2024-06-26T06:40:33.6322049Z + nvidia-smi 2024-06-26T06:40:36.5318388Z Wed Jun 26 06:40:36 2024 2024-06-26T06:40:36.5319264Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:40:36.5320291Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T06:40:36.5321333Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:40:36.5322377Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T06:40:36.5323303Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T06:40:36.5324172Z | | | MIG M. | 2024-06-26T06:40:36.5325021Z |=========================================+========================+======================| 2024-06-26T06:40:36.5375330Z | 0 Tesla M60 Off | 00000000:00:1D.0 Off | 0 | 2024-06-26T06:40:36.5376310Z | N/A 35C P0 37W / 150W | 0MiB / 7680MiB | 99% Default | 2024-06-26T06:40:36.5376868Z | | | N/A | 2024-06-26T06:40:36.5377718Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:40:36.5422724Z | 1 Tesla M60 Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T06:40:36.5423871Z | N/A 38C P0 34W / 150W | 0MiB / 7680MiB | 0% Default | 2024-06-26T06:40:36.5424634Z | | | N/A | 2024-06-26T06:40:36.5425492Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T06:40:36.5426220Z 2024-06-26T06:40:36.5427202Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:40:36.5427929Z | Processes: | 2024-06-26T06:40:36.5428926Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T06:40:36.5429576Z | ID ID Usage | 2024-06-26T06:40:36.5430459Z |=========================================================================================| 2024-06-26T06:40:36.5431099Z | No running processes found | 2024-06-26T06:40:36.5432058Z +-----------------------------------------------------------------------------------------+ 2024-06-26T06:40:37.0862913Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T06:40:38.6193691Z Tesla M60 2024-06-26T06:40:39.5285679Z + NVIDIA_SMI_STATUS=0 2024-06-26T06:40:39.5286508Z + '[' 0 -eq 0 ']' 2024-06-26T06:40:39.5286795Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T06:40:39.5287299Z + set -e 2024-06-26T06:40:39.5287671Z INFO: Ignoring allowed status 0 2024-06-26T06:40:39.5292557Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T06:40:39.5296398Z + sudo yum install -y yum-utils 2024-06-26T06:40:40.0387615Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:40:41.7695241Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T06:40:41.7696175Z Nothing to do 2024-06-26T06:40:41.9170274Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T06:40:41.9171357Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:40:41.9172714Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:40:42.4739565Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:40:42.5041558Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T06:40:42.5043080Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T06:40:42.5044312Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T06:40:42.5187692Z + sudo yum install -y nvidia-docker2 2024-06-26T06:40:43.0253079Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T06:40:44.6394445Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T06:40:44.6395355Z Nothing to do 2024-06-26T06:40:44.7855480Z + sudo systemctl restart docker 2024-06-26T06:41:03.8137567Z Command completed after 1 attempt(s). 2024-06-26T06:41:03.8223144Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T06:41:03.8224237Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T06:41:03.8225629Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T06:41:03.8226469Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T06:41:03.8235882Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:03.8237090Z env: 2024-06-26T06:41:03.8237403Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:03.8237980Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:03.8238618Z ##[endgroup] 2024-06-26T06:41:04.1385351Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T06:41:04.1599810Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T06:41:04.1729164Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T06:41:04.2921946Z Prepare all required actions 2024-06-26T06:41:04.2922625Z Getting action download info 2024-06-26T06:41:04.4264344Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T06:41:04.6204848Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T06:41:04.7559158Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T06:41:04.7559772Z with: 2024-06-26T06:41:04.7560154Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:41:04.7560702Z s3-bucket: gha-artifacts 2024-06-26T06:41:04.7561174Z env: 2024-06-26T06:41:04.7561450Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:04.7562022Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:04.7562655Z ##[endgroup] 2024-06-26T06:41:04.7606861Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T06:41:04.7607405Z with: 2024-06-26T06:41:04.7607862Z name: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:41:04.7608393Z s3-bucket: gha-artifacts 2024-06-26T06:41:04.7608839Z region: us-east-1 2024-06-26T06:41:04.7609206Z env: 2024-06-26T06:41:04.7609524Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:04.7610096Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:04.7610702Z ##[endgroup] 2024-06-26T06:41:05.3750753Z (node:110951) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T06:41:05.3752145Z 2024-06-26T06:41:05.3752620Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T06:41:05.3754549Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T06:41:05.3756871Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T06:41:05.4776710Z Found 1 objects with prefix pytorch/pytorch/9673645538/linux-focal-cuda11.8-py3.10-gcc9/ 2024-06-26T06:41:05.4778097Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T06:41:14.4078114Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T06:41:14.4090258Z Artifact download has finished successfully 2024-06-26T06:41:14.4305104Z ##[group]Run unzip -o artifacts.zip 2024-06-26T06:41:14.4305651Z unzip -o artifacts.zip 2024-06-26T06:41:14.4314294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:14.4314560Z env: 2024-06-26T06:41:14.4314844Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:14.4315390Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:14.4316461Z ##[endgroup] 2024-06-26T06:41:14.4354402Z Archive: artifacts.zip 2024-06-26T06:41:14.4356751Z creating: dist/ 2024-06-26T06:41:16.8810139Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T06:41:16.8810917Z creating: build/custom_test_artifacts/ 2024-06-26T06:41:16.8811591Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T06:41:16.8812511Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T06:41:16.8813547Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:41:16.8820138Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:41:16.8821348Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T06:41:16.8822323Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:41:16.8823761Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:41:16.8825041Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:41:16.8826781Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:41:16.8828354Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:41:16.8829504Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:41:16.8830980Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:41:16.8832456Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:41:16.8833913Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:41:16.8835429Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:41:16.8837191Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:41:16.8838775Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:41:16.8840072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:41:16.8841410Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:41:16.8842695Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:41:16.8898271Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:41:16.8900100Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:41:16.8901975Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:41:16.8903897Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:41:16.8905579Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:41:16.8907374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:41:16.8909180Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:41:16.8910989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:41:16.8912778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:41:16.8954735Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:41:16.9004967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:41:16.9006777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:41:16.9008410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:41:16.9010030Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:41:16.9011609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:41:16.9013014Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:41:16.9014534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:41:16.9016310Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:41:16.9088254Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:41:16.9163828Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:41:16.9165342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:41:16.9166599Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:41:16.9167683Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:41:16.9168877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T06:41:16.9170041Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T06:41:16.9171346Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T06:41:16.9172644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T06:41:16.9174054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T06:41:16.9175386Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T06:41:16.9176763Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T06:41:16.9178153Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T06:41:16.9179516Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T06:41:16.9180895Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T06:41:16.9182257Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T06:41:16.9198076Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T06:41:16.9365470Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T06:41:16.9366754Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T06:41:16.9368164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T06:41:16.9369706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T06:41:16.9371221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T06:41:16.9372412Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T06:41:16.9373861Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T06:41:16.9375316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T06:41:16.9376782Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T06:41:16.9378237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T06:41:16.9379673Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T06:41:16.9397823Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T06:41:16.9503434Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T06:41:16.9505538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:41:16.9507802Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:41:16.9509282Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T06:41:16.9511070Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T06:41:16.9512551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T06:41:16.9513708Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T06:41:16.9514788Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T06:41:16.9515755Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T06:41:16.9517123Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T06:41:16.9652509Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T06:41:16.9731482Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T06:41:16.9732444Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T06:41:16.9733329Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T06:41:16.9734368Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:41:16.9741420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:41:16.9742889Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T06:41:16.9744267Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:41:16.9745781Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:41:16.9747736Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:41:16.9749151Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:41:16.9751073Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:41:16.9752661Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:41:16.9754870Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:41:16.9757098Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:41:16.9759184Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:41:16.9760643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:41:16.9762102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:41:16.9763591Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:41:16.9765064Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:41:16.9766378Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:41:16.9767654Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:41:16.9818257Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:41:16.9821698Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:41:16.9823881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:41:16.9825791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:41:16.9827872Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:41:16.9829769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:41:16.9831608Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:41:16.9833402Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:41:16.9835165Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:41:16.9873485Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:41:16.9924252Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:41:16.9927191Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:41:16.9929426Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:41:16.9931365Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:41:16.9932946Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:41:16.9934521Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:41:16.9936024Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:41:16.9937581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:41:17.0007063Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:41:17.0083607Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:41:17.0086604Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:41:17.0088782Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:41:17.0090748Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:41:17.0091680Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T06:41:17.0092862Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T06:41:17.0094221Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T06:41:17.0095772Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T06:41:17.0097230Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T06:41:17.0098602Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T06:41:17.0100016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T06:41:17.0101439Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T06:41:17.0102851Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T06:41:17.0104271Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T06:41:17.0105670Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T06:41:17.0117054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T06:41:17.0198635Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T06:41:17.0200552Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:41:17.0202461Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:41:17.0203688Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T06:41:17.0205597Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T06:41:17.0207005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T06:41:17.0208148Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T06:41:17.0209208Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T06:41:17.0210187Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T06:41:17.0211207Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T06:41:17.0269198Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T06:41:17.0269953Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T06:41:17.0270909Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T06:41:17.0272024Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T06:41:17.0279299Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T06:41:17.0280713Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T06:41:17.0282286Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T06:41:17.0284104Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T06:41:17.0285995Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T06:41:17.0287713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T06:41:17.0289694Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T06:41:17.0291579Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T06:41:17.0293272Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T06:41:17.0295312Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T06:41:17.0298068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T06:41:17.0299627Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T06:41:17.0301181Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T06:41:17.0302759Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T06:41:17.0304318Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T06:41:17.0305730Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T06:41:17.0307093Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T06:41:17.0357711Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T06:41:17.0361534Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T06:41:17.0364056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T06:41:17.0366096Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T06:41:17.0368073Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T06:41:17.0369929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T06:41:17.0371816Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T06:41:17.0373719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T06:41:17.0375615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T06:41:17.0413031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T06:41:17.0463581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T06:41:17.0467015Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T06:41:17.0469347Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T06:41:17.0471112Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T06:41:17.0472783Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T06:41:17.0474463Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T06:41:17.0476084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T06:41:17.0478035Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T06:41:17.0546279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T06:41:17.0622778Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T06:41:17.0625660Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T06:41:17.0627862Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T06:41:17.0630081Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T06:41:17.0631579Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T06:41:17.0632865Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T06:41:17.0634327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T06:41:17.0636831Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T06:41:17.0638203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T06:41:17.0639654Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T06:41:17.0641401Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T06:41:17.0643058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T06:41:17.0644629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T06:41:17.0646135Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T06:41:17.0647652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T06:41:17.0649272Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T06:41:17.0795426Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T06:41:17.0797963Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T06:41:17.0800589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T06:41:17.0802324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T06:41:17.0803942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T06:41:17.0805490Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T06:41:17.0807097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T06:41:17.0808712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T06:41:17.0810299Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T06:41:17.0811913Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T06:41:17.0813505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T06:41:17.0826688Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T06:41:17.0896979Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T06:41:17.0899213Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T06:41:17.0901239Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T06:41:17.0902566Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T06:41:17.0904432Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T06:41:17.0906081Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T06:41:17.0907324Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T06:41:17.0908466Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T06:41:17.0909532Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T06:41:17.0910625Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T06:41:17.1035003Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T06:41:17.1088803Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T06:41:17.1089622Z creating: build/lib/ 2024-06-26T06:41:17.1101057Z inflating: build/lib/libpthreadpool.a 2024-06-26T06:41:17.1102392Z inflating: build/lib/libclog.a 2024-06-26T06:41:17.1113310Z inflating: build/lib/libcpuinfo.a 2024-06-26T06:41:17.1123764Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T06:41:17.1126611Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T06:41:17.1237958Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T06:41:17.1805145Z inflating: build/lib/libprotobuf.a 2024-06-26T06:41:17.1886201Z inflating: build/lib/libgtest.a 2024-06-26T06:41:17.1979359Z inflating: build/lib/libbenchmark.a 2024-06-26T06:41:17.1988739Z inflating: build/lib/libittnotify.a 2024-06-26T06:41:17.2023352Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T06:41:17.2103130Z inflating: build/lib/libasmjit.a 2024-06-26T06:41:17.2261574Z inflating: build/lib/libgloo.a 2024-06-26T06:41:17.2288873Z inflating: build/lib/libfmt.a 2024-06-26T06:41:17.2291256Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T06:41:17.2405505Z inflating: build/lib/libc10.so 2024-06-26T06:41:17.2406308Z inflating: build/lib/libfoxi_loader.a 2024-06-26T06:41:17.2408379Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T06:41:17.2432374Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T06:41:17.2454953Z inflating: build/lib/libgmock.a 2024-06-26T06:41:17.2456046Z inflating: build/lib/libgtest_main.a 2024-06-26T06:41:17.2457023Z inflating: build/lib/libbenchmark_main.a 2024-06-26T06:41:17.3084072Z inflating: build/lib/libprotoc.a 2024-06-26T06:41:17.3106569Z inflating: build/lib/libnnpack.a 2024-06-26T06:41:18.5733696Z inflating: build/lib/libdnnl.a 2024-06-26T06:41:18.6447141Z inflating: build/lib/libtensorpipe.a 2024-06-26T06:41:18.6514752Z inflating: build/lib/libc10_cuda.so 2024-06-26T06:41:18.6515945Z inflating: build/lib/libgmock_main.a 2024-06-26T06:41:18.8076797Z inflating: build/lib/libfbgemm.a 2024-06-26T06:41:18.8718093Z inflating: build/lib/libkineto.a 2024-06-26T06:41:18.9035687Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T06:41:18.9264922Z inflating: build/lib/libXNNPACK.a 2024-06-26T06:41:18.9749665Z inflating: build/lib/libgloo_cuda.a 2024-06-26T06:41:18.9802531Z inflating: build/lib/libonnx_proto.a 2024-06-26T06:41:19.0663203Z inflating: build/lib/libonnx.a 2024-06-26T06:41:22.0774089Z inflating: build/lib/libtorch_cpu.so 2024-06-26T06:41:22.0778828Z inflating: build/lib/libshm.so 2024-06-26T06:41:22.0784721Z inflating: build/lib/libunbox_lib.a 2024-06-26T06:41:24.5335454Z inflating: build/lib/libtorch_cuda.so 2024-06-26T06:41:24.5336526Z inflating: build/lib/libtorch.so 2024-06-26T06:41:24.5339715Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T06:41:25.6716947Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T06:41:25.6742553Z inflating: build/lib/libjitbackend_test.so 2024-06-26T06:41:25.6828033Z inflating: build/lib/libtorchbind_test.so 2024-06-26T06:41:25.6859893Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T06:41:25.9310551Z inflating: build/lib/libtorch_python.so 2024-06-26T06:41:25.9353368Z inflating: build/lib/libnnapi_backend.so 2024-06-26T06:41:25.9353970Z creating: build/bin/ 2024-06-26T06:41:25.9416448Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T06:41:25.9481230Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T06:41:25.9543894Z inflating: build/bin/c10_Device_test 2024-06-26T06:41:25.9617848Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T06:41:25.9684844Z inflating: build/bin/c10_Scalar_test 2024-06-26T06:41:25.9745871Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T06:41:25.9809507Z inflating: build/bin/c10_SymInt_test 2024-06-26T06:41:25.9877720Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T06:41:25.9946926Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T06:41:26.0017264Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T06:41:26.0105777Z inflating: build/bin/c10_cow_test 2024-06-26T06:41:26.0172454Z inflating: build/bin/c10_Bitset_test 2024-06-26T06:41:26.0233457Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T06:41:26.0295775Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T06:41:26.0359412Z inflating: build/bin/c10_Half_test 2024-06-26T06:41:26.0429464Z inflating: build/bin/c10_LeftRight_test 2024-06-26T06:41:26.0498478Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T06:41:26.0561036Z inflating: build/bin/c10_Synchronized_test 2024-06-26T06:41:26.0630297Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T06:41:26.0694866Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T06:41:26.0759518Z inflating: build/bin/c10_TypeList_test 2024-06-26T06:41:26.0820580Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T06:41:26.0885408Z inflating: build/bin/c10_accumulate_test 2024-06-26T06:41:26.0954552Z inflating: build/bin/c10_bfloat16_test 2024-06-26T06:41:26.1017715Z inflating: build/bin/c10_bit_cast_test 2024-06-26T06:41:26.1088399Z inflating: build/bin/c10_complex_math_test 2024-06-26T06:41:26.1159136Z inflating: build/bin/c10_complex_test 2024-06-26T06:41:26.1224733Z inflating: build/bin/c10_exception_test 2024-06-26T06:41:26.1287729Z inflating: build/bin/c10_flags_test 2024-06-26T06:41:26.1350245Z inflating: build/bin/c10_generic_math_test 2024-06-26T06:41:26.1556055Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T06:41:26.1619682Z inflating: build/bin/c10_irange_test 2024-06-26T06:41:26.1686550Z inflating: build/bin/c10_lazy_test 2024-06-26T06:41:26.1757808Z inflating: build/bin/c10_logging_test 2024-06-26T06:41:26.1851240Z inflating: build/bin/c10_optional_test 2024-06-26T06:41:26.1929224Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T06:41:26.1996335Z inflating: build/bin/c10_registry_test 2024-06-26T06:41:26.2185813Z inflating: build/bin/c10_small_vector_test 2024-06-26T06:41:26.2250769Z inflating: build/bin/c10_ssize_test 2024-06-26T06:41:26.2315491Z inflating: build/bin/c10_string_util_test 2024-06-26T06:41:26.2389013Z inflating: build/bin/c10_string_view_test 2024-06-26T06:41:26.2452106Z inflating: build/bin/c10_tempfile_test 2024-06-26T06:41:26.2512719Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T06:41:26.2582851Z inflating: build/bin/c10_typeid_test 2024-06-26T06:41:26.3140645Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T06:41:26.3697419Z inflating: build/bin/protoc 2024-06-26T06:41:26.3763705Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T06:41:26.3829650Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T06:41:26.3895801Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T06:41:26.3960816Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T06:41:26.4026616Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T06:41:26.4092716Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T06:41:26.4158410Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T06:41:26.4219515Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T06:41:26.4631669Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T06:41:26.5062904Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T06:41:26.5509733Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T06:41:26.5576416Z inflating: build/bin/HashStoreTest 2024-06-26T06:41:26.5642631Z inflating: build/bin/FileStoreTest 2024-06-26T06:41:26.5712011Z inflating: build/bin/TCPStoreTest 2024-06-26T06:41:26.5729149Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T06:41:26.5734392Z inflating: build/bin/torch_shm_manager 2024-06-26T06:41:26.5801828Z inflating: build/bin/test_edge_op_registration 2024-06-26T06:41:26.5805444Z inflating: build/bin/example_allreduce 2024-06-26T06:41:26.5870257Z inflating: build/bin/Dimname_test 2024-06-26T06:41:26.5962546Z inflating: build/bin/Dict_test 2024-06-26T06:41:26.6043472Z inflating: build/bin/MaybeOwned_test 2024-06-26T06:41:26.6118310Z inflating: build/bin/apply_utils_test 2024-06-26T06:41:26.6189725Z inflating: build/bin/NamedTensor_test 2024-06-26T06:41:26.6263595Z inflating: build/bin/atest 2024-06-26T06:41:26.6342274Z inflating: build/bin/basic 2024-06-26T06:41:26.6410750Z inflating: build/bin/broadcast_test 2024-06-26T06:41:26.6474190Z inflating: build/bin/cpu_allocator_test 2024-06-26T06:41:26.6546903Z inflating: build/bin/cpu_generator_test 2024-06-26T06:41:26.6613321Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T06:41:26.6729951Z inflating: build/bin/cpu_rng_test 2024-06-26T06:41:26.6792405Z inflating: build/bin/dispatch_key_set_test 2024-06-26T06:41:26.6855381Z inflating: build/bin/dlconvertor_test 2024-06-26T06:41:26.6928169Z inflating: build/bin/extension_backend_test 2024-06-26T06:41:26.6997000Z inflating: build/bin/half_test 2024-06-26T06:41:26.7117177Z inflating: build/bin/ivalue_test 2024-06-26T06:41:26.7179313Z inflating: build/bin/lazy_tensor_test 2024-06-26T06:41:26.7247605Z inflating: build/bin/math_kernel_test 2024-06-26T06:41:26.7315198Z inflating: build/bin/memory_format_test 2024-06-26T06:41:26.7382199Z inflating: build/bin/memory_overlapping_test 2024-06-26T06:41:26.7448570Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T06:41:26.7511711Z inflating: build/bin/operator_name_test 2024-06-26T06:41:26.7581755Z inflating: build/bin/native_test 2024-06-26T06:41:26.7646143Z inflating: build/bin/operators_test 2024-06-26T06:41:26.7711217Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T06:41:26.7795804Z inflating: build/bin/pow_test 2024-06-26T06:41:26.7867327Z inflating: build/bin/quantized_test 2024-06-26T06:41:26.7930125Z inflating: build/bin/reduce_ops_test 2024-06-26T06:41:26.7994019Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T06:41:26.8064637Z inflating: build/bin/scalar_tensor_test 2024-06-26T06:41:26.8137751Z inflating: build/bin/scalar_test 2024-06-26T06:41:26.8202523Z inflating: build/bin/StorageUtils_test 2024-06-26T06:41:26.8268236Z inflating: build/bin/stride_properties_test 2024-06-26T06:41:26.8366870Z inflating: build/bin/tensor_iterator_test 2024-06-26T06:41:26.8370553Z inflating: build/bin/thread_init_test 2024-06-26T06:41:26.8438197Z inflating: build/bin/test_parallel 2024-06-26T06:41:26.8507798Z inflating: build/bin/type_ptr_test 2024-06-26T06:41:26.8582767Z inflating: build/bin/type_test 2024-06-26T06:41:26.8648308Z inflating: build/bin/undefined_tensor_test 2024-06-26T06:41:26.8650333Z inflating: build/bin/verify_api_visibility 2024-06-26T06:41:26.8714503Z inflating: build/bin/weakref_test 2024-06-26T06:41:26.8801395Z inflating: build/bin/legacy_vmap_test 2024-06-26T06:41:26.8865749Z inflating: build/bin/wrapdim_test 2024-06-26T06:41:26.8930304Z inflating: build/bin/xla_tensor_test 2024-06-26T06:41:26.9005614Z inflating: build/bin/IListRef_test 2024-06-26T06:41:26.9138524Z inflating: build/bin/List_test 2024-06-26T06:41:26.9291362Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T06:41:26.9374615Z inflating: build/bin/KernelFunction_test 2024-06-26T06:41:26.9496303Z inflating: build/bin/kernel_function_test 2024-06-26T06:41:26.9656524Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T06:41:26.9786154Z inflating: build/bin/kernel_lambda_test 2024-06-26T06:41:26.9862833Z inflating: build/bin/kernel_stackbased_test 2024-06-26T06:41:26.9983208Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T06:41:27.0047093Z inflating: build/bin/CppSignature_test 2024-06-26T06:41:27.0116954Z inflating: build/bin/backend_fallback_test 2024-06-26T06:41:27.0177991Z inflating: build/bin/op_allowlist_test 2024-06-26T06:41:27.0257439Z inflating: build/bin/inline_container_test 2024-06-26T06:41:27.0635009Z inflating: build/bin/op_registration_test 2024-06-26T06:41:27.0701253Z inflating: build/bin/cuda_apply_test 2024-06-26T06:41:27.0765098Z inflating: build/bin/cuda_allocator_test 2024-06-26T06:41:27.0838879Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T06:41:27.0907321Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T06:41:27.0995289Z inflating: build/bin/cuda_complex_math_test 2024-06-26T06:41:27.1057287Z inflating: build/bin/cuda_device_test 2024-06-26T06:41:27.1132487Z inflating: build/bin/cuda_complex_test 2024-06-26T06:41:27.1204831Z inflating: build/bin/cuda_cub_test 2024-06-26T06:41:27.1268408Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T06:41:27.1332472Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T06:41:27.1414271Z inflating: build/bin/cuda_distributions_test 2024-06-26T06:41:27.1485318Z inflating: build/bin/cuda_generator_test 2024-06-26T06:41:27.1547308Z inflating: build/bin/cuda_half_test 2024-06-26T06:41:27.1609230Z inflating: build/bin/cuda_optional_test 2024-06-26T06:41:27.1674822Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T06:41:27.1737308Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T06:41:27.1801677Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T06:41:27.1877416Z inflating: build/bin/cuda_stream_test 2024-06-26T06:41:27.1938963Z inflating: build/bin/cuda_cudnn_test 2024-06-26T06:41:27.2003972Z inflating: build/bin/cuda_vectorized_test 2024-06-26T06:41:27.2022730Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T06:41:27.2105176Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T06:41:27.2184326Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T06:41:27.2256207Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T06:41:27.2321973Z inflating: build/bin/ProcessGroupUCCTest 2024-06-26T06:41:27.2400220Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T06:41:27.2468769Z inflating: build/bin/test_dist_autograd 2024-06-26T06:41:27.2555471Z inflating: build/bin/test_cpp_rpc 2024-06-26T06:41:27.2558941Z inflating: build/bin/parallel_benchmark 2024-06-26T06:41:27.3598326Z inflating: build/bin/test_tensorexpr 2024-06-26T06:41:27.3609582Z inflating: build/bin/aot_model_compiler_test 2024-06-26T06:41:27.3693258Z inflating: build/bin/test_mobile_nnc 2024-06-26T06:41:27.4125827Z inflating: build/bin/test_lazy 2024-06-26T06:41:27.5603444Z inflating: build/bin/test_api 2024-06-26T06:41:27.6326048Z inflating: build/bin/test_jit 2024-06-26T06:41:27.6326639Z creating: .additional_ci_files/ 2024-06-26T06:41:27.6395494Z inflating: .additional_ci_files/test-times.json 2024-06-26T06:41:27.6666406Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T06:41:27.6705248Z ##[group]Run rm artifacts.zip 2024-06-26T06:41:27.6705744Z rm artifacts.zip 2024-06-26T06:41:27.6714173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:27.6714823Z env: 2024-06-26T06:41:27.6715164Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:27.6715713Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:27.6716598Z ##[endgroup] 2024-06-26T06:41:27.7909530Z ##[group]Run df -H 2024-06-26T06:41:27.7909927Z df -H 2024-06-26T06:41:27.7918806Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:27.7919443Z env: 2024-06-26T06:41:27.7919789Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:27.7920342Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:27.7920953Z ##[endgroup] 2024-06-26T06:41:27.7955242Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T06:41:27.7955928Z devtmpfs 129G 0 129G 0% /dev 2024-06-26T06:41:27.7957087Z tmpfs 129G 8.2M 129G 1% /dev/shm 2024-06-26T06:41:27.7958602Z tmpfs 129G 533k 129G 1% /run 2024-06-26T06:41:27.7959472Z tmpfs 129G 0 129G 0% /sys/fs/cgroup 2024-06-26T06:41:27.7960254Z /dev/xvda1 162G 37G 125G 23% / 2024-06-26T06:41:27.8029480Z Prepare all required actions 2024-06-26T06:41:27.8030044Z Getting action download info 2024-06-26T06:41:27.9218246Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T06:41:27.9218834Z with: 2024-06-26T06:41:27.9219134Z env: 2024-06-26T06:41:27.9219477Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:27.9220035Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:27.9220647Z ##[endgroup] 2024-06-26T06:41:27.9264415Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T06:41:27.9264973Z with: 2024-06-26T06:41:27.9265288Z name: td_results 2024-06-26T06:41:27.9265683Z s3-bucket: gha-artifacts 2024-06-26T06:41:27.9266113Z region: us-east-1 2024-06-26T06:41:27.9266459Z env: 2024-06-26T06:41:27.9266803Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:27.9267363Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:27.9267952Z ##[endgroup] 2024-06-26T06:41:28.5346544Z (node:110996) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T06:41:28.5347275Z 2024-06-26T06:41:28.5347604Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T06:41:28.5348524Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T06:41:28.5349585Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T06:41:28.6271248Z Found 1 objects with prefix pytorch/pytorch/9673645538/td_results/ 2024-06-26T06:41:28.6272381Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T06:41:28.7635251Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T06:41:28.7646143Z Artifact download has finished successfully 2024-06-26T06:41:28.7831512Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T06:41:28.7832120Z mkdir -p .additional_ci_files 2024-06-26T06:41:28.7832840Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T06:41:28.7841685Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:28.7842319Z env: 2024-06-26T06:41:28.7842661Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:28.7843210Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:28.7843826Z ##[endgroup] 2024-06-26T06:41:28.7930066Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T06:41:28.7930686Z .github/scripts/parse_ref.py 2024-06-26T06:41:28.7938453Z shell: /usr/bin/bash -e {0} 2024-06-26T06:41:28.7938890Z env: 2024-06-26T06:41:28.7939231Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:28.7939778Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:28.7940383Z ##[endgroup] 2024-06-26T06:41:28.8257406Z Prepare all required actions 2024-06-26T06:41:28.8308706Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T06:41:28.8309276Z with: 2024-06-26T06:41:28.8309982Z github-token: *** 2024-06-26T06:41:28.8310347Z env: 2024-06-26T06:41:28.8310678Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:28.8311238Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:28.8311821Z ##[endgroup] 2024-06-26T06:41:28.8335414Z ##[group]Run set -eux 2024-06-26T06:41:28.8335838Z set -eux 2024-06-26T06:41:28.8336584Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T06:41:28.8344435Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:28.8345046Z env: 2024-06-26T06:41:28.8345384Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:28.8345954Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:28.8346763Z GITHUB_TOKEN: *** 2024-06-26T06:41:28.8347159Z ##[endgroup] 2024-06-26T06:41:28.8372083Z + python3 .github/scripts/get_workflow_job_id.py 9673645538 i-05c3ee73f141ef674 2024-06-26T06:41:31.3287795Z setting job-id=26688546634 2024-06-26T06:41:31.3288927Z setting job-name=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:41:31.3548919Z Prepare all required actions 2024-06-26T06:41:31.3549501Z Getting action download info 2024-06-26T06:41:31.4688393Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T06:41:31.4688983Z with: 2024-06-26T06:41:31.4689567Z github-token: *** 2024-06-26T06:41:31.4691414Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]} 2024-06-26T06:41:31.4693774Z job-name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:41:31.4694715Z env: 2024-06-26T06:41:31.4695067Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:31.4695621Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:31.4696239Z ##[endgroup] 2024-06-26T06:41:31.4751906Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T06:41:31.4752611Z with: 2024-06-26T06:41:31.4752944Z shell: bash 2024-06-26T06:41:31.4753313Z timeout_minutes: 10 2024-06-26T06:41:31.4753699Z max_attempts: 5 2024-06-26T06:41:31.4754086Z retry_wait_seconds: 30 2024-06-26T06:41:31.4755601Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T06:41:31.4757457Z polling_interval_seconds: 1 2024-06-26T06:41:31.4757912Z warning_on_retry: true 2024-06-26T06:41:31.4758350Z continue_on_error: false 2024-06-26T06:41:31.4758773Z env: 2024-06-26T06:41:31.4759089Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:31.4759663Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:31.4760496Z GITHUB_TOKEN: *** 2024-06-26T06:41:31.4760870Z ##[endgroup] 2024-06-26T06:41:31.5479405Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T06:41:31.8424339Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T06:41:31.8632211Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T06:41:31.8817267Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T06:41:31.8829739Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T06:41:31.8848084Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T06:41:31.8874662Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T06:41:31.9123276Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T06:41:32.5473908Z Command completed after 1 attempt(s). 2024-06-26T06:41:32.5531833Z ##[group]Run set -x 2024-06-26T06:41:32.5532267Z set -x 2024-06-26T06:41:32.5532638Z  2024-06-26T06:41:32.5533333Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T06:41:32.5534192Z # in runner workspace 2024-06-26T06:41:32.5534862Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T06:41:32.5543589Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:32.5544212Z env: 2024-06-26T06:41:32.5544567Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:32.5545141Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:32.5546024Z ##[endgroup] 2024-06-26T06:41:32.5575075Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T06:41:32.5879605Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T06:41:32.5880318Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T06:41:32.5880917Z echo "Job name: ${JOB_NAME}" 2024-06-26T06:41:32.5881430Z  2024-06-26T06:41:32.5882128Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T06:41:32.5882987Z # in runner workspace 2024-06-26T06:41:32.5883724Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T06:41:32.5884548Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T06:41:32.5885116Z  --job-name "${JOB_NAME}" \ 2024-06-26T06:41:32.5887156Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" \ 2024-06-26T06:41:32.5889247Z  --selected-test-configs "" \ 2024-06-26T06:41:32.5889829Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T06:41:32.5890345Z  --tag "${TAG}" \ 2024-06-26T06:41:32.5890840Z  --event-name "${EVENT_NAME}" \ 2024-06-26T06:41:32.5891406Z  --schedule "${SCHEDULE}" \ 2024-06-26T06:41:32.5891933Z  --branch "${HEAD_BRANCH}" 2024-06-26T06:41:32.5899916Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:32.5900551Z env: 2024-06-26T06:41:32.5900886Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:32.5901463Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:32.5902328Z GITHUB_TOKEN: *** 2024-06-26T06:41:32.5903177Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:41:32.5904095Z PR_NUMBER: 129470 2024-06-26T06:41:32.5904483Z TAG: 2024-06-26T06:41:32.5904836Z EVENT_NAME: pull_request 2024-06-26T06:41:32.5905266Z SCHEDULE: 2024-06-26T06:41:32.5905637Z HEAD_BRANCH: 2024-06-26T06:41:32.5905990Z ##[endgroup] 2024-06-26T06:41:32.5930976Z Workflow: pull 2024-06-26T06:41:32.5931947Z Job name: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:41:32.8252848Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T06:41:32.9714686Z ##[group]Run echo "Filtered matrix:" 2024-06-26T06:41:32.9715260Z echo "Filtered matrix:" 2024-06-26T06:41:32.9717584Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.8xlarge.nvidia.gpu"}]}" 2024-06-26T06:41:32.9720109Z  2024-06-26T06:41:32.9720466Z echo 2024-06-26T06:41:32.9720960Z echo "Is the current job unstable? False" 2024-06-26T06:41:32.9721525Z  2024-06-26T06:41:32.9721874Z echo 2024-06-26T06:41:32.9722327Z echo "Is keep-going label set? False" 2024-06-26T06:41:32.9722869Z  2024-06-26T06:41:32.9723213Z echo 2024-06-26T06:41:32.9723608Z echo "Renabled issues? " 2024-06-26T06:41:32.9731773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:32.9732416Z env: 2024-06-26T06:41:32.9732769Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:32.9733324Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:32.9733939Z ##[endgroup] 2024-06-26T06:41:32.9760122Z Filtered matrix: 2024-06-26T06:41:32.9762332Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 2, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}, {config: distributed, shard: 3, num_shards: 3, runner: linux.8xlarge.nvidia.gpu}]} 2024-06-26T06:41:32.9764168Z 2024-06-26T06:41:32.9764541Z Is the current job unstable? False 2024-06-26T06:41:32.9764927Z 2024-06-26T06:41:32.9765214Z Is keep-going label set? False 2024-06-26T06:41:32.9765562Z 2024-06-26T06:41:32.9765721Z Renabled issues? 2024-06-26T06:41:32.9836830Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T06:41:32.9837796Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T06:41:32.9845868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:41:32.9846504Z env: 2024-06-26T06:41:32.9846850Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:32.9847396Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:32.9848015Z JOB_TIMEOUT: 360 2024-06-26T06:41:32.9848396Z ##[endgroup] 2024-06-26T06:41:32.9945139Z ##[group]Run set -x 2024-06-26T06:41:32.9945692Z set -x 2024-06-26T06:41:32.9946069Z  2024-06-26T06:41:32.9946534Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T06:41:32.9947206Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T06:41:32.9947921Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T06:41:32.9948578Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T06:41:32.9949085Z else 2024-06-26T06:41:32.9949512Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T06:41:32.9950053Z fi 2024-06-26T06:41:32.9950375Z  2024-06-26T06:41:32.9950967Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T06:41:32.9951949Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T06:41:32.9952822Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T06:41:32.9953584Z # shellcheck disable=SC2086,SC2090 2024-06-26T06:41:32.9954160Z container_name=$(docker run \ 2024-06-26T06:41:32.9954707Z  ${GPU_FLAG:-} \ 2024-06-26T06:41:32.9955178Z  -e BUILD_ENVIRONMENT \ 2024-06-26T06:41:32.9955657Z  -e PR_NUMBER \ 2024-06-26T06:41:32.9956108Z  -e GITHUB_ACTIONS \ 2024-06-26T06:41:32.9956908Z  -e GITHUB_REPOSITORY \ 2024-06-26T06:41:32.9957406Z  -e GITHUB_WORKFLOW \ 2024-06-26T06:41:32.9957887Z  -e GITHUB_JOB \ 2024-06-26T06:41:32.9958368Z  -e GITHUB_RUN_ID \ 2024-06-26T06:41:32.9958851Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T06:41:32.9959340Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T06:41:32.9959838Z  -e JOB_ID \ 2024-06-26T06:41:32.9960260Z  -e JOB_NAME \ 2024-06-26T06:41:32.9960676Z  -e BASE_SHA \ 2024-06-26T06:41:32.9961107Z  -e BRANCH \ 2024-06-26T06:41:32.9961522Z  -e SHA1 \ 2024-06-26T06:41:32.9961932Z  -e AWS_DEFAULT_REGION \ 2024-06-26T06:41:32.9962453Z  -e IN_WHEEL_TEST \ 2024-06-26T06:41:32.9962928Z  -e SHARD_NUMBER \ 2024-06-26T06:41:32.9963370Z  -e TEST_CONFIG \ 2024-06-26T06:41:32.9963836Z  -e NUM_TEST_SHARDS \ 2024-06-26T06:41:32.9964325Z  -e REENABLED_ISSUES \ 2024-06-26T06:41:32.9964818Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T06:41:32.9965347Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T06:41:32.9965847Z  -e NO_TEST_TIMEOUT \ 2024-06-26T06:41:32.9966296Z  -e NO_TD \ 2024-06-26T06:41:32.9966722Z  -e TD_DISTRIBUTED \ 2024-06-26T06:41:32.9967197Z  -e PR_LABELS \ 2024-06-26T06:41:32.9967683Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T06:41:32.9968259Z  -e SCCACHE_BUCKET \ 2024-06-26T06:41:32.9968754Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T06:41:32.9969246Z  -e XLA_CUDA \ 2024-06-26T06:41:32.9969756Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T06:41:32.9970623Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T06:41:32.9971264Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T06:41:32.9971918Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T06:41:32.9972511Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T06:41:32.9973017Z  -e DASHBOARD_TAG \ 2024-06-26T06:41:32.9973615Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T06:41:32.9974320Z  --security-opt seccomp=unconfined \ 2024-06-26T06:41:32.9974883Z  --cap-add=SYS_PTRACE \ 2024-06-26T06:41:32.9975377Z  --ipc=host \ 2024-06-26T06:41:32.9975826Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T06:41:32.9976325Z  --tty \ 2024-06-26T06:41:32.9976697Z  --detach \ 2024-06-26T06:41:32.9977142Z  --name="${container_name}" \ 2024-06-26T06:41:32.9977665Z  --user jenkins \ 2024-06-26T06:41:32.9978403Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T06:41:32.9979153Z  -w /var/lib/jenkins/workspace \ 2024-06-26T06:41:32.9979703Z  "${DOCKER_IMAGE}" 2024-06-26T06:41:32.9980117Z ) 2024-06-26T06:41:32.9980624Z # Propagate download.pytorch.org IP to container 2024-06-26T06:41:32.9981799Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T06:41:32.9983026Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T06:41:32.9984181Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T06:41:32.9992552Z shell: /usr/bin/bash -e {0} 2024-06-26T06:41:32.9992993Z env: 2024-06-26T06:41:32.9993341Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:41:32.9993889Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:41:32.9994634Z BUILD_ENVIRONMENT: linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:41:32.9995257Z PR_NUMBER: 129470 2024-06-26T06:41:32.9995663Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T06:41:32.9996407Z GITHUB_WORKFLOW: pull 2024-06-26T06:41:32.9996875Z GITHUB_JOB: test 2024-06-26T06:41:32.9997253Z GITHUB_RUN_ID: 9673645538 2024-06-26T06:41:32.9997698Z GITHUB_RUN_NUMBER: 219936 2024-06-26T06:41:32.9998150Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T06:41:32.9998571Z JOB_ID: 26688546634 2024-06-26T06:41:32.9999387Z JOB_NAME: linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:41:33.0000329Z BRANCH: pull/129470 2024-06-26T06:41:33.0000777Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:41:33.0001424Z BASE_SHA: 4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:41:33.0002019Z TEST_CONFIG: distributed 2024-06-26T06:41:33.0002440Z SHARD_NUMBER: 2 2024-06-26T06:41:33.0002822Z NUM_TEST_SHARDS: 3 2024-06-26T06:41:33.0003217Z REENABLED_ISSUES: 2024-06-26T06:41:33.0003629Z CONTINUE_THROUGH_ERROR: False 2024-06-26T06:41:33.0004119Z VERBOSE_TEST_LOGS: False 2024-06-26T06:41:33.0004556Z NO_TEST_TIMEOUT: False 2024-06-26T06:41:33.0004952Z NO_TD: False 2024-06-26T06:41:33.0005325Z TD_DISTRIBUTED: False 2024-06-26T06:41:33.0005842Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T06:41:33.0006451Z SCCACHE_S3_KEY_PREFIX: pull 2024-06-26T06:41:33.0006883Z SHM_SIZE: 2g 2024-06-26T06:41:33.0008083Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:41:33.0009404Z XLA_CUDA: 2024-06-26T06:41:33.0009986Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:41:33.0010754Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T06:41:33.0011292Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T06:41:33.0011784Z DASHBOARD_TAG: 2024-06-26T06:41:33.0012179Z HUGGING_FACE_HUB_TOKEN: 2024-06-26T06:41:33.0012608Z ##[endgroup] 2024-06-26T06:41:33.0038501Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-06-26T06:41:33.0039258Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *onnx* ]] 2024-06-26T06:41:33.0039880Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T06:41:33.0049050Z +++ nproc --ignore=2 2024-06-26T06:41:33.0066885Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673645538 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:41:44.1931834Z + container_name=30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:41:44.1934554Z + grep download.pytorch.org /etc/hosts 2024-06-26T06:41:44.1936088Z + docker exec -i 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T06:41:44.2596079Z + echo DOCKER_CONTAINER_ID=30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:41:44.2600247Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T06:41:44.2602399Z + docker exec -t 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T06:41:44.7941972Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T06:41:45.9194007Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T06:41:45.9201808Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T06:41:45.9206860Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T06:41:45.9212019Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T06:41:45.9217169Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T06:41:45.9222347Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T06:41:45.9260732Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T06:41:45.9343439Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T06:41:45.9958429Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T06:41:46.0186107Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T06:41:47.1214677Z Installing collected packages: torch 2024-06-26T06:41:59.7393113Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T06:41:59.8074916Z ++ dirname .ci/pytorch/test.sh 2024-06-26T06:41:59.8082134Z + source .ci/pytorch/common.sh 2024-06-26T06:41:59.8085831Z +++ dirname .ci/pytorch/common.sh 2024-06-26T06:41:59.8093477Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T06:41:59.8095361Z +++ declare -f -t trap_add 2024-06-26T06:41:59.8102595Z ++ set -ex 2024-06-26T06:41:59.8105255Z ++ [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:41:59.8105832Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T06:41:59.8106575Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-06-26T06:41:59.8107670Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T06:41:59.8120597Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T06:41:59.8120908Z + trap_add cleanup_workspace EXIT 2024-06-26T06:41:59.8121589Z + trap_add_cmd=cleanup_workspace 2024-06-26T06:41:59.8122019Z + shift 2024-06-26T06:41:59.8122380Z + for trap_add_name in "$@" 2024-06-26T06:41:59.8129257Z +++ trap -p EXIT 2024-06-26T06:41:59.8131948Z ++ eval 'extract_trap_cmd ' 2024-06-26T06:41:59.8132515Z +++ extract_trap_cmd 2024-06-26T06:41:59.8133054Z +++ printf '%s\n' '' 2024-06-26T06:41:59.8133250Z ++ printf '%s\n' cleanup_workspace 2024-06-26T06:41:59.8135116Z + trap -- ' 2024-06-26T06:41:59.8135192Z cleanup_workspace' EXIT 2024-06-26T06:41:59.8135729Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T06:42:00.3524568Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T06:42:00.3541879Z + echo 'Environment variables:' 2024-06-26T06:42:00.3542384Z Environment variables: 2024-06-26T06:42:00.3542667Z + env 2024-06-26T06:42:00.3550837Z INSTALLED_DB=yes 2024-06-26T06:42:00.3551471Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-06-26T06:42:00.3551945Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T06:42:00.3552445Z NV_NVML_DEV_VERSION=11.8.86-1 2024-06-26T06:42:00.3553147Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:42:00.3555509Z CONTINUE_THROUGH_ERROR=False 2024-06-26T06:42:00.3556390Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-06-26T06:42:00.3557351Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:42:00.3557879Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:42:00.3558456Z HOSTNAME=30e36954f97c 2024-06-26T06:42:00.3559964Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3560953Z GITHUB_ACTION=__self 2024-06-26T06:42:00.3561247Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T06:42:00.3563898Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-06-26T06:42:00.3566942Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-06-26T06:42:00.3567382Z NV_NVTX_VERSION=11.8.86-1 2024-06-26T06:42:00.3567767Z GITHUB_RUN_NUMBER=219936 2024-06-26T06:42:00.3568320Z TEST_CONFIG=distributed 2024-06-26T06:42:00.3568842Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T06:42:00.3569474Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T06:42:00.3570211Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-06-26T06:42:00.3570893Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-06-26T06:42:00.3571524Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-06-26T06:42:00.3571968Z GITHUB_TRIGGERING_ACTOR=leslie-fang-intel 2024-06-26T06:42:00.3572670Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T06:42:00.3573271Z GITHUB_REF_TYPE=branch 2024-06-26T06:42:00.3573806Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T06:42:00.3574278Z NCCL_VERSION=2.16.2-1 2024-06-26T06:42:00.3574514Z BASE_SHA=4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:42:00.3575378Z XLA_CUDA= 2024-06-26T06:42:00.3575782Z HUGGING_FACE_HUB_TOKEN= 2024-06-26T06:42:00.3582577Z *** 2024-06-26T06:42:00.3582753Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T06:42:00.3583731Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T06:42:00.3583783Z GITHUB_ACTIONS=true 2024-06-26T06:42:00.3584103Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:42:00.3585001Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-06-26T06:42:00.3585458Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-06-26T06:42:00.3586195Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:42:00.3586655Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T06:42:00.3587400Z GITHUB_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:42:00.3588563Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/129470/merge 2024-06-26T06:42:00.3589628Z UCC_HOME=/usr 2024-06-26T06:42:00.3589768Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-06-26T06:42:00.3590136Z VERBOSE_TEST_LOGS=False 2024-06-26T06:42:00.3590544Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T06:42:00.3591266Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-06-26T06:42:00.3591702Z GITHUB_REF=refs/pull/129470/merge 2024-06-26T06:42:00.3592200Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-06-26T06:42:00.3592664Z SHARD_NUMBER=2 2024-06-26T06:42:00.3593516Z GITHUB_REF_PROTECTED=false 2024-06-26T06:42:00.3593530Z HOME=/var/lib/jenkins 2024-06-26T06:42:00.3593914Z GITHUB_API_URL=https://api.github.com 2024-06-26T06:42:00.3594464Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T06:42:00.3595039Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T06:42:00.3595635Z SCCACHE_S3_KEY_PREFIX=pull 2024-06-26T06:42:00.3596531Z CUDA_VERSION=11.8.0 2024-06-26T06:42:00.3596867Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-06-26T06:42:00.3597694Z NUM_TEST_SHARDS=3 2024-06-26T06:42:00.3597810Z UCX_HOME=/usr 2024-06-26T06:42:00.3598430Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-06-26T06:42:00.3599795Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3601345Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:42:00.3602883Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3604740Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T06:42:00.3605135Z GITHUB_EVENT_NAME=pull_request 2024-06-26T06:42:00.3605592Z DASHBOARD_TAG= 2024-06-26T06:42:00.3605948Z GITHUB_RUN_ID=9673645538 2024-06-26T06:42:00.3606511Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-06-26T06:42:00.3607185Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-06-26T06:42:00.3608435Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3609596Z GITHUB_ACTOR=leslie-fang-intel 2024-06-26T06:42:00.3610103Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-06-26T06:42:00.3610557Z PR_NUMBER=129470 2024-06-26T06:42:00.3610935Z GITHUB_RUN_ATTEMPT=1 2024-06-26T06:42:00.3611350Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T06:42:00.3611889Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T06:42:00.3612468Z TERM=xterm 2024-06-26T06:42:00.3612882Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-06-26T06:42:00.3613373Z INSTALLED_VISION=yes 2024-06-26T06:42:00.3613774Z BRANCH=pull/129470 2024-06-26T06:42:00.3614173Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T06:42:00.3614655Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T06:42:00.3615187Z CUDA_PATH=/usr/local/cuda 2024-06-26T06:42:00.3616143Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T06:42:00.3617255Z GITHUB_SERVER_URL=https://github.com 2024-06-26T06:42:00.3618316Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T06:42:00.3618449Z REENABLED_ISSUES= 2024-06-26T06:42:00.3618795Z SHLVL=1 2024-06-26T06:42:00.3619118Z MAX_JOBS=30 2024-06-26T06:42:00.3619503Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-06-26T06:42:00.3619923Z NVARCH=x86_64 2024-06-26T06:42:00.3620290Z GITHUB_ACTOR_ID=53841472 2024-06-26T06:42:00.3620860Z GITHUB_WORKFLOW_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:42:00.3621494Z GITHUB_REF_NAME=129470/merge 2024-06-26T06:42:00.3622027Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-06-26T06:42:00.3622858Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:42:00.3623561Z GITHUB_JOB=test 2024-06-26T06:42:00.3624056Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-06-26T06:42:00.3624756Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T06:42:00.3625382Z NO_TEST_TIMEOUT=False 2024-06-26T06:42:00.3625925Z TD_DISTRIBUTED=False 2024-06-26T06:42:00.3626467Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-06-26T06:42:00.3626996Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T06:42:00.3627520Z NV_NVPROF_VERSION=11.8.87-1 2024-06-26T06:42:00.3627970Z GITHUB_RETENTION_DAYS=90 2024-06-26T06:42:00.3628388Z OPENSSL_DIR=/opt/openssl 2024-06-26T06:42:00.3628822Z GITHUB_ACTION_REPOSITORY= 2024-06-26T06:42:00.3630151Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:42:00.3631814Z GITHUB_BASE_REF=gh/leslie-fang-intel/126/base 2024-06-26T06:42:00.3632423Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T06:42:00.3632899Z CI=true 2024-06-26T06:42:00.3633243Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:42:00.3633766Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T06:42:00.3634080Z JOB_ID=26688546634 2024-06-26T06:42:00.3634610Z INSTALLED_PROTOBUF=yes 2024-06-26T06:42:00.3635164Z GITHUB_HEAD_REF=gh/leslie-fang-intel/126/head 2024-06-26T06:42:00.3635715Z GITHUB_ACTION_REF= 2024-06-26T06:42:00.3636509Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T06:42:00.3637009Z GITHUB_WORKFLOW=pull 2024-06-26T06:42:00.3637431Z DEBIAN_FRONTEND=noninteractive 2024-06-26T06:42:00.3638472Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3639422Z NO_TD=False 2024-06-26T06:42:00.3639792Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T06:42:00.3640228Z _=/usr/bin/env 2024-06-26T06:42:00.3640821Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T06:42:00.3780263Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T06:42:00.3782079Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T06:42:00.3783682Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T06:42:00.3785732Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T06:42:00.3786749Z + BUILD_DIR=build 2024-06-26T06:42:00.3787168Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T06:42:00.3787683Z + BUILD_BIN_DIR=build/bin 2024-06-26T06:42:00.3788268Z + SHARD_NUMBER=2 2024-06-26T06:42:00.3788844Z + NUM_TEST_SHARDS=3 2024-06-26T06:42:00.3789370Z + export VALGRIND=ON 2024-06-26T06:42:00.3790105Z + VALGRIND=ON 2024-06-26T06:42:00.3790799Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *clang9* ]] 2024-06-26T06:42:00.3791401Z + [[ 0 == \1 ]] 2024-06-26T06:42:00.3791764Z + [[ False == \1 ]] 2024-06-26T06:42:00.3792609Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *bazel* ]] 2024-06-26T06:42:00.3793470Z ++ realpath build/custom_test_artifacts 2024-06-26T06:42:00.3796772Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T06:42:00.3797869Z + [[ -n '' ]] 2024-06-26T06:42:00.3798180Z + echo 'Environment variables' 2024-06-26T06:42:00.3798654Z Environment variables 2024-06-26T06:42:00.3799275Z + env 2024-06-26T06:42:00.3804767Z INSTALLED_DB=yes 2024-06-26T06:42:00.3805236Z NV_LIBCUBLAS_VERSION=11.11.3.6-1 2024-06-26T06:42:00.3805778Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T06:42:00.3806343Z NV_NVML_DEV_VERSION=11.8.86-1 2024-06-26T06:42:00.3807416Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:42:00.3808388Z CONTINUE_THROUGH_ERROR=False 2024-06-26T06:42:00.3809046Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8 2024-06-26T06:42:00.3809925Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:42:00.3810685Z BUILD_ENVIRONMENT=linux-focal-cuda11.8-py3.10-gcc9 2024-06-26T06:42:00.3811190Z HOSTNAME=30e36954f97c 2024-06-26T06:42:00.3812494Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3814173Z GITHUB_ACTION=__self 2024-06-26T06:42:00.3814806Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T06:42:00.3818148Z NVIDIA_REQUIRE_CUDA=cuda>=11.8 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 2024-06-26T06:42:00.3821578Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-8=11.11.3.6-1 2024-06-26T06:42:00.3822235Z NV_NVTX_VERSION=11.8.86-1 2024-06-26T06:42:00.3822765Z GITHUB_RUN_NUMBER=219936 2024-06-26T06:42:00.3823087Z TEST_CONFIG=distributed 2024-06-26T06:42:00.3823530Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T06:42:00.3824117Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T06:42:00.3824669Z NV_CUDA_CUDART_DEV_VERSION=11.8.89-1 2024-06-26T06:42:00.3825214Z NV_LIBCUSPARSE_VERSION=11.7.5.86-1 2024-06-26T06:42:00.3825741Z NV_LIBNPP_VERSION=11.8.0.86-1 2024-06-26T06:42:00.3826362Z GITHUB_TRIGGERING_ACTOR=leslie-fang-intel 2024-06-26T06:42:00.3826970Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T06:42:00.3827571Z GITHUB_REF_TYPE=branch 2024-06-26T06:42:00.3827975Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T06:42:00.3828625Z NCCL_VERSION=2.16.2-1 2024-06-26T06:42:00.3828929Z BASE_SHA=4b9c9a9cc9c9283380a011310ba180c105c3dcb9 2024-06-26T06:42:00.3829478Z XLA_CUDA= 2024-06-26T06:42:00.3829974Z HUGGING_FACE_HUB_TOKEN= 2024-06-26T06:42:00.3830334Z *** 2024-06-26T06:42:00.3830662Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T06:42:00.3831174Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T06:42:00.3831618Z GITHUB_ACTIONS=true 2024-06-26T06:42:00.3832011Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:42:00.3832621Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-8=11.8.87-1 2024-06-26T06:42:00.3833433Z NV_LIBNPP_PACKAGE=libnpp-11-8=11.8.0.86-1 2024-06-26T06:42:00.3833853Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:42:00.3834632Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T06:42:00.3835103Z GITHUB_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:42:00.3836457Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/129470/merge 2024-06-26T06:42:00.3837160Z UCC_HOME=/usr 2024-06-26T06:42:00.3837495Z NV_LIBCUBLAS_DEV_VERSION=11.11.3.6-1 2024-06-26T06:42:00.3837980Z VERBOSE_TEST_LOGS=False 2024-06-26T06:42:00.3838410Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T06:42:00.3838981Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-8 2024-06-26T06:42:00.3839565Z GITHUB_REF=refs/pull/129470/merge 2024-06-26T06:42:00.3840072Z NV_CUDA_CUDART_VERSION=11.8.89-1 2024-06-26T06:42:00.3840716Z SHARD_NUMBER=2 2024-06-26T06:42:00.3840903Z GITHUB_REF_PROTECTED=false 2024-06-26T06:42:00.3841339Z HOME=/var/lib/jenkins 2024-06-26T06:42:00.3841963Z GITHUB_API_URL=https://api.github.com 2024-06-26T06:42:00.3842318Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T06:42:00.3843096Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T06:42:00.3843513Z SCCACHE_S3_KEY_PREFIX=pull 2024-06-26T06:42:00.3843930Z CUDA_VERSION=11.8.0 2024-06-26T06:42:00.3844665Z NV_LIBCUBLAS_PACKAGE=libcublas-11-8=11.11.3.6-1 2024-06-26T06:42:00.3845198Z NUM_TEST_SHARDS=3 2024-06-26T06:42:00.3845554Z UCX_HOME=/usr 2024-06-26T06:42:00.3846191Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-11-8=11.8.0-1 2024-06-26T06:42:00.3847554Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3849091Z JOB_NAME=linux-focal-cuda11.8-py3.10-gcc9 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu) 2024-06-26T06:42:00.3850819Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3852053Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T06:42:00.3852879Z GITHUB_EVENT_NAME=pull_request 2024-06-26T06:42:00.3853334Z DASHBOARD_TAG= 2024-06-26T06:42:00.3853707Z GITHUB_RUN_ID=9673645538 2024-06-26T06:42:00.3854381Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-8=11.8.0.86-1 2024-06-26T06:42:00.3855090Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-8 2024-06-26T06:42:00.3856345Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3857506Z GITHUB_ACTOR=leslie-fang-intel 2024-06-26T06:42:00.3857996Z NV_LIBNPP_DEV_VERSION=11.8.0.86-1 2024-06-26T06:42:00.3858468Z PR_NUMBER=129470 2024-06-26T06:42:00.3858846Z GITHUB_RUN_ATTEMPT=1 2024-06-26T06:42:00.3859221Z VALGRIND=ON 2024-06-26T06:42:00.3859587Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T06:42:00.3860144Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T06:42:00.3860817Z TERM=xterm 2024-06-26T06:42:00.3861228Z NV_LIBCUSPARSE_DEV_VERSION=11.7.5.86-1 2024-06-26T06:42:00.3861742Z INSTALLED_VISION=yes 2024-06-26T06:42:00.3862122Z BRANCH=pull/129470 2024-06-26T06:42:00.3862519Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T06:42:00.3862998Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T06:42:00.3863539Z CUDA_PATH=/usr/local/cuda 2024-06-26T06:42:00.3864495Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T06:42:00.3865456Z GITHUB_SERVER_URL=https://github.com 2024-06-26T06:42:00.3866045Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T06:42:00.3866626Z REENABLED_ISSUES= 2024-06-26T06:42:00.3866992Z SHLVL=1 2024-06-26T06:42:00.3867293Z MAX_JOBS=30 2024-06-26T06:42:00.3867679Z NV_CUDA_LIB_VERSION=11.8.0-1 2024-06-26T06:42:00.3868114Z NVARCH=x86_64 2024-06-26T06:42:00.3868461Z GITHUB_ACTOR_ID=53841472 2024-06-26T06:42:00.3869033Z GITHUB_WORKFLOW_SHA=4b51b1a62a63a1add1b3a0f7882f5c7dc66b8f8d 2024-06-26T06:42:00.3869682Z GITHUB_REF_NAME=129470/merge 2024-06-26T06:42:00.3870199Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-8 2024-06-26T06:42:00.3871028Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T06:42:00.3871750Z GITHUB_JOB=test 2024-06-26T06:42:00.3872225Z NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8 2024-06-26T06:42:00.3872931Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T06:42:00.3873584Z NO_TEST_TIMEOUT=False 2024-06-26T06:42:00.3873975Z TD_DISTRIBUTED=False 2024-06-26T06:42:00.3874447Z NV_CUDA_NSIGHT_COMPUTE_VERSION=11.8.0-1 2024-06-26T06:42:00.3874993Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T06:42:00.3875496Z NV_NVPROF_VERSION=11.8.87-1 2024-06-26T06:42:00.3876398Z GITHUB_RETENTION_DAYS=90 2024-06-26T06:42:00.3876729Z OPENSSL_DIR=/opt/openssl 2024-06-26T06:42:00.3877156Z GITHUB_ACTION_REPOSITORY= 2024-06-26T06:42:00.3878334Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:42:00.3879858Z GITHUB_BASE_REF=gh/leslie-fang-intel/126/base 2024-06-26T06:42:00.3880422Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T06:42:00.3880881Z CI=true 2024-06-26T06:42:00.3881265Z NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1 2024-06-26T06:42:00.3881764Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T06:42:00.3882387Z JOB_ID=26688546634 2024-06-26T06:42:00.3882772Z INSTALLED_PROTOBUF=yes 2024-06-26T06:42:00.3883299Z GITHUB_HEAD_REF=gh/leslie-fang-intel/126/head 2024-06-26T06:42:00.3883853Z GITHUB_ACTION_REF= 2024-06-26T06:42:00.3884388Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T06:42:00.3884957Z GITHUB_WORKFLOW=pull 2024-06-26T06:42:00.3885378Z DEBIAN_FRONTEND=noninteractive 2024-06-26T06:42:00.3886519Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_1d576d99-9359-4776-85b1-d093bec6e6f9 2024-06-26T06:42:00.3887539Z NO_TD=False 2024-06-26T06:42:00.3887911Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T06:42:00.3888365Z _=/usr/bin/env 2024-06-26T06:42:00.3888765Z + echo 'Testing pytorch' 2024-06-26T06:42:00.3889189Z Testing pytorch 2024-06-26T06:42:00.3889589Z + export LANG=C.UTF-8 2024-06-26T06:42:00.3889984Z + LANG=C.UTF-8 2024-06-26T06:42:00.3890347Z + PR_NUMBER=129470 2024-06-26T06:42:00.3890767Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-06-26T06:42:00.3891436Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T06:42:00.3892162Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:42:00.3892763Z + [[ distributed == \s\l\o\w ]] 2024-06-26T06:42:00.3893439Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *slow-gradcheck* ]] 2024-06-26T06:42:00.3894250Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-06-26T06:42:00.3894907Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T06:42:00.3895479Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T06:42:00.3896012Z + [[ distributed == *crossref* ]] 2024-06-26T06:42:00.3896643Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *rocm* ]] 2024-06-26T06:42:00.3897353Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *xpu* ]] 2024-06-26T06:42:00.3898102Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:42:00.3898774Z + pip_install --user ninja==1.10.2 2024-06-26T06:42:00.3899416Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T06:42:00.9285822Z Collecting ninja==1.10.2 2024-06-26T06:42:00.9472077Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T06:42:00.9650432Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T06:42:02.1571631Z Installing collected packages: ninja 2024-06-26T06:42:02.1673309Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T06:42:02.1674920Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T06:42:02.1728621Z Successfully installed ninja-1.10.2 2024-06-26T06:42:02.2432509Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:42:02.2435254Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:42:02.2437352Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-06-26T06:42:02.2437979Z + install_tlparse 2024-06-26T06:42:02.2438413Z + pip_install --user tlparse==0.3.7 2024-06-26T06:42:02.2439102Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T06:42:02.7545323Z Collecting tlparse==0.3.7 2024-06-26T06:42:02.7727133Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T06:42:02.7851597Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T06:42:03.9308357Z Installing collected packages: tlparse 2024-06-26T06:42:03.9796536Z Successfully installed tlparse-0.3.7 2024-06-26T06:42:04.0507308Z ++ python -m site --user-base 2024-06-26T06:42:04.0734735Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T06:42:04.0737050Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *asan* ]] 2024-06-26T06:42:04.0737763Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-debug* ]] 2024-06-26T06:42:04.0738533Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:42:04.0739786Z + echo 'We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass' 2024-06-26T06:42:04.0741006Z We are not in debug mode: linux-focal-cuda11.8-py3.10-gcc9. Expect the assertion to pass 2024-06-26T06:42:04.0741861Z + cd test 2024-06-26T06:42:04.0742523Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T06:42:06.1775105Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T06:42:06.1775541Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T06:42:06.1776388Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *-bazel-* ]] 2024-06-26T06:42:06.1777095Z + pushd test 2024-06-26T06:42:06.1778106Z ~/workspace/test ~/workspace 2024-06-26T06:42:06.1779351Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T06:42:08.2233760Z + CUDA_VERSION=11.8 2024-06-26T06:42:08.2234354Z + '[' 11.8 == 12.4 ']' 2024-06-26T06:42:08.2234762Z + ISCUDA124= 2024-06-26T06:42:08.2235118Z + popd 2024-06-26T06:42:08.2235440Z ~/workspace 2024-06-26T06:42:08.2241070Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T06:42:08.2241675Z + [[ distributed == *dynamo_eager* ]] 2024-06-26T06:42:08.2242028Z + [[ distributed == *aot_eager* ]] 2024-06-26T06:42:08.2242662Z + [[ distributed == *aot_inductor* ]] 2024-06-26T06:42:08.2242957Z + [[ distributed == *inductor* ]] 2024-06-26T06:42:08.2243457Z + [[ distributed == *dynamic* ]] 2024-06-26T06:42:08.2243946Z + [[ distributed == *cpu_inductor* ]] 2024-06-26T06:42:08.2244577Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T06:42:08.2278174Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-06-26T06:42:08.2279011Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *-bazel-* ]] 2024-06-26T06:42:08.2281649Z + cd test 2024-06-26T06:42:08.2282187Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T06:42:10.0971946Z PyTorch built with: 2024-06-26T06:42:10.0972748Z - GCC 9.4 2024-06-26T06:42:10.0972966Z - C++ Version: 201703 2024-06-26T06:42:10.0973987Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T06:42:10.0975361Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T06:42:10.0976186Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T06:42:10.0976834Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T06:42:10.0977456Z - NNPACK is enabled 2024-06-26T06:42:10.0977910Z - CPU capability usage: AVX2 2024-06-26T06:42:10.0978419Z - CUDA Runtime 11.8 2024-06-26T06:42:10.0979073Z - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52 2024-06-26T06:42:10.0979753Z - CuDNN 90.1 2024-06-26T06:42:10.0980182Z - Magma 2.6.1 2024-06-26T06:42:10.0989420Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.8, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T06:42:10.0998900Z 2024-06-26T06:42:10.3563521Z + cd test 2024-06-26T06:42:10.3564306Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T06:42:12.1731045Z ATen/Parallel: 2024-06-26T06:42:12.1731846Z at::get_num_threads() : 16 2024-06-26T06:42:12.1732434Z at::get_num_interop_threads() : 16 2024-06-26T06:42:12.1732939Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T06:42:12.1733435Z omp_get_max_threads() : 16 2024-06-26T06:42:12.1734649Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T06:42:12.1735656Z mkl_get_max_threads() : 16 2024-06-26T06:42:12.1736813Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T06:42:12.1737677Z std::thread::hardware_concurrency() : 32 2024-06-26T06:42:12.1738191Z Environment variables: 2024-06-26T06:42:12.1738623Z OMP_NUM_THREADS : [not set] 2024-06-26T06:42:12.1739086Z MKL_NUM_THREADS : [not set] 2024-06-26T06:42:12.1739536Z ATen parallel backend: OpenMP 2024-06-26T06:42:12.1739876Z 2024-06-26T06:42:12.4054420Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *aarch64* ]] 2024-06-26T06:42:12.4055030Z + [[ distributed == *backward* ]] 2024-06-26T06:42:12.4055532Z + [[ distributed == *xla* ]] 2024-06-26T06:42:12.4056006Z + [[ distributed == *executorch* ]] 2024-06-26T06:42:12.4056546Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T06:42:12.4057262Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *libtorch* ]] 2024-06-26T06:42:12.4057903Z + [[ distributed == distributed ]] 2024-06-26T06:42:12.4058388Z + test_distributed 2024-06-26T06:42:12.4058862Z + echo 'Testing distributed python tests' 2024-06-26T06:42:12.4059458Z Testing distributed python tests 2024-06-26T06:42:12.4060237Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2024-06-26T06:42:12.5338463Z /var/lib/jenkins/workspace/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T06:42:12.5339899Z import pkg_resources 2024-06-26T06:42:16.9117873Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json 2024-06-26T06:42:17.0312315Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json 2024-06-26T06:42:17.0777845Z Ignoring disabled issues: [''] 2024-06-26T06:42:17.0906599Z Found test times from artifacts 2024-06-26T06:42:17.1400561Z Found test times from artifacts 2024-06-26T06:42:17.1415727Z Running 25% of tests based on TD 2024-06-26T06:42:17.1539583Z Running parallel tests on 2 processes 2024-06-26T06:42:17.1540855Z Name: tests to run (est. time: 5.82min) 2024-06-26T06:42:17.1542253Z Serial tests (20): 2024-06-26T06:42:17.1543229Z distributed/pipelining/test_transformer 1/1 2024-06-26T06:42:17.1543989Z distributed/_tensor/debug/test_op_coverage 1/1 2024-06-26T06:42:17.1544871Z distributed/_tools/test_mod_tracker 1/1 2024-06-26T06:42:17.1545160Z distributed/_tensor/test_api 1/1 2024-06-26T06:42:17.1546696Z distributed/_shard/sharded_tensor/test_logger 1/1 2024-06-26T06:42:17.1548034Z distributed/test_symmetric_memory 1/1 2024-06-26T06:42:17.1549299Z distributed/_tensor/test_dtensor_compile 1/1 2024-06-26T06:42:17.1550459Z distributed/_composable/test_contract 1/1 2024-06-26T06:42:17.1551441Z distributed/_tensor/test_dtensor 1/1 2024-06-26T06:42:17.1551853Z distributed/_tensor/test_dtensor_ops 1/1 2024-06-26T06:42:17.1552624Z distributed/checkpoint/test_dedup_tensors 1/1 2024-06-26T06:42:17.1553391Z distributed/_tools/test_mem_tracker 1/1 2024-06-26T06:42:17.1553857Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2024-06-26T06:42:17.1554677Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2024-06-26T06:42:17.1555350Z distributed/_tensor/test_tensor_ops 1/1 2024-06-26T06:42:17.1555954Z distributed/pipelining/test_composability 1/1 2024-06-26T06:42:17.1557156Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 2024-06-26T06:42:17.1557794Z distributed/_tensor/test_math_ops 1/1 2024-06-26T06:42:17.1558308Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 2024-06-26T06:42:17.1558993Z distributed/test_compute_comm_reordering 1/1 2024-06-26T06:42:17.1559565Z Parallel tests (0): 2024-06-26T06:42:17.1560131Z Name: excluded (est. time: 75.77min) 2024-06-26T06:42:17.1560513Z Serial tests (56): 2024-06-26T06:42:17.1560960Z distributed/test_distributed_spawn 2/9 2024-06-26T06:42:17.1561544Z distributed/test_distributed_spawn 5/9 2024-06-26T06:42:17.1562103Z distributed/test_distributed_spawn 8/9 2024-06-26T06:42:17.1562823Z distributed/test_c10d_nccl 1/2 2024-06-26T06:42:17.1563373Z distributed/fsdp/test_fsdp_core 1/2 2024-06-26T06:42:17.1563933Z distributed/fsdp/test_fsdp_state_dict 2/2 2024-06-26T06:42:17.1564498Z distributed/test_c10d_ucc 1/1 2024-06-26T06:42:17.1565116Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2024-06-26T06:42:17.1565797Z distributed/checkpoint/test_state_dict 1/1 2024-06-26T06:42:17.1566453Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 2024-06-26T06:42:17.1567079Z distributed/test_pg_wrapper 1/1 2024-06-26T06:42:17.1567635Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-06-26T06:42:17.1568232Z distributed/fsdp/test_fsdp_misc 1/1 2024-06-26T06:42:17.1568777Z distributed/test_functional_api 1/1 2024-06-26T06:42:17.1569344Z distributed/fsdp/test_fsdp_flatten_params 1/1 2024-06-26T06:42:17.1569965Z distributed/fsdp/test_fsdp_comm 1/1 2024-06-26T06:42:17.1570487Z distributed/test_c10d_spawn_nccl 1/1 2024-06-26T06:42:17.1571194Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-06-26T06:42:17.1571634Z distributed/_tensor/test_matrix_ops 1/1 2024-06-26T06:42:17.1572304Z distributed/launcher/test_run 1/1 2024-06-26T06:42:17.1572981Z distributed/elastic/multiprocessing/api_test 1/1 2024-06-26T06:42:17.1573615Z distributed/_composable/test_replicate 1/1 2024-06-26T06:42:17.1574069Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-06-26T06:42:17.1574876Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2024-06-26T06:42:17.1575323Z distributed/_tensor/test_view_ops 1/1 2024-06-26T06:42:17.1575999Z distributed/_tensor/test_utils 1/1 2024-06-26T06:42:17.1576569Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-06-26T06:42:17.1577354Z distributed/_tensor/experimental/test_local_map 1/1 2024-06-26T06:42:17.1578118Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-06-26T06:42:17.1578863Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2024-06-26T06:42:17.1579689Z distributed/elastic/timer/local_timer_example 1/1 2024-06-26T06:42:17.1580272Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-06-26T06:42:17.1581068Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-06-26T06:42:17.1605378Z distributed/fsdp/test_fsdp_pure_fp16 1/1 2024-06-26T06:42:17.1605993Z distributed/fsdp/test_shard_utils 1/1 2024-06-26T06:42:17.1606491Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 2024-06-26T06:42:17.1607274Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2024-06-26T06:42:17.1607944Z distributed/pipelining/test_schedule 1/1 2024-06-26T06:42:17.1608531Z distributed/elastic/test_control_plane 1/1 2024-06-26T06:42:17.1609101Z distributed/fsdp/test_fsdp_uneven 1/1 2024-06-26T06:42:17.1609704Z distributed/checkpoint/test_dtensor_checkpoint 1/1 2024-06-26T06:42:17.1610477Z distributed/checkpoint/e2e/test_pipeline 1/1 2024-06-26T06:42:17.1611109Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2024-06-26T06:42:17.1612062Z distributed/test_data_parallel 1/1 2024-06-26T06:42:17.1612440Z distributed/test_multi_threaded_pg 1/1 2024-06-26T06:42:17.1613019Z distributed/test_nccl 1/1 2024-06-26T06:42:17.1613613Z distributed/_tensor/debug/test_comm_mode 1/1 2024-06-26T06:42:17.1614078Z distributed/fsdp/test_utils 1/1 2024-06-26T06:42:17.1614624Z distributed/elastic/utils/logging_test 1/1 2024-06-26T06:42:17.1615492Z distributed/nn/jit/test_instantiator 1/1 2024-06-26T06:42:17.1615735Z distributed/_tensor/test_xla_integration 1/1 2024-06-26T06:42:17.1616355Z distributed/test_cuda_p2p 1/1 2024-06-26T06:42:17.1617344Z distributed/_composable/fully_shard/test_fully_shard_compile 1/1 2024-06-26T06:42:17.1617559Z distributed/_spmd/test_tracing 1/1 2024-06-26T06:42:17.1618162Z distributed/optim/test_apply_optimizer_in_backward 1/1 2024-06-26T06:42:17.1618788Z distributed/rpc/test_faulty_agent 1/1 2024-06-26T06:42:17.1619512Z Parallel tests (0): 2024-06-26T06:42:17.1620175Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T06:42:17.1621312Z Running distributed/pipelining/test_transformer 1/1 ... [2024-06-26 06:42:17.160960] 2024-06-26T06:42:17.1623757Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:42:17.161361] 2024-06-26T06:42:22.2870149Z 2024-06-26T06:42:22.2872283Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_cb3e6cc41dc3344c_.log 2024-06-26T06:42:22.2874398Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTests::test_ir 2024-06-26T06:42:22.2875368Z 2024-06-26T06:42:22.2877106Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-06-26 06:42:22.287386] 2024-06-26T06:42:22.2883089Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:42:22.287837] 2024-06-26T06:42:26.4108907Z 2024-06-26T06:42:26.4110705Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_13fd67295295643d_.log 2024-06-26T06:42:26.4112964Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-06-26T06:42:26.4113939Z 2024-06-26T06:42:26.4116750Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-06-26 06:42:26.411309] 2024-06-26T06:42:26.4122832Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:42:26.411813] 2024-06-26T06:42:30.2842002Z 2024-06-26T06:42:30.2843814Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_ad193f0ef958d4f9_.log 2024-06-26T06:42:30.2847144Z Running 3 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-06-26T06:42:30.2849367Z 2024-06-26T06:42:30.2849864Z Running distributed/_tensor/test_api 1/1 ... [2024-06-26 06:42:30.284505] 2024-06-26T06:42:30.2864198Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:42:30.285945] 2024-06-26T06:43:04.0752451Z 2024-06-26T06:43:04.0755303Z distributed/_tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_api_1.1_292d8830d6e1a910_.log 2024-06-26T06:43:04.0767098Z Running 8 items in this shard: test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/_tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding 2024-06-26T06:43:04.0773606Z 2024-06-26T06:43:04.0774305Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2024-06-26 06:43:04.075814] 2024-06-26T06:43:04.0776864Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:43:04.076439] 2024-06-26T06:43:07.9990134Z 2024-06-26T06:43:07.9992027Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_b2c738b874384998_.log 2024-06-26T06:43:07.9994492Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger 2024-06-26T06:43:07.9995407Z 2024-06-26T06:43:07.9996254Z Running distributed/test_symmetric_memory 1/1 ... [2024-06-26 06:43:07.999260] 2024-06-26T06:43:08.0001734Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:43:07.999744] 2024-06-26T06:43:11.4210035Z 2024-06-26T06:43:11.4211613Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_9e8293588a96fc7c_.log 2024-06-26T06:43:11.4214318Z Running 2 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent 2024-06-26T06:43:11.4216130Z 2024-06-26T06:43:11.4216707Z Running distributed/_tensor/test_dtensor_compile 1/1 ... [2024-06-26 06:43:11.421307] 2024-06-26T06:43:11.4221733Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:43:11.421740] 2024-06-26T06:43:36.0410811Z 2024-06-26T06:43:36.0416064Z distributed/_tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_compile_1.1_d913cea00f86e5fb_.log 2024-06-26T06:43:36.0433662Z Running 24 items in this shard: test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False, test/distributed/_tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True 2024-06-26T06:43:36.0450664Z 2024-06-26T06:43:36.0451440Z Running distributed/_composable/test_contract 1/1 ... [2024-06-26 06:43:36.041562] 2024-06-26T06:43:36.0453851Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:43:36.042001] 2024-06-26T06:43:39.8651897Z 2024-06-26T06:43:39.8654078Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_5150b453b76c665b_.log 2024-06-26T06:43:39.8657490Z Running 4 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state 2024-06-26T06:43:39.8659608Z 2024-06-26T06:43:39.8660267Z Running distributed/_tensor/test_dtensor 1/1 ... [2024-06-26 06:43:39.865517] 2024-06-26T06:43:39.8664864Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:43:39.866034] 2024-06-26T06:45:51.0719902Z 2024-06-26T06:45:51.0723626Z distributed/_tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_1.1_499eee0b657397c1_.log 2024-06-26T06:45:51.0741015Z Running 30 items in this shard: test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/_tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/_tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/_tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D 2024-06-26T06:45:51.0757469Z 2024-06-26T06:45:51.0758100Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-06-26 06:45:51.072337] 2024-06-26T06:45:51.0760485Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:51.072815] 2024-06-26T06:45:55.8087736Z 2024-06-26T06:45:55.8089878Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_0caed136f3bcf736_.log 2024-06-26T06:45:55.8091699Z Running 0 items in this shard: 2024-06-26T06:45:55.8092209Z 2024-06-26T06:45:55.8094276Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2024-06-26 06:45:55.809086] 2024-06-26T06:45:55.8099934Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:55.809527] 2024-06-26T06:45:59.5822848Z 2024-06-26T06:45:59.5824737Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_44ffd00d022cefd7_.log 2024-06-26T06:45:59.5827455Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards 2024-06-26T06:45:59.5828214Z 2024-06-26T06:45:59.5828733Z Running distributed/_tools/test_mem_tracker 1/1 ... [2024-06-26 06:45:59.582519] 2024-06-26T06:45:59.5833654Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:45:59.582922] 2024-06-26T06:46:05.3587160Z 2024-06-26T06:46:05.3590556Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_9ef127c1cc9764a4_.log 2024-06-26T06:46:05.3595289Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTrackerCUDA::test_cuda_tracker_peak_equvivalence_bsz_2048_n_layers_10_dim_2048_bfloat16_cuda_bfloat16, test/distributed/_tools/test_mem_tracker.py::TestMemTrackerCUDA::test_tracker_attribution_bsz_512_n_layers_3_dim_1024_float32_cuda_float32, test/distributed/_tools/test_mem_tracker.py::TestMemTrackerCUDA::test_tracker_with_activation_checkpointing_bsz_128_n_layers_4_dim_1024_bfloat16_cuda_bfloat16 2024-06-26T06:46:05.3598756Z 2024-06-26T06:46:05.3599182Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2024-06-26 06:46:05.359055] 2024-06-26T06:46:05.3601578Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:46:05.359474] 2024-06-26T06:46:12.8394664Z 2024-06-26T06:46:12.8397079Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_9d924b884e79306a_.log 2024-06-26T06:46:12.8399528Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2024-06-26T06:46:12.8400146Z 2024-06-26T06:46:12.8401075Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2024-06-26 06:46:12.839737] 2024-06-26T06:46:12.8406097Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:46:12.840171] 2024-06-26T06:46:16.8628258Z 2024-06-26T06:46:16.8630303Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_005c3ce85c02241d_.log 2024-06-26T06:46:16.8635333Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2024-06-26T06:46:16.8639492Z 2024-06-26T06:46:16.8640043Z Running distributed/_tensor/test_tensor_ops 1/1 ... [2024-06-26 06:46:16.863051] 2024-06-26T06:46:16.8642450Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:46:16.863488] 2024-06-26T06:48:08.9652165Z 2024-06-26T06:48:08.9655978Z distributed/_tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_tensor_ops_1.1_06ef911c9ec8aa6d_.log 2024-06-26T06:48:08.9670673Z Running 24 items in this shard: test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/_tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum 2024-06-26T06:48:08.9683789Z 2024-06-26T06:48:08.9684347Z Running distributed/pipelining/test_composability 1/1 ... [2024-06-26 06:48:08.965654] 2024-06-26T06:48:08.9686879Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:48:08.966152] 2024-06-26T06:48:12.6276893Z 2024-06-26T06:48:12.6278670Z distributed/pipelining/test_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_composability_1.1_e1dd00b29ffcd76d_.log 2024-06-26T06:48:12.6280184Z 2024-06-26T06:48:12.6283009Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2024-06-26 06:48:12.627936] 2024-06-26T06:48:12.6287983Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:48:12.628331] 2024-06-26T06:48:16.1252775Z 2024-06-26T06:48:16.1255131Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_90bb543c3ae1ecc7_.log 2024-06-26T06:48:16.1257030Z Running 0 items in this shard: 2024-06-26T06:48:16.1257231Z 2024-06-26T06:48:16.1258177Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-06-26 06:48:16.125521] 2024-06-26T06:48:16.1263944Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:48:16.125964] 2024-06-26T06:49:09.2013015Z 2024-06-26T06:49:09.2014864Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_eff07b52951c62e1_.log 2024-06-26T06:49:09.2022033Z Running 10 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk 2024-06-26T06:49:09.2027126Z 2024-06-26T06:49:09.2028059Z Running distributed/tensor/parallel/test_ddp_2d_parallel 1/1 ... [2024-06-26 06:49:09.201507] 2024-06-26T06:49:09.2030666Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_ddp_2d_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:49:09.201913] 2024-06-26T06:49:16.4827565Z 2024-06-26T06:49:16.4829559Z distributed/tensor/parallel/test_ddp_2d_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_222bb1264cedd569_.log 2024-06-26T06:49:16.4832373Z Running 1 items in this shard: test/distributed/tensor/parallel/test_ddp_2d_parallel.py::Test2dParallelIntegration::test_2d_ddp_integration_functionality 2024-06-26T06:49:16.4833885Z 2024-06-26T06:49:16.4834471Z Running distributed/test_compute_comm_reordering 1/1 ... [2024-06-26 06:49:16.483038] 2024-06-26T06:49:16.4839538Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 06:49:16.483470] 2024-06-26T06:49:24.1133613Z 2024-06-26T06:49:24.1135926Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_f71dd6c00eb5889a_.log 2024-06-26T06:49:24.1141672Z Running 6 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms 2024-06-26T06:49:24.1146538Z 2024-06-26T06:49:25.0338753Z 2024-06-26T06:49:25.0339559Z real 7m12.628s 2024-06-26T06:49:25.0340038Z user 18m36.341s 2024-06-26T06:49:25.0340162Z sys 5m50.405s 2024-06-26T06:49:25.0340453Z + assert_git_not_dirty 2024-06-26T06:49:25.0341178Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *rocm* ]] 2024-06-26T06:49:25.0342208Z + [[ linux-focal-cuda11.8-py3.10-gcc9 != *xla* ]] 2024-06-26T06:49:25.0344368Z ++ git status --porcelain 2024-06-26T06:49:25.0345304Z ++ grep -v '?? third_party' 2024-06-26T06:49:27.9139964Z ++ true 2024-06-26T06:49:27.9140881Z + git_status= 2024-06-26T06:49:27.9141756Z + [[ -n '' ]] 2024-06-26T06:49:27.9142430Z + [[ linux-focal-cuda11.8-py3.10-gcc9 == *cuda* ]] 2024-06-26T06:49:27.9143157Z + [[ 2 == 1 ]] 2024-06-26T06:49:27.9143401Z + [[ 2 == 1 ]] 2024-06-26T06:49:27.9143774Z + cleanup_workspace 2024-06-26T06:49:27.9144828Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T06:49:27.9146058Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T06:49:27.9147276Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T06:49:27.9147954Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T06:49:27.9148973Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T06:49:27.9150375Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T06:49:27.9151312Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T06:49:28.4762854Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T06:49:28.4763542Z cat test/**/*_toprint.log || true 2024-06-26T06:49:28.4771914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.4772536Z env: 2024-06-26T06:49:28.4772884Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.4773454Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.4774406Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.4775221Z ##[endgroup] 2024-06-26T06:49:28.4847583Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T06:49:28.4886618Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T06:49:28.4887228Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T06:49:28.4894946Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.4895597Z env: 2024-06-26T06:49:28.4895932Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.4896510Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.4897455Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.4898299Z MONITOR_SCRIPT_PID: 110914 2024-06-26T06:49:28.4898745Z ##[endgroup] 2024-06-26T06:49:28.5141942Z Prepare all required actions 2024-06-26T06:49:28.5142534Z Getting action download info 2024-06-26T06:49:28.6752192Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T06:49:28.8377076Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T06:49:28.8377670Z with: 2024-06-26T06:49:28.8378271Z file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634 2024-06-26T06:49:28.8379049Z s3-bucket: gha-artifacts 2024-06-26T06:49:28.8379451Z env: 2024-06-26T06:49:28.8379785Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.8380345Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.8381281Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.8382095Z ##[endgroup] 2024-06-26T06:49:28.8437620Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T06:49:28.8438446Z # Remove any previous test jsons if they exist 2024-06-26T06:49:28.8439080Z rm -f test-jsons-*.zip 2024-06-26T06:49:28.8439738Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T06:49:28.8448011Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.8448626Z env: 2024-06-26T06:49:28.8448961Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.8449525Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.8450470Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.8451547Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634 2024-06-26T06:49:28.8452517Z ##[endgroup] 2024-06-26T06:49:28.8698078Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T06:49:28.8732928Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T06:49:28.8733634Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T06:49:28.8740666Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T06:49:28.8743465Z adding: test/test-reports/td_exclusions-d08d4aab3e2595093016.json (deflated 86%) 2024-06-26T06:49:28.8744618Z adding: test/.pytorch-slow-tests.json (deflated 77%) 2024-06-26T06:49:28.8753478Z adding: test/.pytorch-disabled-tests.json (deflated 84%) 2024-06-26T06:49:28.8788719Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T06:49:28.8789527Z # Remove any previous test reports if they exist 2024-06-26T06:49:28.8790196Z rm -f test-reports-*.zip 2024-06-26T06:49:28.8790942Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T06:49:28.8799162Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.8799799Z env: 2024-06-26T06:49:28.8800143Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.8800688Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.8801646Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.8802745Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634 2024-06-26T06:49:28.8803493Z ##[endgroup] 2024-06-26T06:49:28.9013187Z adding: test/test-reports/python-pytest/distributed.pipelining.test_transformer/distributed.pipelining.test_transformer-56cea9f58570a082.xml (deflated 39%) 2024-06-26T06:49:28.9016092Z adding: test/test-reports/python-pytest/distributed._tensor.debug.test_op_coverage/distributed._tensor.debug.test_op_coverage-ef1d53e8d6ba3187.xml (deflated 34%) 2024-06-26T06:49:28.9018999Z adding: test/test-reports/python-pytest/distributed._tools.test_mod_tracker/distributed._tools.test_mod_tracker-13621d4f438530b5.xml (deflated 59%) 2024-06-26T06:49:28.9021654Z adding: test/test-reports/python-pytest/distributed._tensor.test_api/distributed._tensor.test_api-354f2a31b54240de.xml (deflated 75%) 2024-06-26T06:49:28.9024238Z adding: test/test-reports/python-pytest/distributed._shard.sharded_tensor.test_logger/distributed._shard.sharded_tensor.test_logger-f7e9a313a36c83e3.xml (deflated 37%) 2024-06-26T06:49:28.9026982Z adding: test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-4048da99cdacf19f.xml (deflated 65%) 2024-06-26T06:49:28.9029221Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_compile/distributed._tensor.test_dtensor_compile-9ef16bc21a79e015.xml (deflated 83%) 2024-06-26T06:49:28.9031459Z adding: test/test-reports/python-pytest/distributed._composable.test_contract/distributed._composable.test_contract-7bb537aec49e88d1.xml (deflated 66%) 2024-06-26T06:49:28.9033603Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor/distributed._tensor.test_dtensor-c625847f475492a3.xml (deflated 85%) 2024-06-26T06:49:28.9036068Z adding: test/test-reports/python-pytest/distributed._tensor.test_dtensor_ops/distributed._tensor.test_dtensor_ops-b07a5965bcf57f69.xml (deflated 28%) 2024-06-26T06:49:28.9038584Z adding: test/test-reports/python-pytest/distributed.checkpoint.test_dedup_tensors/distributed.checkpoint.test_dedup_tensors-cc92a44c28d97380.xml (deflated 37%) 2024-06-26T06:49:28.9040836Z adding: test/test-reports/python-pytest/distributed._tools.test_mem_tracker/distributed._tools.test_mem_tracker-0fc3726fcdb21513.xml (deflated 59%) 2024-06-26T06:49:28.9043075Z adding: test/test-reports/python-pytest/distributed.checkpoint.e2e.test_fsdp_ep/distributed.checkpoint.e2e.test_fsdp_ep-176e779f5c83fb13.xml (deflated 50%) 2024-06-26T06:49:28.9045804Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_state/distributed._composable.fsdp.test_fully_shard_state-3d9b492fe67f3442.xml (deflated 73%) 2024-06-26T06:49:28.9048508Z adding: test/test-reports/python-pytest/distributed._tensor.test_tensor_ops/distributed._tensor.test_tensor_ops-1b5b442f0be96b7e.xml (deflated 84%) 2024-06-26T06:49:28.9050919Z adding: test/test-reports/python-pytest/distributed._composable.fsdp.test_fully_shard_logging/distributed._composable.fsdp.test_fully_shard_logging-ff751cff73d59bd0.xml (deflated 28%) 2024-06-26T06:49:28.9053247Z adding: test/test-reports/python-pytest/distributed._tensor.test_math_ops/distributed._tensor.test_math_ops-a3c73eb952bb87d6.xml (deflated 75%) 2024-06-26T06:49:28.9055582Z adding: test/test-reports/python-pytest/distributed.tensor.parallel.test_ddp_2d_parallel/distributed.tensor.parallel.test_ddp_2d_parallel-d0d18cea2f12a783.xml (deflated 51%) 2024-06-26T06:49:28.9058000Z adding: test/test-reports/python-pytest/distributed.test_compute_comm_reordering/distributed.test_compute_comm_reordering-5bba38fd8e9ecde8.xml (deflated 81%) 2024-06-26T06:49:28.9089119Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T06:49:28.9089918Z # Remove any previous usage logs if they exist 2024-06-26T06:49:28.9090547Z rm -f logs-*.zip 2024-06-26T06:49:28.9091404Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T06:49:28.9092417Z # so check to see if the file exists first 2024-06-26T06:49:28.9093036Z if [ -f 'usage_log.txt' ]; then 2024-06-26T06:49:28.9093702Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T06:49:28.9094317Z fi 2024-06-26T06:49:28.9095002Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T06:49:28.9095483Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T06:49:28.9096113Z fi 2024-06-26T06:49:28.9103429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.9104061Z env: 2024-06-26T06:49:28.9104415Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.9104969Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.9105979Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.9107015Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634 2024-06-26T06:49:28.9107904Z ##[endgroup] 2024-06-26T06:49:28.9152343Z adding: usage_log.txt (deflated 92%) 2024-06-26T06:49:28.9370262Z adding: test/test-reports/distributed.pipelining.test_transformer_1.1_cb3e6cc41dc3344c_.log (deflated 54%) 2024-06-26T06:49:28.9372321Z adding: test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_13fd67295295643d_.log (deflated 55%) 2024-06-26T06:49:28.9373819Z adding: test/test-reports/distributed._tools.test_mod_tracker_1.1_ad193f0ef958d4f9_.log (deflated 61%) 2024-06-26T06:49:28.9376011Z adding: test/test-reports/distributed._tensor.test_api_1.1_292d8830d6e1a910_.log (deflated 89%) 2024-06-26T06:49:28.9377237Z adding: test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_b2c738b874384998_.log (deflated 55%) 2024-06-26T06:49:28.9379641Z adding: test/test-reports/distributed.test_symmetric_memory_1.1_9e8293588a96fc7c_.log (deflated 59%) 2024-06-26T06:49:28.9381459Z adding: test/test-reports/distributed._tensor.test_dtensor_compile_1.1_d913cea00f86e5fb_.log (deflated 82%) 2024-06-26T06:49:28.9383072Z adding: test/test-reports/distributed._composable.test_contract_1.1_5150b453b76c665b_.log (deflated 64%) 2024-06-26T06:49:28.9385333Z adding: test/test-reports/distributed._tensor.test_dtensor_1.1_499eee0b657397c1_.log (deflated 86%) 2024-06-26T06:49:28.9387233Z adding: test/test-reports/distributed._tensor.test_dtensor_ops_1.1_0caed136f3bcf736_.log (deflated 52%) 2024-06-26T06:49:28.9388768Z adding: test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_44ffd00d022cefd7_.log (deflated 52%) 2024-06-26T06:49:28.9390299Z adding: test/test-reports/distributed._tools.test_mem_tracker_1.1_9ef127c1cc9764a4_.log (deflated 65%) 2024-06-26T06:49:28.9392336Z adding: test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_9d924b884e79306a_.log (deflated 62%) 2024-06-26T06:49:28.9393987Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_005c3ce85c02241d_.log (deflated 71%) 2024-06-26T06:49:28.9395593Z adding: test/test-reports/distributed._tensor.test_tensor_ops_1.1_06ef911c9ec8aa6d_.log (deflated 77%) 2024-06-26T06:49:28.9397385Z adding: test/test-reports/distributed.pipelining.test_composability_1.1_e1dd00b29ffcd76d_.log (deflated 3%) 2024-06-26T06:49:28.9399041Z adding: test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_90bb543c3ae1ecc7_.log (deflated 53%) 2024-06-26T06:49:28.9400629Z adding: test/test-reports/distributed._tensor.test_math_ops_1.1_eff07b52951c62e1_.log (deflated 72%) 2024-06-26T06:49:28.9402213Z adding: test/test-reports/distributed.tensor.parallel.test_ddp_2d_parallel_1.1_222bb1264cedd569_.log (deflated 53%) 2024-06-26T06:49:28.9403845Z adding: test/test-reports/distributed.test_compute_comm_reordering_1.1_f71dd6c00eb5889a_.log (deflated 73%) 2024-06-26T06:49:28.9434375Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T06:49:28.9435293Z # Remove any previous debugging artifacts if they exist 2024-06-26T06:49:28.9436004Z rm -f debug-*.zip 2024-06-26T06:49:28.9436742Z if [ -d 'test/debug' ]; then 2024-06-26T06:49:28.9437373Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T06:49:28.9437979Z fi 2024-06-26T06:49:28.9445368Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:28.9445978Z env: 2024-06-26T06:49:28.9446322Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.9446893Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.9447820Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.9448912Z FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634 2024-06-26T06:49:28.9449673Z ##[endgroup] 2024-06-26T06:49:28.9549493Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:49:28.9550027Z with: 2024-06-26T06:49:28.9550370Z s3-bucket: gha-artifacts 2024-06-26T06:49:28.9550899Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:28.9551630Z retention-days: 14 2024-06-26T06:49:28.9552068Z if-no-files-found: warn 2024-06-26T06:49:28.9552515Z path: test-jsons-*.zip 2024-06-26T06:49:28.9552916Z name: artifact 2024-06-26T06:49:28.9553288Z region: us-east-1 2024-06-26T06:49:28.9553655Z env: 2024-06-26T06:49:28.9553969Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:28.9554528Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:28.9555465Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:28.9556270Z ##[endgroup] 2024-06-26T06:49:29.3991153Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:49:29.3992757Z With the provided path, there will be 1 file uploaded 2024-06-26T06:49:29.3993542Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:29.4040621Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:29.5474278Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:29.5638905Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:49:29.5639448Z with: 2024-06-26T06:49:29.5639789Z s3-bucket: gha-artifacts 2024-06-26T06:49:29.5640321Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:29.5640891Z retention-days: 14 2024-06-26T06:49:29.5641307Z if-no-files-found: error 2024-06-26T06:49:29.5641757Z path: test-reports-*.zip 2024-06-26T06:49:29.5642170Z name: artifact 2024-06-26T06:49:29.5642543Z region: us-east-1 2024-06-26T06:49:29.5642914Z env: 2024-06-26T06:49:29.5643228Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:29.5644008Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:29.5644944Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:29.5645745Z ##[endgroup] 2024-06-26T06:49:30.0135532Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:49:30.0137075Z With the provided path, there will be 1 file uploaded 2024-06-26T06:49:30.0137714Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:30.0185300Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:30.1886459Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:30.2050396Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:49:30.2050930Z with: 2024-06-26T06:49:30.2051277Z s3-bucket: gha-artifacts 2024-06-26T06:49:30.2051810Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:30.2052380Z retention-days: 14 2024-06-26T06:49:30.2052818Z if-no-files-found: ignore 2024-06-26T06:49:30.2053269Z path: logs-*.zip 2024-06-26T06:49:30.2053631Z name: artifact 2024-06-26T06:49:30.2054006Z region: us-east-1 2024-06-26T06:49:30.2054378Z env: 2024-06-26T06:49:30.2054691Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:30.2055253Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:30.2056215Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:30.2057022Z ##[endgroup] 2024-06-26T06:49:30.6444565Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T06:49:30.6445837Z With the provided path, there will be 1 file uploaded 2024-06-26T06:49:30.6446480Z Uploading to s3 prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:30.6493418Z Starting upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:30.8061338Z Finished upload of logs-test-distributed-2-3-linux.8xlarge.nvidia.gpu_26688546634.zip 2024-06-26T06:49:30.8228639Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T06:49:30.8229182Z with: 2024-06-26T06:49:30.8229524Z s3-bucket: gha-artifacts 2024-06-26T06:49:30.8230057Z s3-prefix: pytorch/pytorch/9673645538/1/artifact 2024-06-26T06:49:30.8230632Z retention-days: 14 2024-06-26T06:49:30.8231055Z if-no-files-found: ignore 2024-06-26T06:49:30.8231691Z path: debug-*.zip 2024-06-26T06:49:30.8232080Z name: artifact 2024-06-26T06:49:30.8232456Z region: us-east-1 2024-06-26T06:49:30.8232824Z env: 2024-06-26T06:49:30.8233137Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:30.8233701Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:30.8234642Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:30.8235452Z ##[endgroup] 2024-06-26T06:49:31.2579204Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T06:49:31.2741814Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T06:49:31.2742466Z # shellcheck disable=SC2156 2024-06-26T06:49:31.2743479Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T06:49:31.2752073Z shell: /usr/bin/bash -e {0} 2024-06-26T06:49:31.2752514Z env: 2024-06-26T06:49:31.2752853Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:31.2753397Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:31.2754333Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:31.2755155Z ##[endgroup] 2024-06-26T06:49:31.5762551Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T06:49:31.5763269Z with: 2024-06-26T06:49:31.5763582Z env: 2024-06-26T06:49:31.5763917Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:31.5764469Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:31.5765409Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:31.5766447Z ##[endgroup] 2024-06-26T06:49:31.5880774Z ##[group]Run set -eou pipefail 2024-06-26T06:49:31.5881121Z set -eou pipefail 2024-06-26T06:49:31.5881667Z  2024-06-26T06:49:31.5882204Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T06:49:31.5883003Z for _ in $(seq 1440); do 2024-06-26T06:49:31.5883595Z  # Break if no ssh session exists anymore 2024-06-26T06:49:31.5884223Z  if [ "$(who)" = "" ]; then 2024-06-26T06:49:31.5884719Z  break 2024-06-26T06:49:31.5885168Z  fi 2024-06-26T06:49:31.5885539Z  echo "." 2024-06-26T06:49:31.5885954Z  sleep 5 2024-06-26T06:49:31.5886349Z done 2024-06-26T06:49:31.5894343Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:31.5894978Z env: 2024-06-26T06:49:31.5895325Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:31.5895879Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:31.5896839Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:31.5897665Z ##[endgroup] 2024-06-26T06:49:31.5924174Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T06:49:31.5976044Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:49:31.5977027Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T06:49:31.5977788Z # shellcheck disable=SC2046 2024-06-26T06:49:31.5978377Z docker stop $(docker ps -q) || true 2024-06-26T06:49:31.5978972Z # Prune all of the docker images 2024-06-26T06:49:31.5979538Z docker system prune -af 2024-06-26T06:49:31.5987223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:31.5987837Z env: 2024-06-26T06:49:31.5988186Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:49:31.5988764Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T06:49:31.5989703Z DOCKER_CONTAINER_ID: 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:31.5990527Z ##[endgroup] 2024-06-26T06:49:31.9525917Z 30e36954f97c 2024-06-26T06:49:32.5192576Z Deleted Containers: 2024-06-26T06:49:32.5193253Z 30e36954f97c8c377dd076485e8aafd9fa7bca92b6fda1cd88a52317f4a3105e 2024-06-26T06:49:32.5194093Z 2024-06-26T06:49:39.2477151Z Deleted Images: 2024-06-26T06:49:39.2479151Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:49:39.2481544Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda11.8-cudnn9-py3-gcc9@sha256:493391c3b293fbc39c8ca4208a5b70ab17031329cb9c855a641ab5a208c52fe6 2024-06-26T06:49:39.2483375Z deleted: sha256:1fde65d68e1ba467b7c461d097349da1965328db5e6e5252ccb25702fcc3b83e 2024-06-26T06:49:39.2484652Z deleted: sha256:d7f4fd7abbe071a8cbdda2bae83a151d7bcde19abdef895ea3019cdeadd13a18 2024-06-26T06:49:39.2485672Z deleted: sha256:ee085d46602902c38ef79d90a27d337dae20947f1eeb9c4fccabd1a92e01e58e 2024-06-26T06:49:39.2486657Z deleted: sha256:e860cd33597e4e0f5736fd4d6e2f4702fa16a9940209ad8e214ca7081f44a287 2024-06-26T06:49:39.2487667Z deleted: sha256:a18d95c977fe0d23a7007fdc3ee2379aa8a8c2d42e59e8ef95c3e677d2578fcd 2024-06-26T06:49:39.2488739Z deleted: sha256:52e65c84df1c54be8f442eee573bca8c158f9fc906b5fbee0ff6953553aea0e4 2024-06-26T06:49:39.2489830Z deleted: sha256:bb7adddb154f86f311acf257b7a21b39fa5702dda66249163efd1ae6729d3782 2024-06-26T06:49:39.2491174Z deleted: sha256:1997f0e776385e3c40db4b42b13e2e9006fb6204d0d1dcf32f29db95c117372a 2024-06-26T06:49:39.2492251Z deleted: sha256:cad19b1260ec60216e3d0b105b8f813b992973a7c792a45774531d346384edc1 2024-06-26T06:49:39.2493321Z deleted: sha256:ecc44fe33e9088e5eb858752a97ca6e2d729a64cd42d6a5225b567f8da1ec73f 2024-06-26T06:49:39.2494419Z deleted: sha256:f352c3ff12d96e8bd1de0ee0fddddf2df4aaadbad5433589930269739439c345 2024-06-26T06:49:39.2495720Z deleted: sha256:e6f2acebe916afc5f24b61dd806cd8194326c8dd940a061c8b6acd11a1af4683 2024-06-26T06:49:39.2496777Z deleted: sha256:c94a6342cc6b933354589534bdead86649bbc1464c794cc9d21577fe82437b3f 2024-06-26T06:49:39.2497846Z deleted: sha256:73c84e0e40a121ef226ee59ad4e41c40f6f8e3dd3aa6e40b580b2342181ff842 2024-06-26T06:49:39.2498936Z deleted: sha256:5f650bdb1edb4afa0f22a8c2de8706ea07e360c55298d0bac711f5f4dd750406 2024-06-26T06:49:39.2499999Z deleted: sha256:e3c78b112f621be151f2591f8a82ab325ffedd19be99e3b478c3017e051b5355 2024-06-26T06:49:39.2501074Z deleted: sha256:e058ad0c4f09075c613e2fb7abd0ef0e7465733d555b7778902ed13846dfacb6 2024-06-26T06:49:39.2502139Z deleted: sha256:2b91783cec90ca4caa17a1a03b734c733623e7f8f185334d5bbb21fe9a0f1e5d 2024-06-26T06:49:39.2503200Z deleted: sha256:014aca11d6f33d59cda58c68a53f4a22dadeedb3f6b487fa66f556fcec903607 2024-06-26T06:49:39.2504295Z deleted: sha256:32b4d98aa83b992ac9dd109c618da5a02af41078f6d5bd2a174b69f208fedb0c 2024-06-26T06:49:39.2505390Z deleted: sha256:f4dad79cdbcd5b49f4b3526d3f3c0dae331ddad5ee9be01b21d289ce2be060c0 2024-06-26T06:49:39.2506509Z deleted: sha256:755dbc164caaa04e7f3ac761fcfa41914cb61bbefd288772a7fae92e6429c790 2024-06-26T06:49:39.2507584Z deleted: sha256:3bb6a2f1711ddd563f93bd2b39fb20a667aad822781c1586c9d04a049ffff691 2024-06-26T06:49:39.2508710Z deleted: sha256:25731f18356bbce9f9cf003c08edb416482d8c85becb79fd1af3b83806d147b0 2024-06-26T06:49:39.2509800Z deleted: sha256:6f3c5cbd50ad5b397dcac877079fae290058a3af2bb7c5390c202f221fe1d8f2 2024-06-26T06:49:39.2510950Z deleted: sha256:ea584ac8474876e97af40f7148363e9a6df9e84b52540b61d3488f76fe3fda19 2024-06-26T06:49:39.2512056Z deleted: sha256:c99897453df1a53f8d8caf80177dfabc306d7ad968db5e62b0084b2fcc38ff9b 2024-06-26T06:49:39.2513003Z deleted: sha256:0fa44105790cb410839a6f2cc4410c3be63657d803ef603c9ef7a4b355d5f380 2024-06-26T06:49:39.2514067Z deleted: sha256:2fd5ab6ef7d9cb567b70dddfa5a1a5e65e324446bb14f6feba5599f4e730b35b 2024-06-26T06:49:39.2515139Z deleted: sha256:b99692dfc620bf7adfe66568b6fe1f742be148802aa2d7c6028593bef0210571 2024-06-26T06:49:39.2516517Z deleted: sha256:21daffc134560cc20e34bee6ed4684f5c0de46241501e2a7a89cb7bf546d4227 2024-06-26T06:49:39.2517884Z deleted: sha256:819249fb48326412a787749ddc95957655df6636f2e0f1c5265bf6881c7111e9 2024-06-26T06:49:39.2518664Z deleted: sha256:610d96248b1aa37c8d0b05fdf39c38fa1d491f505e454525de8f8ac3b0bd48f4 2024-06-26T06:49:39.2519736Z deleted: sha256:22f47f538c346eb9b1142a65fdc01f85cd5e10621a8a28cd8aae7e205880bada 2024-06-26T06:49:39.2520813Z deleted: sha256:09c6cd1d8ea04ed3b74420884bf62eaed12a455f041633ebec3a2cf41468e057 2024-06-26T06:49:39.2521890Z deleted: sha256:2f81f71a0bfa140425b1f34efbb9e8a09ce133c20bce173876ae51ec55c53767 2024-06-26T06:49:39.2522948Z deleted: sha256:7083f25055a8577c549ee03196e895f72ec1e26bb509cf4f531e68a9458ec9ff 2024-06-26T06:49:39.2524081Z deleted: sha256:2d353f3abd6ba69b7052291fab5b672c6a31571f9ca14eb579f1324928be9a17 2024-06-26T06:49:39.2525310Z deleted: sha256:f8cb1af08a381526d68d11ccadf89749f855c9f643bced1fa51c146109d894fb 2024-06-26T06:49:39.2526218Z deleted: sha256:538faaa902f68d805ad2a2dc8511f8fea33499e28ed8dd54a2e885b6069b94ed 2024-06-26T06:49:39.2527314Z deleted: sha256:6c0cabeb2da5f02c2781f2f0ca3706fdb328da082a3decdbd8f82ae596f513f3 2024-06-26T06:49:39.2528395Z deleted: sha256:5722d912cb634e82f4826ea4170754898a77efee6df3b6ea282ba78b4aaa43d2 2024-06-26T06:49:39.2529467Z deleted: sha256:dc0beaa37049a8d70d2e4da52db402c3876391d86720df30a9b686ae6ac5cb24 2024-06-26T06:49:39.2530516Z deleted: sha256:a2793888ee929a46917ad906e46fa352b0b00c68298d9c47b995cdd3715f0b44 2024-06-26T06:49:39.2531739Z deleted: sha256:fccc9e0f8fd9de6f71e5b000f1d79435e896c89cd8554c718604b56c993d93ed 2024-06-26T06:49:39.2532841Z deleted: sha256:99df2ce205527dab671443417aa5c24070144ad78f7597f7b01a2fbdc75db4a0 2024-06-26T06:49:39.2533881Z deleted: sha256:ab3da06e7304132443910308818f60abaee5e6aaea7aadc74ff905e3071786a6 2024-06-26T06:49:39.2534958Z deleted: sha256:c43d30eed3c50d7f00e7d83ef78a94dcdf07c763c37d83f9185e72377d1a5f20 2024-06-26T06:49:39.2536031Z deleted: sha256:8f2d8265fe3b438415439e09c19ba6a68b4150eaa562224d7e8ac2710dc2eb14 2024-06-26T06:49:39.2537196Z deleted: sha256:28f2bc432836b7a4424f890747021c36c0b33371c62b6fdb714244d6b90203aa 2024-06-26T06:49:39.2538241Z deleted: sha256:f6286286a1dbe990ec2f0b147e01827e45b747cdb10499b6102daa2b25922727 2024-06-26T06:49:39.2539318Z deleted: sha256:c6e97d8ca374fb6c9ca255b80d6c36af85de5fd66e448ea23491ad9958f19b66 2024-06-26T06:49:39.2540416Z deleted: sha256:b88b1ab78ee5abbcbfd6500e00629d125d67a3037ce62eddb8e0866ffef82fe9 2024-06-26T06:49:39.2541470Z deleted: sha256:1512ffb3610f68874de48684966014158e1bdd95d80f21f97570bd10e88b4ed4 2024-06-26T06:49:39.2542530Z deleted: sha256:fd6f5133b44e40b8da2ecb4666c25328e71d7c29daf9247f740daba07621adc2 2024-06-26T06:49:39.2543596Z deleted: sha256:f9f206b62ad20270a766d66358f0a8cda72f3819c9805b5fe5e929be08487da6 2024-06-26T06:49:39.2544630Z deleted: sha256:f73b05159be211077e8f3d9ff623e66c1c4735d4450aeb21229f951235dc0adb 2024-06-26T06:49:39.2545684Z deleted: sha256:ef71994fba318c56792ce905d971dc876347c60961aa4fd4a2bdf22f026df5ba 2024-06-26T06:49:39.2546757Z deleted: sha256:2c70714aac7394ad9e1e42a55b728f30f38fbd46e545b118bf0d625f4b1eec4e 2024-06-26T06:49:39.2547825Z deleted: sha256:49adb4771e439b38cd6de3c8ef02299fc5672fdfe39ed6a08ecc41893a00d16f 2024-06-26T06:49:39.2548910Z deleted: sha256:fa77ab1294ff3ad236f4a809e697cb4b9b150b11fc7641ea3c508198e0059582 2024-06-26T06:49:39.2549975Z deleted: sha256:06332c1fdd9151f70f43605c3bd5a2967db0f7d03398d97147e645c6ad1034b6 2024-06-26T06:49:39.2551058Z deleted: sha256:c217de6cbf77b426bac257f8f17b0ff3ca3a514a5511deaa7bdfc6f42d405375 2024-06-26T06:49:39.2552131Z deleted: sha256:1248854931e90cf6124b5f6d68fe5bcb43a328f6c246cad949ef775809a2dd40 2024-06-26T06:49:39.2553192Z deleted: sha256:903fa51ff1654718fa673df7ccde5b99f43fc40cd5d5c541f674554c68b9dd00 2024-06-26T06:49:39.2554255Z deleted: sha256:80c567c8b1b790b0859417382b69fb69bd01a15552aadf1ab1aba69882cba1d4 2024-06-26T06:49:39.2555319Z deleted: sha256:8fc1dd27a3b4d2bbf11127923b7bd5063a9faacb77ed389a2eb18bc7d9732abc 2024-06-26T06:49:39.2556712Z deleted: sha256:08147e1d91de147ab86d7cab3a87eac310ab1d06d857b2506890fd3ea12fbf6c 2024-06-26T06:49:39.2557791Z deleted: sha256:89edf8c906678c217b9a88160e1561991f53909c3fd27a4a5e1f0ca0f7ff7888 2024-06-26T06:49:39.2558857Z deleted: sha256:d5ba67f8f9b0c2b6da06d2d5d9d5fedc9f17cbe1a5e13a1a4033a4619f4528ba 2024-06-26T06:49:39.2559930Z deleted: sha256:5d61311e1a293c0c9642e34871719525dfeec7028e4b45e08344a469e3fe24bf 2024-06-26T06:49:39.2560987Z deleted: sha256:b89bb70f745a94c0edd5e974aee294e160735f76fd666a1a06a0b9b1be3caf63 2024-06-26T06:49:39.2562119Z deleted: sha256:9e5477f1f21a38c3dfe21a56a7d77d1b1a42453aed3e11729afcbb9d5731770b 2024-06-26T06:49:39.2563195Z deleted: sha256:b76f4af91cffe2bb7b3f09960f5bdbb26f741419defda65c451c4407fc5b0a92 2024-06-26T06:49:39.2564285Z deleted: sha256:3f8748a00ebd394c0e9d9d481972d764dff58896841ef0150b3f7efaac0f4a4a 2024-06-26T06:49:39.2565346Z deleted: sha256:13989c771e3ba4d40f618344e55e5cb7062a35041b5efd69e2f740ba819f46ee 2024-06-26T06:49:39.2566401Z deleted: sha256:9ebb27837f6c67ad8c17e9af4e63c008faae03fa226c3641dd8bbbfeefb106c4 2024-06-26T06:49:39.2567477Z deleted: sha256:8c16907d735950f8093582bc3cf392b70e002456bf9c4224de5a5b86b78b3ea2 2024-06-26T06:49:39.2568530Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T06:49:39.2569173Z 2024-06-26T06:49:39.2569377Z Total reclaimed space: 25.98GB 2024-06-26T06:49:39.2655583Z Post job cleanup. 2024-06-26T06:49:39.2721975Z Post job cleanup. 2024-06-26T06:49:39.3892981Z [command]/usr/bin/git version 2024-06-26T06:49:39.3946452Z git version 2.40.1 2024-06-26T06:49:39.4009427Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b2f38e71-79cf-4ad5-802c-28364b6c4943' before making global git config changes 2024-06-26T06:49:39.4010757Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:49:39.4016949Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:49:39.4059627Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:49:39.4097166Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:49:39.4412421Z Entering 'android/libs/fbjni' 2024-06-26T06:49:39.4464042Z Entering 'third_party/FP16' 2024-06-26T06:49:39.4514653Z Entering 'third_party/FXdiv' 2024-06-26T06:49:39.4565479Z Entering 'third_party/NNPACK' 2024-06-26T06:49:39.4617556Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:49:39.4668359Z Entering 'third_party/XNNPACK' 2024-06-26T06:49:39.4739899Z Entering 'third_party/benchmark' 2024-06-26T06:49:39.4791560Z Entering 'third_party/cpp-httplib' 2024-06-26T06:49:39.4843309Z Entering 'third_party/cpuinfo' 2024-06-26T06:49:39.4894257Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:49:39.4947629Z Entering 'third_party/cutlass' 2024-06-26T06:49:39.5008689Z Entering 'third_party/eigen' 2024-06-26T06:49:39.5062338Z Entering 'third_party/fbgemm' 2024-06-26T06:49:39.5115368Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:49:39.5164997Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:49:39.5215707Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:49:39.5274008Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:49:39.5325163Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:49:39.5375793Z Entering 'third_party/flatbuffers' 2024-06-26T06:49:39.5431295Z Entering 'third_party/fmt' 2024-06-26T06:49:39.5484096Z Entering 'third_party/foxi' 2024-06-26T06:49:39.5534313Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:49:39.5587522Z Entering 'third_party/gloo' 2024-06-26T06:49:39.5640754Z Entering 'third_party/googletest' 2024-06-26T06:49:39.5691051Z Entering 'third_party/ideep' 2024-06-26T06:49:39.5742291Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:49:39.5800775Z Entering 'third_party/ittapi' 2024-06-26T06:49:39.5852016Z Entering 'third_party/kineto' 2024-06-26T06:49:39.5902432Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:49:39.5953357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:49:39.6007048Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:49:39.6058814Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:49:39.6110287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:49:39.6161278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:49:39.6213328Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:49:39.6264301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:49:39.6315370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:49:39.6367736Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:49:39.6420430Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:49:39.6471240Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:49:39.6524194Z Entering 'third_party/mimalloc' 2024-06-26T06:49:39.6576647Z Entering 'third_party/nccl/nccl' 2024-06-26T06:49:39.6628360Z Entering 'third_party/nlohmann' 2024-06-26T06:49:39.6681298Z Entering 'third_party/onnx' 2024-06-26T06:49:39.6751330Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:49:39.6803306Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:49:39.6856606Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:49:39.6909812Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:49:39.6960328Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:49:39.7011896Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:49:39.7062709Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:49:39.7113871Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:49:39.7164279Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:49:39.7213736Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:49:39.7262666Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:49:39.7315041Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:49:39.7368059Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:49:39.7441066Z Entering 'third_party/pocketfft' 2024-06-26T06:49:39.7492537Z Entering 'third_party/protobuf' 2024-06-26T06:49:39.7546927Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:49:39.7597496Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:49:39.7648535Z Entering 'third_party/psimd' 2024-06-26T06:49:39.7699169Z Entering 'third_party/pthreadpool' 2024-06-26T06:49:39.7750679Z Entering 'third_party/pybind11' 2024-06-26T06:49:39.7801922Z Entering 'third_party/python-peachpy' 2024-06-26T06:49:39.7852960Z Entering 'third_party/sleef' 2024-06-26T06:49:39.7904795Z Entering 'third_party/tensorpipe' 2024-06-26T06:49:39.7955646Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:49:39.8007644Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:49:39.8058018Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:49:39.8108329Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:49:39.8157754Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:49:39.8228189Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:49:39.8254713Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8266339Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T06:49:39.8302581Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:49:39.8612994Z Entering 'android/libs/fbjni' 2024-06-26T06:49:39.8646879Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8681751Z Entering 'third_party/FP16' 2024-06-26T06:49:39.8714398Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8748698Z Entering 'third_party/FXdiv' 2024-06-26T06:49:39.8782052Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8816636Z Entering 'third_party/NNPACK' 2024-06-26T06:49:39.8849720Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8883724Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T06:49:39.8916517Z http.https://github.com/.extraheader 2024-06-26T06:49:39.8951147Z Entering 'third_party/XNNPACK' 2024-06-26T06:49:39.8984441Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9037387Z Entering 'third_party/benchmark' 2024-06-26T06:49:39.9070132Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9104226Z Entering 'third_party/cpp-httplib' 2024-06-26T06:49:39.9138082Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9172291Z Entering 'third_party/cpuinfo' 2024-06-26T06:49:39.9206234Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9240388Z Entering 'third_party/cudnn_frontend' 2024-06-26T06:49:39.9273237Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9306783Z Entering 'third_party/cutlass' 2024-06-26T06:49:39.9340441Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9382498Z Entering 'third_party/eigen' 2024-06-26T06:49:39.9415382Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9451318Z Entering 'third_party/fbgemm' 2024-06-26T06:49:39.9484113Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9517981Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T06:49:39.9550606Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9585008Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T06:49:39.9616595Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9651518Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T06:49:39.9682921Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9723767Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T06:49:39.9755967Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9790312Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T06:49:39.9822399Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9857231Z Entering 'third_party/flatbuffers' 2024-06-26T06:49:39.9890979Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9927433Z Entering 'third_party/fmt' 2024-06-26T06:49:39.9961128Z http.https://github.com/.extraheader 2024-06-26T06:49:39.9994018Z Entering 'third_party/foxi' 2024-06-26T06:49:40.0026658Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0060767Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T06:49:40.0094015Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0130624Z Entering 'third_party/gloo' 2024-06-26T06:49:40.0163696Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0197218Z Entering 'third_party/googletest' 2024-06-26T06:49:40.0229046Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0263674Z Entering 'third_party/ideep' 2024-06-26T06:49:40.0297066Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0329463Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T06:49:40.0361483Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0403347Z Entering 'third_party/ittapi' 2024-06-26T06:49:40.0436023Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0470028Z Entering 'third_party/kineto' 2024-06-26T06:49:40.0504792Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0538809Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T06:49:40.0572079Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0606719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T06:49:40.0639643Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0675354Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T06:49:40.0707681Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0742456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T06:49:40.0775939Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0810183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T06:49:40.0843409Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0877077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T06:49:40.0909207Z http.https://github.com/.extraheader 2024-06-26T06:49:40.0944796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T06:49:40.0978078Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1012788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T06:49:40.1044897Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1079156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T06:49:40.1111087Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1146686Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T06:49:40.1178972Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1214111Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T06:49:40.1246613Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1280437Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T06:49:40.1311920Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1347052Z Entering 'third_party/mimalloc' 2024-06-26T06:49:40.1379944Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1413645Z Entering 'third_party/nccl/nccl' 2024-06-26T06:49:40.1446973Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1481049Z Entering 'third_party/nlohmann' 2024-06-26T06:49:40.1513448Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1548438Z Entering 'third_party/onnx' 2024-06-26T06:49:40.1581159Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1633042Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T06:49:40.1665567Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1699923Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T06:49:40.1732502Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1768605Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T06:49:40.1802228Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1837605Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T06:49:40.1869742Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1904099Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T06:49:40.1936312Z http.https://github.com/.extraheader 2024-06-26T06:49:40.1971181Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T06:49:40.2003290Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2037534Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T06:49:40.2069267Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2104882Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T06:49:40.2136779Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2170438Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T06:49:40.2203017Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2235450Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T06:49:40.2267433Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2300590Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T06:49:40.2333194Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2370657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T06:49:40.2402913Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2438410Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T06:49:40.2471344Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2528166Z Entering 'third_party/pocketfft' 2024-06-26T06:49:40.2560971Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2595165Z Entering 'third_party/protobuf' 2024-06-26T06:49:40.2627616Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2665084Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T06:49:40.2698056Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2732639Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T06:49:40.2764619Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2801284Z Entering 'third_party/psimd' 2024-06-26T06:49:40.2834064Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2867796Z Entering 'third_party/pthreadpool' 2024-06-26T06:49:40.2900915Z http.https://github.com/.extraheader 2024-06-26T06:49:40.2934712Z Entering 'third_party/pybind11' 2024-06-26T06:49:40.2967647Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3001479Z Entering 'third_party/python-peachpy' 2024-06-26T06:49:40.3034026Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3067863Z Entering 'third_party/sleef' 2024-06-26T06:49:40.3102564Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3136557Z Entering 'third_party/tensorpipe' 2024-06-26T06:49:40.3169840Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3203916Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T06:49:40.3235952Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3269599Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T06:49:40.3301510Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3335334Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T06:49:40.3367804Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3402464Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T06:49:40.3434132Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3466934Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T06:49:40.3498806Z http.https://github.com/.extraheader 2024-06-26T06:49:40.3638319Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T06:49:40.3662482Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:49:40.3669992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:49:40.3670650Z ##[endgroup] 2024-06-26T06:49:41.1101033Z Cleaning up orphan processes