2024-06-26T07:51:13.4557827Z Current runner version: '2.317.0' 2024-06-26T07:51:13.4562626Z Runner name: 'GPU9449' 2024-06-26T07:51:13.4563181Z Runner group name: 'linux.rocm.gpu.group' 2024-06-26T07:51:13.4563838Z Machine name: 'GPU9449' 2024-06-26T07:51:13.4565885Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T07:51:13.4567388Z Contents: read 2024-06-26T07:51:13.4567934Z Metadata: read 2024-06-26T07:51:13.4568248Z ##[endgroup] 2024-06-26T07:51:13.4571190Z Secret source: Actions 2024-06-26T07:51:13.4571809Z Prepare workflow directory 2024-06-26T07:51:13.8762454Z Prepare all required actions 2024-06-26T07:51:13.8943983Z Getting action download info 2024-06-26T07:51:14.1573745Z Download action repository 'pytorch/pytorch@main' (SHA:ead97ee4863253b37e03e4d14a71f19d435db4cf) 2024-06-26T07:51:20.4009302Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-06-26T07:51:21.4295836Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-06-26T07:51:22.4367272Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T07:51:23.7172217Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T07:51:24.6995923Z Getting action download info 2024-06-26T07:51:24.8562557Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T07:51:25.8931152Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T07:51:25.8933119Z ##[group] Inputs 2024-06-26T07:51:25.8933447Z build-environment: linux-focal-rocm6.1-py3.8 2024-06-26T07:51:25.8934496Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-06-26T07:51:25.8935855Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:51:25.8936531Z sync-tag: 2024-06-26T07:51:25.8937157Z timeout-minutes: 300 2024-06-26T07:51:25.8937954Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-06-26T07:51:25.8938795Z ##[endgroup] 2024-06-26T07:51:25.8939392Z Complete job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:51:26.0219803Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T07:51:26.0220256Z with: 2024-06-26T07:51:26.0220472Z no-sudo: true 2024-06-26T07:51:26.0220708Z submodules: recursive 2024-06-26T07:51:26.0220951Z fetch-depth: 0 2024-06-26T07:51:26.0221165Z env: 2024-06-26T07:51:26.0221373Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:51:26.0221630Z ##[endgroup] 2024-06-26T07:51:26.0407561Z ##[group]Run retry () { 2024-06-26T07:51:26.0407972Z retry () { 2024-06-26T07:51:26.0408356Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T07:51:26.0408747Z } 2024-06-26T07:51:26.0408988Z echo "${GITHUB_WORKSPACE}" 2024-06-26T07:51:26.0409360Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T07:51:26.0409767Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:51:26.0410222Z else 2024-06-26T07:51:26.0410596Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T07:51:26.0411065Z fi 2024-06-26T07:51:26.0411362Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T07:51:26.0441773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:51:26.0442147Z env: 2024-06-26T07:51:26.0442349Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:51:26.0442628Z NO_SUDO: true 2024-06-26T07:51:26.0442840Z ##[endgroup] 2024-06-26T07:51:26.0611358Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T07:51:28.0210501Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T07:51:28.0211237Z with: 2024-06-26T07:51:28.0211703Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:51:28.0212355Z fetch-depth: 0 2024-06-26T07:51:28.0212816Z submodules: recursive 2024-06-26T07:51:28.0213329Z quiet-checkout: true 2024-06-26T07:51:28.0213853Z repository: pytorch/pytorch 2024-06-26T07:51:28.0214663Z token: *** 2024-06-26T07:51:28.0215074Z ssh-strict: true 2024-06-26T07:51:28.0215550Z persist-credentials: true 2024-06-26T07:51:28.0216086Z clean: true 2024-06-26T07:51:28.0216539Z sparse-checkout-cone-mode: true 2024-06-26T07:51:28.0217100Z lfs: false 2024-06-26T07:51:28.0217531Z set-safe-directory: true 2024-06-26T07:51:28.0217996Z env: 2024-06-26T07:51:28.0218392Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:51:28.0218896Z ##[endgroup] 2024-06-26T07:51:28.1376044Z Syncing repository: pytorch/pytorch 2024-06-26T07:51:28.1377353Z ##[group]Getting Git version info 2024-06-26T07:51:28.1377922Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:51:28.1378579Z [command]/usr/bin/git version 2024-06-26T07:51:28.1378859Z git version 2.34.1 2024-06-26T07:51:28.1397147Z ##[endgroup] 2024-06-26T07:51:28.1423241Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/6e5d86a4-ff79-4915-9ed1-4f3b31f59090' before making global git config changes 2024-06-26T07:51:28.1425493Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T07:51:28.1427504Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T07:51:28.1467435Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-06-26T07:51:28.1470792Z ##[group]Initializing the repository 2024-06-26T07:51:28.1472860Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T07:51:28.1518784Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T07:51:28.1520365Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T07:51:28.1521783Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T07:51:28.1522775Z hint: 2024-06-26T07:51:28.1523672Z hint: git config --global init.defaultBranch 2024-06-26T07:51:28.1524487Z hint: 2024-06-26T07:51:28.1525419Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T07:51:28.1527387Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T07:51:28.1528573Z hint: 2024-06-26T07:51:28.1529258Z hint: git branch -m 2024-06-26T07:51:28.1530596Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T07:51:28.1533660Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T07:51:28.1583397Z ##[endgroup] 2024-06-26T07:51:28.1584446Z ##[group]Disabling automatic garbage collection 2024-06-26T07:51:28.1585580Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T07:51:28.1627889Z ##[endgroup] 2024-06-26T07:51:28.1628823Z ##[group]Setting up auth 2024-06-26T07:51:28.1631340Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T07:51:28.1678076Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T07:51:28.2031821Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T07:51:28.2083440Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T07:51:28.2443516Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:51:28.2512252Z ##[endgroup] 2024-06-26T07:51:28.2513315Z ##[group]Fetching the repository 2024-06-26T07:51:28.2524593Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T07:51:31.5754673Z remote: Enumerating objects: 987607 2024-06-26T07:51:31.5755704Z remote: Enumerating objects: 990245, done. 2024-06-26T07:51:31.5763295Z remote: Counting objects: 0% (1/2638) 2024-06-26T07:51:31.5764330Z remote: Counting objects: 1% (27/2638) 2024-06-26T07:51:31.5765270Z remote: Counting objects: 2% (53/2638) 2024-06-26T07:51:31.5766194Z remote: Counting objects: 3% (80/2638) 2024-06-26T07:51:31.5788736Z remote: Counting objects: 4% (106/2638) 2024-06-26T07:51:31.5789779Z remote: Counting objects: 5% (132/2638) 2024-06-26T07:51:31.5790711Z remote: Counting objects: 6% (159/2638) 2024-06-26T07:51:31.5791607Z remote: Counting objects: 7% (185/2638) 2024-06-26T07:51:31.5792479Z remote: Counting objects: 8% (212/2638) 2024-06-26T07:51:31.5793307Z remote: Counting objects: 9% (238/2638) 2024-06-26T07:51:31.5794223Z remote: Counting objects: 10% (264/2638) 2024-06-26T07:51:31.5795099Z remote: Counting objects: 11% (291/2638) 2024-06-26T07:51:31.5795993Z remote: Counting objects: 12% (317/2638) 2024-06-26T07:51:31.5796841Z remote: Counting objects: 13% (343/2638) 2024-06-26T07:51:31.5797680Z remote: Counting objects: 14% (370/2638) 2024-06-26T07:51:31.5798502Z remote: Counting objects: 15% (396/2638) 2024-06-26T07:51:31.5799509Z remote: Counting objects: 16% (423/2638) 2024-06-26T07:51:31.5800553Z remote: Counting objects: 17% (449/2638) 2024-06-26T07:51:31.5801448Z remote: Counting objects: 18% (475/2638) 2024-06-26T07:51:31.5802293Z remote: Counting objects: 19% (502/2638) 2024-06-26T07:51:31.5803148Z remote: Counting objects: 20% (528/2638) 2024-06-26T07:51:31.5804044Z remote: Counting objects: 21% (554/2638) 2024-06-26T07:51:31.5804879Z remote: Counting objects: 22% (581/2638) 2024-06-26T07:51:31.5805739Z remote: Counting objects: 23% (607/2638) 2024-06-26T07:51:31.5806574Z remote: Counting objects: 24% (634/2638) 2024-06-26T07:51:31.5807401Z remote: Counting objects: 25% (660/2638) 2024-06-26T07:51:31.5808636Z remote: Counting objects: 26% (686/2638) 2024-06-26T07:51:31.5810177Z remote: Counting objects: 27% (713/2638) 2024-06-26T07:51:31.5811029Z remote: Counting objects: 28% (739/2638) 2024-06-26T07:51:31.5811869Z remote: Counting objects: 29% (766/2638) 2024-06-26T07:51:31.5812715Z remote: Counting objects: 30% (792/2638) 2024-06-26T07:51:31.5813565Z remote: Counting objects: 31% (818/2638) 2024-06-26T07:51:31.5814428Z remote: Counting objects: 32% (845/2638) 2024-06-26T07:51:31.5815274Z remote: Counting objects: 33% (871/2638) 2024-06-26T07:51:31.5816113Z remote: Counting objects: 34% (897/2638) 2024-06-26T07:51:31.5816928Z remote: Counting objects: 35% (924/2638) 2024-06-26T07:51:31.5817763Z remote: Counting objects: 36% (950/2638) 2024-06-26T07:51:31.5818594Z remote: Counting objects: 37% (977/2638) 2024-06-26T07:51:31.5819449Z remote: Counting objects: 38% (1003/2638) 2024-06-26T07:51:31.5820363Z remote: Counting objects: 39% (1029/2638) 2024-06-26T07:51:31.5821284Z remote: Counting objects: 40% (1056/2638) 2024-06-26T07:51:31.5822181Z remote: Counting objects: 41% (1082/2638) 2024-06-26T07:51:31.5823081Z remote: Counting objects: 42% (1108/2638) 2024-06-26T07:51:31.5823967Z remote: Counting objects: 43% (1135/2638) 2024-06-26T07:51:31.5824833Z remote: Counting objects: 44% (1161/2638) 2024-06-26T07:51:31.5825903Z remote: Counting objects: 45% (1188/2638) 2024-06-26T07:51:31.5826778Z remote: Counting objects: 46% (1214/2638) 2024-06-26T07:51:31.5827633Z remote: Counting objects: 47% (1240/2638) 2024-06-26T07:51:31.5828496Z remote: Counting objects: 48% (1267/2638) 2024-06-26T07:51:31.5829366Z remote: Counting objects: 49% (1293/2638) 2024-06-26T07:51:31.5830214Z remote: Counting objects: 50% (1319/2638) 2024-06-26T07:51:31.5831109Z remote: Counting objects: 51% (1346/2638) 2024-06-26T07:51:31.5831964Z remote: Counting objects: 52% (1372/2638) 2024-06-26T07:51:31.5832840Z remote: Counting objects: 53% (1399/2638) 2024-06-26T07:51:31.5833698Z remote: Counting objects: 54% (1425/2638) 2024-06-26T07:51:31.5834555Z remote: Counting objects: 55% (1451/2638) 2024-06-26T07:51:31.5835514Z remote: Counting objects: 56% (1478/2638) 2024-06-26T07:51:31.5836374Z remote: Counting objects: 57% (1504/2638) 2024-06-26T07:51:31.5837243Z remote: Counting objects: 58% (1531/2638) 2024-06-26T07:51:31.5838105Z remote: Counting objects: 59% (1557/2638) 2024-06-26T07:51:31.5838959Z remote: Counting objects: 60% (1583/2638) 2024-06-26T07:51:31.5839825Z remote: Counting objects: 61% (1610/2638) 2024-06-26T07:51:31.5840681Z remote: Counting objects: 62% (1636/2638) 2024-06-26T07:51:31.5841565Z remote: Counting objects: 63% (1662/2638) 2024-06-26T07:51:31.5842415Z remote: Counting objects: 64% (1689/2638) 2024-06-26T07:51:31.5843286Z remote: Counting objects: 65% (1715/2638) 2024-06-26T07:51:31.5844146Z remote: Counting objects: 66% (1742/2638) 2024-06-26T07:51:31.5845009Z remote: Counting objects: 67% (1768/2638) 2024-06-26T07:51:31.5845853Z remote: Counting objects: 68% (1794/2638) 2024-06-26T07:51:31.5846716Z remote: Counting objects: 69% (1821/2638) 2024-06-26T07:51:31.5847593Z remote: Counting objects: 70% (1847/2638) 2024-06-26T07:51:31.5848667Z remote: Counting objects: 71% (1873/2638) 2024-06-26T07:51:31.5849524Z remote: Counting objects: 72% (1900/2638) 2024-06-26T07:51:31.5850386Z remote: Counting objects: 73% (1926/2638) 2024-06-26T07:51:31.5851227Z remote: Counting objects: 74% (1953/2638) 2024-06-26T07:51:31.5852092Z remote: Counting objects: 75% (1979/2638) 2024-06-26T07:51:31.5852956Z remote: Counting objects: 76% (2005/2638) 2024-06-26T07:51:31.5853808Z remote: Counting objects: 77% (2032/2638) 2024-06-26T07:51:31.5854834Z remote: Counting objects: 78% (2058/2638) 2024-06-26T07:51:31.5855707Z remote: Counting objects: 79% (2085/2638) 2024-06-26T07:51:31.5856547Z remote: Counting objects: 80% (2111/2638) 2024-06-26T07:51:31.5857395Z remote: Counting objects: 81% (2137/2638) 2024-06-26T07:51:31.5858264Z remote: Counting objects: 82% (2164/2638) 2024-06-26T07:51:31.5859120Z remote: Counting objects: 83% (2190/2638) 2024-06-26T07:51:31.5859980Z remote: Counting objects: 84% (2216/2638) 2024-06-26T07:51:31.5860834Z remote: Counting objects: 85% (2243/2638) 2024-06-26T07:51:31.5861688Z remote: Counting objects: 86% (2269/2638) 2024-06-26T07:51:31.5862539Z remote: Counting objects: 87% (2296/2638) 2024-06-26T07:51:31.5863401Z remote: Counting objects: 88% (2322/2638) 2024-06-26T07:51:31.5864259Z remote: Counting objects: 89% (2348/2638) 2024-06-26T07:51:31.5865104Z remote: Counting objects: 90% (2375/2638) 2024-06-26T07:51:31.5865973Z remote: Counting objects: 91% (2401/2638) 2024-06-26T07:51:31.5866833Z remote: Counting objects: 92% (2427/2638) 2024-06-26T07:51:31.5867676Z remote: Counting objects: 93% (2454/2638) 2024-06-26T07:51:31.5868545Z remote: Counting objects: 94% (2480/2638) 2024-06-26T07:51:31.5869395Z remote: Counting objects: 95% (2507/2638) 2024-06-26T07:51:31.5870405Z remote: Counting objects: 96% (2533/2638) 2024-06-26T07:51:31.5871272Z remote: Counting objects: 97% (2559/2638) 2024-06-26T07:51:31.5872134Z remote: Counting objects: 98% (2586/2638) 2024-06-26T07:51:31.5872979Z remote: Counting objects: 99% (2612/2638) 2024-06-26T07:51:31.5873834Z remote: Counting objects: 100% (2638/2638) 2024-06-26T07:51:31.5874774Z remote: Counting objects: 100% (2638/2638), done. 2024-06-26T07:51:31.5944203Z remote: Compressing objects: 0% (1/1097) 2024-06-26T07:51:31.6312125Z remote: Compressing objects: 1% (11/1097) 2024-06-26T07:51:31.6699537Z remote: Compressing objects: 2% (22/1097) 2024-06-26T07:51:31.7071813Z remote: Compressing objects: 3% (33/1097) 2024-06-26T07:51:31.7232278Z remote: Compressing objects: 4% (44/1097) 2024-06-26T07:51:31.7499337Z remote: Compressing objects: 5% (55/1097) 2024-06-26T07:51:31.7658074Z remote: Compressing objects: 6% (66/1097) 2024-06-26T07:51:31.8017175Z remote: Compressing objects: 7% (77/1097) 2024-06-26T07:51:31.8993477Z remote: Compressing objects: 8% (88/1097) 2024-06-26T07:51:31.9579132Z remote: Compressing objects: 9% (99/1097) 2024-06-26T07:51:32.0096760Z remote: Compressing objects: 10% (110/1097) 2024-06-26T07:51:32.0184509Z remote: Compressing objects: 11% (121/1097) 2024-06-26T07:51:32.0273120Z remote: Compressing objects: 12% (132/1097) 2024-06-26T07:51:32.0356297Z remote: Compressing objects: 13% (143/1097) 2024-06-26T07:51:32.0392690Z remote: Compressing objects: 14% (154/1097) 2024-06-26T07:51:32.0393802Z remote: Compressing objects: 15% (165/1097) 2024-06-26T07:51:32.0443518Z remote: Compressing objects: 16% (176/1097) 2024-06-26T07:51:32.0444615Z remote: Compressing objects: 17% (187/1097) 2024-06-26T07:51:32.0452110Z remote: Compressing objects: 18% (198/1097) 2024-06-26T07:51:32.0469132Z remote: Compressing objects: 19% (209/1097) 2024-06-26T07:51:32.0488468Z remote: Compressing objects: 20% (220/1097) 2024-06-26T07:51:32.0507925Z remote: Compressing objects: 21% (231/1097) 2024-06-26T07:51:32.0517894Z remote: Compressing objects: 22% (242/1097) 2024-06-26T07:51:32.0539980Z remote: Compressing objects: 23% (253/1097) 2024-06-26T07:51:32.0546544Z remote: Compressing objects: 24% (264/1097) 2024-06-26T07:51:32.0551481Z remote: Compressing objects: 25% (275/1097) 2024-06-26T07:51:32.0558592Z remote: Compressing objects: 26% (286/1097) 2024-06-26T07:51:32.0563579Z remote: Compressing objects: 27% (297/1097) 2024-06-26T07:51:32.0576259Z remote: Compressing objects: 28% (308/1097) 2024-06-26T07:51:32.0577934Z remote: Compressing objects: 29% (319/1097) 2024-06-26T07:51:32.0583972Z remote: Compressing objects: 30% (330/1097) 2024-06-26T07:51:32.0590942Z remote: Compressing objects: 31% (341/1097) 2024-06-26T07:51:32.0592054Z remote: Compressing objects: 32% (352/1097) 2024-06-26T07:51:32.0607971Z remote: Compressing objects: 33% (363/1097) 2024-06-26T07:51:32.0621418Z remote: Compressing objects: 34% (373/1097) 2024-06-26T07:51:32.0633307Z remote: Compressing objects: 35% (384/1097) 2024-06-26T07:51:32.0659966Z remote: Compressing objects: 36% (395/1097) 2024-06-26T07:51:32.0666553Z remote: Compressing objects: 37% (406/1097) 2024-06-26T07:51:32.0677337Z remote: Compressing objects: 38% (417/1097) 2024-06-26T07:51:32.0694245Z remote: Compressing objects: 39% (428/1097) 2024-06-26T07:51:32.0709161Z remote: Compressing objects: 40% (439/1097) 2024-06-26T07:51:32.0722780Z remote: Compressing objects: 41% (450/1097) 2024-06-26T07:51:32.0740975Z remote: Compressing objects: 42% (461/1097) 2024-06-26T07:51:32.0744274Z remote: Compressing objects: 43% (472/1097) 2024-06-26T07:51:32.0749563Z remote: Compressing objects: 44% (483/1097) 2024-06-26T07:51:32.0753776Z remote: Compressing objects: 45% (494/1097) 2024-06-26T07:51:32.0771994Z remote: Compressing objects: 46% (505/1097) 2024-06-26T07:51:32.0787656Z remote: Compressing objects: 47% (516/1097) 2024-06-26T07:51:32.0790505Z remote: Compressing objects: 48% (527/1097) 2024-06-26T07:51:32.0797697Z remote: Compressing objects: 49% (538/1097) 2024-06-26T07:51:32.0804820Z remote: Compressing objects: 50% (549/1097) 2024-06-26T07:51:32.0812980Z remote: Compressing objects: 51% (560/1097) 2024-06-26T07:51:32.0827158Z remote: Compressing objects: 52% (571/1097) 2024-06-26T07:51:32.0843473Z remote: Compressing objects: 53% (582/1097) 2024-06-26T07:51:32.0863721Z remote: Compressing objects: 54% (593/1097) 2024-06-26T07:51:32.0872054Z remote: Compressing objects: 55% (604/1097) 2024-06-26T07:51:32.0875144Z remote: Compressing objects: 56% (615/1097) 2024-06-26T07:51:32.0888851Z remote: Compressing objects: 57% (626/1097) 2024-06-26T07:51:32.0901260Z remote: Compressing objects: 58% (637/1097) 2024-06-26T07:51:32.0905949Z remote: Compressing objects: 59% (648/1097) 2024-06-26T07:51:32.0917188Z remote: Compressing objects: 60% (659/1097) 2024-06-26T07:51:32.0929611Z remote: Compressing objects: 61% (670/1097) 2024-06-26T07:51:32.0935255Z remote: Compressing objects: 62% (681/1097) 2024-06-26T07:51:32.0943521Z remote: Compressing objects: 63% (692/1097) 2024-06-26T07:51:32.0954439Z remote: Compressing objects: 64% (703/1097) 2024-06-26T07:51:32.0959976Z remote: Compressing objects: 65% (714/1097) 2024-06-26T07:51:32.0962970Z remote: Compressing objects: 66% (725/1097) 2024-06-26T07:51:32.0978464Z remote: Compressing objects: 67% (735/1097) 2024-06-26T07:51:32.0983733Z remote: Compressing objects: 68% (746/1097) 2024-06-26T07:51:32.0984662Z remote: Compressing objects: 69% (757/1097) 2024-06-26T07:51:32.0987991Z remote: Compressing objects: 70% (768/1097) 2024-06-26T07:51:32.0988900Z remote: Compressing objects: 71% (779/1097) 2024-06-26T07:51:32.0989798Z remote: Compressing objects: 72% (790/1097) 2024-06-26T07:51:32.0991754Z remote: Compressing objects: 73% (801/1097) 2024-06-26T07:51:32.0992646Z remote: Compressing objects: 74% (812/1097) 2024-06-26T07:51:32.0993559Z remote: Compressing objects: 75% (823/1097) 2024-06-26T07:51:32.0994457Z remote: Compressing objects: 76% (834/1097) 2024-06-26T07:51:32.0995586Z remote: Compressing objects: 77% (845/1097) 2024-06-26T07:51:32.0996504Z remote: Compressing objects: 78% (856/1097) 2024-06-26T07:51:32.0997406Z remote: Compressing objects: 79% (867/1097) 2024-06-26T07:51:32.0998638Z remote: Compressing objects: 80% (878/1097) 2024-06-26T07:51:32.1001654Z remote: Compressing objects: 81% (889/1097) 2024-06-26T07:51:32.1002568Z remote: Compressing objects: 82% (900/1097) 2024-06-26T07:51:32.1006131Z remote: Compressing objects: 83% (911/1097) 2024-06-26T07:51:32.1008567Z remote: Compressing objects: 84% (922/1097) 2024-06-26T07:51:32.1009467Z remote: Compressing objects: 85% (933/1097) 2024-06-26T07:51:32.1010380Z remote: Compressing objects: 86% (944/1097) 2024-06-26T07:51:32.1012131Z remote: Compressing objects: 87% (955/1097) 2024-06-26T07:51:32.1016760Z remote: Compressing objects: 88% (966/1097) 2024-06-26T07:51:32.1017817Z remote: Compressing objects: 89% (977/1097) 2024-06-26T07:51:32.1020705Z remote: Compressing objects: 90% (988/1097) 2024-06-26T07:51:32.1021635Z remote: Compressing objects: 91% (999/1097) 2024-06-26T07:51:32.1022570Z remote: Compressing objects: 92% (1010/1097) 2024-06-26T07:51:32.1023515Z remote: Compressing objects: 93% (1021/1097) 2024-06-26T07:51:32.1025067Z remote: Compressing objects: 94% (1032/1097) 2024-06-26T07:51:32.1025962Z remote: Compressing objects: 95% (1043/1097) 2024-06-26T07:51:32.1026865Z remote: Compressing objects: 96% (1054/1097) 2024-06-26T07:51:32.1027754Z remote: Compressing objects: 97% (1065/1097) 2024-06-26T07:51:32.1028648Z remote: Compressing objects: 98% (1076/1097) 2024-06-26T07:51:32.1029507Z remote: Compressing objects: 99% (1087/1097) 2024-06-26T07:51:32.1030377Z remote: Compressing objects: 100% (1097/1097) 2024-06-26T07:51:32.1031426Z remote: Compressing objects: 100% (1097/1097), done. 2024-06-26T07:51:51.9802515Z remote: Total 990245 (delta 2089), reused 1897 (delta 1540), pack-reused 987607 2024-06-26T07:52:01.4532585Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:52:01.4580424Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:52:01.4588672Z ##[endgroup] 2024-06-26T07:52:01.4597248Z ##[group]Determining the checkout info 2024-06-26T07:52:01.4600076Z ##[endgroup] 2024-06-26T07:52:01.4601003Z ##[group]Checking out the ref 2024-06-26T07:52:01.4602800Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:52:02.5131454Z ##[endgroup] 2024-06-26T07:52:02.5132657Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:52:02.5139265Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:52:02.5216257Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:52:02.5270686Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:52:02.5329480Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:52:02.5377511Z ##[endgroup] 2024-06-26T07:52:02.5378480Z ##[group]Fetching submodules 2024-06-26T07:52:02.5385263Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:52:02.5777188Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:52:02.6154768Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:52:02.6157791Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:52:02.6163719Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:52:02.6168339Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:52:02.6175495Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:52:02.6178508Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:52:02.6183080Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:52:02.6189355Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:52:02.6194494Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:52:02.6200177Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:52:02.6205574Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:52:02.6210999Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:52:02.6217065Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:52:02.6223234Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:52:02.6233297Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:52:02.6239635Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:52:02.6246027Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:52:02.6252622Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:52:02.6259350Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:52:02.6266050Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:52:02.6272923Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:52:02.6279768Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:52:02.6286743Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:52:02.6294780Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:52:02.6302037Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:52:02.6309645Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:52:02.6317805Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:52:02.6325007Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:52:02.6333466Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:52:02.6341185Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:52:02.6349852Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:52:02.6357178Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:52:02.6365287Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:52:02.6374529Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:52:02.6383042Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:52:02.6481765Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:52:03.6647629Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:52:04.4225888Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:52:05.0987873Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:52:06.7707266Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:52:09.2407895Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:52:16.6004778Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:52:17.8663028Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:52:19.1461351Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:52:20.4685569Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:52:22.5961238Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:52:25.0984361Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:52:29.6164046Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:52:31.8431578Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:52:34.2312265Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:52:36.0043946Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:52:36.6537895Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:52:37.7591990Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:52:38.9760368Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:52:49.0549345Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:52:50.0868679Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:52:51.0102017Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:52:53.5659744Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:52:54.9830687Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:52:56.4474283Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:53:02.4227495Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:53:05.1751972Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:53:10.5219381Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:53:11.4586664Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:53:18.9510531Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:53:19.6136214Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:53:20.5070845Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:53:22.5591993Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:53:23.6512007Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:53:25.0492800Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:53:26.3164037Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:53:26.3542735Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:53:26.3842753Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:53:26.4395852Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:53:26.5142513Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:53:27.2488110Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:53:27.3048902Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:53:27.3963946Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:53:27.5278643Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:53:27.6011717Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:53:28.0227144Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:53:28.2661998Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:53:28.3870150Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:53:28.3943881Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:53:28.3949269Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:53:28.3954748Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:53:28.3959024Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:53:28.3965151Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:53:28.4033206Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:53:30.3182017Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:53:31.6931274Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:53:34.1235989Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:53:35.7169882Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:53:36.7246076Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:53:36.8510259Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:53:37.1897478Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:53:37.3030346Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:53:37.3362350Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:53:37.4697187Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:53:37.5462172Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:53:37.5771148Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:53:37.6619241Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:53:37.7191362Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:53:37.7981948Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:53:37.8363655Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:53:37.8426673Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:53:37.8486891Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:53:48.5073411Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:53:48.5558063Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:53:48.6831167Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:53:48.6896571Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:53:48.6900530Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:53:48.6907553Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:53:48.6971599Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:53:49.8637670Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:53:51.6331775Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:53:53.2991118Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:53:53.3065523Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:53:53.3069928Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:53:53.3075374Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:53:53.3080553Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:53:53.3086132Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:53:53.3091685Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:53:53.3096962Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:53:53.3103000Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:53:53.3170857Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:53:55.0597123Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:53:56.5483793Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:53:58.5858273Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:53:59.7079717Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:54:01.0176202Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:54:02.5567483Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:54:08.8899456Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:54:10.3481995Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:54:10.3941204Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:54:10.4684767Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:54:10.5035420Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:54:10.5105105Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:10.5168970Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:54:11.8054867Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:54:11.8506240Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:54:11.9279280Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:54:12.0526840Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:54:12.0947139Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:54:12.1706990Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:54:12.2754135Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:54:12.3547380Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:54:12.4097430Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:54:12.5400484Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:54:12.8440386Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:54:12.8534991Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:12.8538883Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:12.8629575Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:54:14.0719357Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:54:15.9435916Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:54:16.0159848Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:54:16.1157236Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:54:16.1228575Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:16.1232980Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:16.1239395Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:16.1243758Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:16.1250154Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:16.1254684Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:16.1261455Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:16.1265921Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:54:16.1327990Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:54:18.2081835Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:54:19.7926292Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:54:20.8936793Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:54:26.8803941Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:54:27.7490468Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:54:28.8595634Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:54:29.8816289Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:54:39.3948034Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:54:39.4788493Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:54:39.5199101Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:54:39.6501093Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:54:39.6862882Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:54:39.7245162Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:54:39.7642220Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:54:39.7707781Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:54:39.7716079Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:54:39.7777529Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:54:42.0259098Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:54:46.0617451Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:54:46.1416064Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:54:46.5148667Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:54:46.5523501Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:54:46.8121574Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:54:46.8187683Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:54:46.8193038Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:54:46.8259154Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:54:48.0264176Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:54:49.5728131Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:54:49.7024493Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:54:49.7336924Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:54:49.7704062Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:54:49.8446519Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:54:49.9085887Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:54:49.9910355Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:54:50.0493190Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:54:50.0560869Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:54:50.0566788Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:54:50.0569848Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:54:50.0575503Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:54:50.0638008Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:54:51.7395099Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:54:52.6422386Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:54:54.5510480Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:54:56.5146572Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:54:56.5538321Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:54:56.6695453Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:54:56.7311606Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:54:56.7379682Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:54:56.7442441Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:54:57.5748276Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:54:57.5884058Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:54:57.6278877Z Entering 'android/libs/fbjni' 2024-06-26T07:54:57.6339211Z Entering 'third_party/FP16' 2024-06-26T07:54:57.6400030Z Entering 'third_party/FXdiv' 2024-06-26T07:54:57.6457252Z Entering 'third_party/NNPACK' 2024-06-26T07:54:57.6516753Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:54:57.6573544Z Entering 'third_party/XNNPACK' 2024-06-26T07:54:57.6656531Z Entering 'third_party/benchmark' 2024-06-26T07:54:57.6715885Z Entering 'third_party/cpp-httplib' 2024-06-26T07:54:57.6767542Z Entering 'third_party/cpuinfo' 2024-06-26T07:54:57.6827826Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:54:57.6882324Z Entering 'third_party/cutlass' 2024-06-26T07:54:57.6949306Z Entering 'third_party/eigen' 2024-06-26T07:54:57.7008591Z Entering 'third_party/fbgemm' 2024-06-26T07:54:57.7069051Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:54:57.7122767Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:54:57.7178473Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:54:57.7238944Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:54:57.7295859Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:54:57.7350631Z Entering 'third_party/flatbuffers' 2024-06-26T07:54:57.7414942Z Entering 'third_party/fmt' 2024-06-26T07:54:57.7472968Z Entering 'third_party/foxi' 2024-06-26T07:54:57.7528418Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:54:57.7586784Z Entering 'third_party/gloo' 2024-06-26T07:54:57.7642653Z Entering 'third_party/googletest' 2024-06-26T07:54:57.7698235Z Entering 'third_party/ideep' 2024-06-26T07:54:57.7756493Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:54:57.7818469Z Entering 'third_party/ittapi' 2024-06-26T07:54:57.7874710Z Entering 'third_party/kineto' 2024-06-26T07:54:57.7936954Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:54:57.7983647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:54:57.8043339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:54:57.8100345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:54:57.8152536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:54:57.8200832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:57.8255666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:54:57.8309023Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:54:57.8360108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:54:57.8414578Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:54:57.8468234Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:54:57.8520359Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:54:57.8578859Z Entering 'third_party/mimalloc' 2024-06-26T07:54:57.8634673Z Entering 'third_party/nccl/nccl' 2024-06-26T07:54:57.8694082Z Entering 'third_party/nlohmann' 2024-06-26T07:54:57.8747300Z Entering 'third_party/onnx' 2024-06-26T07:54:57.8823155Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:57.8875444Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:57.8934741Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:54:57.8989418Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:57.9040346Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:57.9095956Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:57.9147099Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:57.9199012Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:57.9248949Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:57.9298997Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:57.9346072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:54:57.9398614Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:54:57.9456837Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:54:57.9540220Z Entering 'third_party/pocketfft' 2024-06-26T07:54:57.9593543Z Entering 'third_party/protobuf' 2024-06-26T07:54:57.9654150Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:54:57.9703169Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:54:57.9763374Z Entering 'third_party/psimd' 2024-06-26T07:54:57.9817198Z Entering 'third_party/pthreadpool' 2024-06-26T07:54:57.9870750Z Entering 'third_party/pybind11' 2024-06-26T07:54:57.9927468Z Entering 'third_party/python-peachpy' 2024-06-26T07:54:57.9980842Z Entering 'third_party/sleef' 2024-06-26T07:54:58.0032275Z Entering 'third_party/tensorpipe' 2024-06-26T07:54:58.0089323Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:54:58.0140045Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:54:58.0195502Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:54:58.0248767Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:54:58.0298564Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:54:58.0382123Z ##[endgroup] 2024-06-26T07:54:58.0383349Z ##[group]Persisting credentials for submodules 2024-06-26T07:54:58.0386642Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:54:58.0737279Z Entering 'android/libs/fbjni' 2024-06-26T07:54:58.0802677Z Entering 'third_party/FP16' 2024-06-26T07:54:58.0864471Z Entering 'third_party/FXdiv' 2024-06-26T07:54:58.0928539Z Entering 'third_party/NNPACK' 2024-06-26T07:54:58.0999430Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:54:58.1067406Z Entering 'third_party/XNNPACK' 2024-06-26T07:54:58.1150707Z Entering 'third_party/benchmark' 2024-06-26T07:54:58.1216037Z Entering 'third_party/cpp-httplib' 2024-06-26T07:54:58.1272187Z Entering 'third_party/cpuinfo' 2024-06-26T07:54:58.1334637Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:54:58.1394237Z Entering 'third_party/cutlass' 2024-06-26T07:54:58.1466933Z Entering 'third_party/eigen' 2024-06-26T07:54:58.1534847Z Entering 'third_party/fbgemm' 2024-06-26T07:54:58.1592346Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:54:58.1654830Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:54:58.1714583Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:54:58.1783658Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:54:58.1842584Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:54:58.1906432Z Entering 'third_party/flatbuffers' 2024-06-26T07:54:58.1972096Z Entering 'third_party/fmt' 2024-06-26T07:54:58.2035680Z Entering 'third_party/foxi' 2024-06-26T07:54:58.2097962Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:54:58.2161484Z Entering 'third_party/gloo' 2024-06-26T07:54:58.2227419Z Entering 'third_party/googletest' 2024-06-26T07:54:58.2295605Z Entering 'third_party/ideep' 2024-06-26T07:54:58.2350446Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:54:58.2419715Z Entering 'third_party/ittapi' 2024-06-26T07:54:58.2480708Z Entering 'third_party/kineto' 2024-06-26T07:54:58.2547255Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:54:58.2604860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:54:58.2665080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:54:58.2724055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:54:58.2786483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:54:58.2843846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:58.2905859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:54:58.2965603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:54:58.3025717Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:54:58.3086155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:54:58.3146737Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:54:58.3206252Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:54:58.3265534Z Entering 'third_party/mimalloc' 2024-06-26T07:54:58.3325214Z Entering 'third_party/nccl/nccl' 2024-06-26T07:54:58.3385387Z Entering 'third_party/nlohmann' 2024-06-26T07:54:58.3453147Z Entering 'third_party/onnx' 2024-06-26T07:54:58.3538186Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:58.3600127Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:58.3671101Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:54:58.3734704Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:58.3799051Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:58.3858969Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:58.3921195Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:58.3982025Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:58.4039839Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:58.4100255Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:58.4157717Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:54:58.4222131Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:54:58.4284398Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:54:58.4373511Z Entering 'third_party/pocketfft' 2024-06-26T07:54:58.4435525Z Entering 'third_party/protobuf' 2024-06-26T07:54:58.4501324Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:54:58.4560567Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:54:58.4621647Z Entering 'third_party/psimd' 2024-06-26T07:54:58.4683654Z Entering 'third_party/pthreadpool' 2024-06-26T07:54:58.4746368Z Entering 'third_party/pybind11' 2024-06-26T07:54:58.4814912Z Entering 'third_party/python-peachpy' 2024-06-26T07:54:58.4871204Z Entering 'third_party/sleef' 2024-06-26T07:54:58.4935235Z Entering 'third_party/tensorpipe' 2024-06-26T07:54:58.4990929Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:54:58.5057295Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:54:58.5118104Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:54:58.5179587Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:54:58.5233336Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:54:58.5320428Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:54:58.5686278Z Entering 'android/libs/fbjni' 2024-06-26T07:54:58.5739167Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:54:58.5767421Z Entering 'third_party/FP16' 2024-06-26T07:54:58.5832821Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:54:58.5860810Z Entering 'third_party/FXdiv' 2024-06-26T07:54:58.5915265Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:54:58.5945708Z Entering 'third_party/NNPACK' 2024-06-26T07:54:58.6001486Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:54:58.6035612Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:54:58.6095187Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:54:58.6122221Z Entering 'third_party/XNNPACK' 2024-06-26T07:54:58.6175422Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:54:58.6223392Z Entering 'third_party/benchmark' 2024-06-26T07:54:58.6278541Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:54:58.6306166Z Entering 'third_party/cpp-httplib' 2024-06-26T07:54:58.6359175Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:54:58.6386799Z Entering 'third_party/cpuinfo' 2024-06-26T07:54:58.6440154Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:54:58.6468830Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:54:58.6520544Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:54:58.6548504Z Entering 'third_party/cutlass' 2024-06-26T07:54:58.6604258Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:54:58.6642278Z Entering 'third_party/eigen' 2024-06-26T07:54:58.6695674Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:54:58.6723656Z Entering 'third_party/fbgemm' 2024-06-26T07:54:58.6776806Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:54:58.6804256Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:54:58.6857990Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:54:58.6884733Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:54:58.6937139Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:54:58.6964703Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:54:58.7017281Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:54:58.7054544Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:54:58.7104853Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:58.7135225Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:54:58.7183191Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:54:58.7214172Z Entering 'third_party/flatbuffers' 2024-06-26T07:54:58.7264401Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:54:58.7295347Z Entering 'third_party/fmt' 2024-06-26T07:54:58.7352358Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:54:58.7379498Z Entering 'third_party/foxi' 2024-06-26T07:54:58.7433959Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:54:58.7461727Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:54:58.7513320Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:54:58.7542120Z Entering 'third_party/gloo' 2024-06-26T07:54:58.7595127Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:54:58.7621505Z Entering 'third_party/googletest' 2024-06-26T07:54:58.7678220Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:58.7706524Z Entering 'third_party/ideep' 2024-06-26T07:54:58.7764586Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:54:58.7797473Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:54:58.7850197Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:54:58.7896424Z Entering 'third_party/ittapi' 2024-06-26T07:54:58.7946470Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:54:58.7977986Z Entering 'third_party/kineto' 2024-06-26T07:54:58.8034772Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:54:58.8061751Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:54:58.8117369Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:54:58.8144938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:54:58.8198733Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:54:58.8229288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:54:58.8296805Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:54:58.8324571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:54:58.8380778Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:54:58.8408468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:54:58.8461052Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:54:58.8486015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:58.8542066Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:54:58.8577730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:54:58.8628353Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:54:58.8659597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:54:58.8712396Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:58.8738564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:54:58.8792764Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:54:58.8821395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:54:58.8875686Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:54:58.8905951Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:54:58.8959944Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:54:58.8985107Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:54:58.9036499Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:54:58.9066450Z Entering 'third_party/mimalloc' 2024-06-26T07:54:58.9119545Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:54:58.9147258Z Entering 'third_party/nccl/nccl' 2024-06-26T07:54:58.9200424Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:54:58.9227805Z Entering 'third_party/nlohmann' 2024-06-26T07:54:58.9284070Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:54:58.9314982Z Entering 'third_party/onnx' 2024-06-26T07:54:58.9367559Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:54:58.9419308Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:58.9474389Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:54:58.9501402Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:58.9553692Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:54:58.9586023Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:54:58.9641518Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:54:58.9671188Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:58.9723354Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:54:58.9749570Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:58.9804534Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:58.9830619Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:58.9882580Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:54:58.9908740Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:58.9960937Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:54:58.9989320Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:59.0042057Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:54:59.0066024Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:59.0118621Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:54:59.0145053Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:59.0199760Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:54:59.0224339Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:54:59.0278739Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:54:59.0309786Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:54:59.0360536Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:54:59.0390080Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:54:59.0444831Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:54:59.0501483Z Entering 'third_party/pocketfft' 2024-06-26T07:54:59.0554521Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:54:59.0583590Z Entering 'third_party/protobuf' 2024-06-26T07:54:59.0639720Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:54:59.0677807Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:54:59.0729489Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:54:59.0763154Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:54:59.0816731Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:59.0846824Z Entering 'third_party/psimd' 2024-06-26T07:54:59.0906257Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:54:59.0936926Z Entering 'third_party/pthreadpool' 2024-06-26T07:54:59.0991892Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:54:59.1021401Z Entering 'third_party/pybind11' 2024-06-26T07:54:59.1075801Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:54:59.1104846Z Entering 'third_party/python-peachpy' 2024-06-26T07:54:59.1158904Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:54:59.1187069Z Entering 'third_party/sleef' 2024-06-26T07:54:59.1242027Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:54:59.1269786Z Entering 'third_party/tensorpipe' 2024-06-26T07:54:59.1323980Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:54:59.1351932Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:54:59.1415787Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:54:59.1442151Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:54:59.1497574Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:54:59.1523937Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:54:59.1578121Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:54:59.1605128Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:54:59.1659021Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:54:59.1682628Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:54:59.1740372Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:54:59.2168195Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:54:59.2534967Z Entering 'android/libs/fbjni' 2024-06-26T07:54:59.2590342Z Entering 'third_party/FP16' 2024-06-26T07:54:59.2644779Z Entering 'third_party/FXdiv' 2024-06-26T07:54:59.2703029Z Entering 'third_party/NNPACK' 2024-06-26T07:54:59.2758530Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:54:59.2814217Z Entering 'third_party/XNNPACK' 2024-06-26T07:54:59.2887805Z Entering 'third_party/benchmark' 2024-06-26T07:54:59.2946592Z Entering 'third_party/cpp-httplib' 2024-06-26T07:54:59.3000603Z Entering 'third_party/cpuinfo' 2024-06-26T07:54:59.3059515Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:54:59.3120267Z Entering 'third_party/cutlass' 2024-06-26T07:54:59.3186352Z Entering 'third_party/eigen' 2024-06-26T07:54:59.3243793Z Entering 'third_party/fbgemm' 2024-06-26T07:54:59.3298151Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:54:59.3350835Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:54:59.3403709Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:54:59.3462460Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:54:59.3517819Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:54:59.3577195Z Entering 'third_party/flatbuffers' 2024-06-26T07:54:59.3631999Z Entering 'third_party/fmt' 2024-06-26T07:54:59.3685652Z Entering 'third_party/foxi' 2024-06-26T07:54:59.3743378Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:54:59.3797387Z Entering 'third_party/gloo' 2024-06-26T07:54:59.3856449Z Entering 'third_party/googletest' 2024-06-26T07:54:59.3915075Z Entering 'third_party/ideep' 2024-06-26T07:54:59.3967863Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:54:59.4030674Z Entering 'third_party/ittapi' 2024-06-26T07:54:59.4086447Z Entering 'third_party/kineto' 2024-06-26T07:54:59.4137551Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:54:59.4187973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:54:59.4241925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:54:59.4295121Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:54:59.4345599Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:54:59.4396760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:59.4456255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:54:59.4504805Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:54:59.4561402Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:54:59.4617730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:54:59.4670396Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:54:59.4720096Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:54:59.4777759Z Entering 'third_party/mimalloc' 2024-06-26T07:54:59.4830750Z Entering 'third_party/nccl/nccl' 2024-06-26T07:54:59.4883667Z Entering 'third_party/nlohmann' 2024-06-26T07:54:59.4944899Z Entering 'third_party/onnx' 2024-06-26T07:54:59.5017569Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:59.5075301Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:59.5135085Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:54:59.5190792Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:59.5241904Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:59.5298010Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:59.5348722Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:59.5401753Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:59.5455292Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:59.5505323Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:59.5555690Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:54:59.5615388Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:54:59.5662857Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:54:59.5750891Z Entering 'third_party/pocketfft' 2024-06-26T07:54:59.5799665Z Entering 'third_party/protobuf' 2024-06-26T07:54:59.5857703Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:54:59.5913884Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:54:59.5969961Z Entering 'third_party/psimd' 2024-06-26T07:54:59.6016550Z Entering 'third_party/pthreadpool' 2024-06-26T07:54:59.6069400Z Entering 'third_party/pybind11' 2024-06-26T07:54:59.6123920Z Entering 'third_party/python-peachpy' 2024-06-26T07:54:59.6177540Z Entering 'third_party/sleef' 2024-06-26T07:54:59.6235601Z Entering 'third_party/tensorpipe' 2024-06-26T07:54:59.6288017Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:54:59.6338986Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:54:59.6395641Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:54:59.6452074Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:54:59.6494022Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:54:59.6570605Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:54:59.6944614Z Entering 'android/libs/fbjni' 2024-06-26T07:54:59.6998094Z Entering 'third_party/FP16' 2024-06-26T07:54:59.7061004Z Entering 'third_party/FXdiv' 2024-06-26T07:54:59.7122469Z Entering 'third_party/NNPACK' 2024-06-26T07:54:59.7177009Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:54:59.7234456Z Entering 'third_party/XNNPACK' 2024-06-26T07:54:59.7310536Z Entering 'third_party/benchmark' 2024-06-26T07:54:59.7365318Z Entering 'third_party/cpp-httplib' 2024-06-26T07:54:59.7418734Z Entering 'third_party/cpuinfo' 2024-06-26T07:54:59.7475360Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:54:59.7528751Z Entering 'third_party/cutlass' 2024-06-26T07:54:59.7594914Z Entering 'third_party/eigen' 2024-06-26T07:54:59.7655221Z Entering 'third_party/fbgemm' 2024-06-26T07:54:59.7704850Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:54:59.7755568Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:54:59.7808228Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:54:59.7867956Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:54:59.7919040Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:54:59.7975252Z Entering 'third_party/flatbuffers' 2024-06-26T07:54:59.8032437Z Entering 'third_party/fmt' 2024-06-26T07:54:59.8094468Z Entering 'third_party/foxi' 2024-06-26T07:54:59.8143959Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:54:59.8200762Z Entering 'third_party/gloo' 2024-06-26T07:54:59.8257564Z Entering 'third_party/googletest' 2024-06-26T07:54:59.8312526Z Entering 'third_party/ideep' 2024-06-26T07:54:59.8366091Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:54:59.8429740Z Entering 'third_party/ittapi' 2024-06-26T07:54:59.8483347Z Entering 'third_party/kineto' 2024-06-26T07:54:59.8537914Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:54:59.8589291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:54:59.8644074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:54:59.8698148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:54:59.8749621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:54:59.8797740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:54:59.8854195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:54:59.8909083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:54:59.8963758Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:54:59.9017956Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:54:59.9077467Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:54:59.9126139Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:54:59.9179692Z Entering 'third_party/mimalloc' 2024-06-26T07:54:59.9237704Z Entering 'third_party/nccl/nccl' 2024-06-26T07:54:59.9296368Z Entering 'third_party/nlohmann' 2024-06-26T07:54:59.9352578Z Entering 'third_party/onnx' 2024-06-26T07:54:59.9429631Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:54:59.9484061Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:54:59.9543569Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:54:59.9602193Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:54:59.9657252Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:54:59.9708102Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:54:59.9761772Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:54:59.9817130Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:54:59.9868126Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:54:59.9919281Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:54:59.9968817Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:55:00.0022932Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:55:00.0078285Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:55:00.0155255Z Entering 'third_party/pocketfft' 2024-06-26T07:55:00.0214953Z Entering 'third_party/protobuf' 2024-06-26T07:55:00.0270084Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:55:00.0322127Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:55:00.0374894Z Entering 'third_party/psimd' 2024-06-26T07:55:00.0428661Z Entering 'third_party/pthreadpool' 2024-06-26T07:55:00.0483305Z Entering 'third_party/pybind11' 2024-06-26T07:55:00.0537897Z Entering 'third_party/python-peachpy' 2024-06-26T07:55:00.0590571Z Entering 'third_party/sleef' 2024-06-26T07:55:00.0646972Z Entering 'third_party/tensorpipe' 2024-06-26T07:55:00.0700294Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:55:00.0757070Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:55:00.0813604Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:55:00.0876131Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:55:00.0915139Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:55:00.0988554Z ##[endgroup] 2024-06-26T07:55:00.1049078Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:55:00.1096746Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:55:00.1408793Z Prepare all required actions 2024-06-26T07:55:00.1409813Z Getting action download info 2024-06-26T07:55:00.1537892Z ##[group]Run ./.github/actions/setup-rocm 2024-06-26T07:55:00.1538494Z env: 2024-06-26T07:55:00.1538889Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.1539382Z ##[endgroup] 2024-06-26T07:55:00.1570306Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-06-26T07:55:00.1571731Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-06-26T07:55:00.1617839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.1618580Z env: 2024-06-26T07:55:00.1618970Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.1619459Z ##[endgroup] 2024-06-26T07:55:00.1749128Z ##[group]Run set -ex 2024-06-26T07:55:00.1749692Z set -ex 2024-06-26T07:55:00.1750159Z  2024-06-26T07:55:00.1750689Z cat ~/.docker/config.json || true 2024-06-26T07:55:00.1752527Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-06-26T07:55:00.1754263Z rm -f ~/.docker/config.json 2024-06-26T07:55:00.1807656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.1808589Z env: 2024-06-26T07:55:00.1809040Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.1809834Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.1810699Z ##[endgroup] 2024-06-26T07:55:00.1883504Z + cat /home/pytorchci/.docker/config.json 2024-06-26T07:55:00.1897714Z { 2024-06-26T07:55:00.1898137Z "auths": {} 2024-06-26T07:55:00.1900164Z + rm -f /home/pytorchci/.docker/config.json 2024-06-26T07:55:00.1917830Z } 2024-06-26T07:55:00.1955016Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:55:00.1956316Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:55:00.1957302Z # shellcheck disable=SC2046 2024-06-26T07:55:00.1958071Z docker stop $(docker ps -q) || true 2024-06-26T07:55:00.1958868Z # Prune all stopped containers. 2024-06-26T07:55:00.1959639Z docker container prune -f 2024-06-26T07:55:00.2003660Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.2004499Z env: 2024-06-26T07:55:00.2004940Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.2005616Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.2006340Z ##[endgroup] 2024-06-26T07:55:00.2518016Z "docker stop" requires at least 1 argument. 2024-06-26T07:55:00.2519338Z See 'docker stop --help'. 2024-06-26T07:55:00.2519750Z 2024-06-26T07:55:00.2520242Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:55:00.2522118Z 2024-06-26T07:55:00.2522396Z Stop one or more running containers 2024-06-26T07:55:00.2751385Z Total reclaimed space: 0B 2024-06-26T07:55:00.2817166Z ##[group]Run cat /etc/os-release || true 2024-06-26T07:55:00.2817937Z cat /etc/os-release || true 2024-06-26T07:55:00.2818763Z cat /etc/apt/sources.list.d/rocm.list || true 2024-06-26T07:55:00.2819659Z cat /opt/rocm/.info/version || true 2024-06-26T07:55:00.2820349Z whoami 2024-06-26T07:55:00.2863732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.2864567Z env: 2024-06-26T07:55:00.2865010Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.2865694Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.2866439Z ##[endgroup] 2024-06-26T07:55:00.2946756Z PRETTY_NAME="Ubuntu 22.04.4 LTS" 2024-06-26T07:55:00.2947461Z NAME="Ubuntu" 2024-06-26T07:55:00.2947943Z VERSION_ID="22.04" 2024-06-26T07:55:00.2948512Z VERSION="22.04.4 LTS (Jammy Jellyfish)" 2024-06-26T07:55:00.2949187Z VERSION_CODENAME=jammy 2024-06-26T07:55:00.2949834Z ID=ubuntu 2024-06-26T07:55:00.2950263Z ID_LIKE=debian 2024-06-26T07:55:00.2950797Z HOME_URL="https://www.ubuntu.com/" 2024-06-26T07:55:00.2951544Z SUPPORT_URL="https://help.ubuntu.com/" 2024-06-26T07:55:00.2952754Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-06-26T07:55:00.2956474Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-06-26T07:55:00.2957728Z UBUNTU_CODENAME=jammy 2024-06-26T07:55:00.2961886Z cat: /etc/apt/sources.list.d/rocm.list: No such file or directory 2024-06-26T07:55:00.2975147Z 5.7.1-98 2024-06-26T07:55:00.2989956Z pytorchci 2024-06-26T07:55:00.3029213Z ##[group]Run rocm-smi 2024-06-26T07:55:00.3029773Z rocm-smi 2024-06-26T07:55:00.3079534Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.3080397Z env: 2024-06-26T07:55:00.3080852Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.3081549Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.3082268Z ##[endgroup] 2024-06-26T07:55:00.4597816Z 2024-06-26T07:55:00.4597952Z 2024-06-26T07:55:00.4598312Z ========================= ROCm System Management Interface ========================= 2024-06-26T07:55:00.4598939Z =================================== Concise Info =================================== 2024-06-26T07:55:00.4599600Z GPU Temp (DieEdge) AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-06-26T07:55:00.4600461Z 0 40.0c 100.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-06-26T07:55:00.4601250Z 1 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-06-26T07:55:00.4601922Z 2 42.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-06-26T07:55:00.4603686Z 3 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-06-26T07:55:00.4605050Z 4 44.0c 98.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-06-26T07:55:00.4606232Z 5 36.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-06-26T07:55:00.4607360Z 6 45.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-06-26T07:55:00.4608827Z 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-06-26T07:55:00.4609888Z ==================================================================================== 2024-06-26T07:55:00.4610917Z =============================== End of ROCm SMI Log ================================ 2024-06-26T07:55:00.4753587Z ##[group]Run rocminfo 2024-06-26T07:55:00.4754136Z rocminfo 2024-06-26T07:55:00.4804547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.4805402Z env: 2024-06-26T07:55:00.4805840Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.4806524Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.4807533Z ##[endgroup] 2024-06-26T07:55:00.5800038Z ROCk module is loaded 2024-06-26T07:55:00.5800760Z ===================== 2024-06-26T07:55:00.5801363Z HSA System Attributes 2024-06-26T07:55:00.5802038Z ===================== 2024-06-26T07:55:00.5802675Z Runtime Version: 1.1 2024-06-26T07:55:00.5803417Z System Timestamp Freq.: 1000.000000MHz 2024-06-26T07:55:00.5804578Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-06-26T07:55:00.5805808Z Machine Model: LARGE 2024-06-26T07:55:00.5806805Z System Endianness: LITTLE 2024-06-26T07:55:00.5807687Z Mwaitx: DISABLED 2024-06-26T07:55:00.5808506Z DMAbuf Support: YES 2024-06-26T07:55:00.5808933Z 2024-06-26T07:55:00.5809125Z ========== 2024-06-26T07:55:00.5809658Z HSA Agents 2024-06-26T07:55:00.5810197Z ========== 2024-06-26T07:55:00.5810751Z ******* 2024-06-26T07:55:00.5811246Z Agent 1 2024-06-26T07:55:00.5811771Z ******* 2024-06-26T07:55:00.5812672Z Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:55:00.5813699Z Uuid: CPU-XX 2024-06-26T07:55:00.5814759Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:55:00.5816195Z Vendor Name: CPU 2024-06-26T07:55:00.5817124Z Feature: None specified 2024-06-26T07:55:00.5818071Z Profile: FULL_PROFILE 2024-06-26T07:55:00.5819047Z Float Round Mode: NEAR 2024-06-26T07:55:00.5820021Z Max Queue Number: 0(0x0) 2024-06-26T07:55:00.5820996Z Queue Min Size: 0(0x0) 2024-06-26T07:55:00.5821910Z Queue Max Size: 0(0x0) 2024-06-26T07:55:00.5822877Z Queue Type: MULTI 2024-06-26T07:55:00.5823750Z Node: 0 2024-06-26T07:55:00.5824618Z Device Type: CPU 2024-06-26T07:55:00.5825391Z Cache Info: 2024-06-26T07:55:00.5826094Z L1: 32768(0x8000) KB 2024-06-26T07:55:00.5826980Z Chip ID: 0(0x0) 2024-06-26T07:55:00.5827886Z ASIC Revision: 0(0x0) 2024-06-26T07:55:00.5828856Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.5829816Z Max Clock Freq. (MHz): 2000 2024-06-26T07:55:00.5830707Z BDFID: 0 2024-06-26T07:55:00.5831593Z Internal Node ID: 0 2024-06-26T07:55:00.5832542Z Compute Unit: 64 2024-06-26T07:55:00.5833420Z SIMDs per CU: 0 2024-06-26T07:55:00.5834334Z Shader Engines: 0 2024-06-26T07:55:00.5835253Z Shader Arrs. per Eng.: 0 2024-06-26T07:55:00.5836243Z WatchPts on Addr. Ranges:1 2024-06-26T07:55:00.5837052Z Features: None 2024-06-26T07:55:00.5837653Z Pool Info: 2024-06-26T07:55:00.5838222Z Pool 1 2024-06-26T07:55:00.5839006Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.5839973Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:55:00.5840921Z Allocatable: TRUE 2024-06-26T07:55:00.5841888Z Alloc Granule: 4KB 2024-06-26T07:55:00.5843093Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5844096Z Accessible by all: TRUE 2024-06-26T07:55:00.5844902Z Pool 2 2024-06-26T07:55:00.5845674Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-06-26T07:55:00.5846633Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:55:00.5847579Z Allocatable: TRUE 2024-06-26T07:55:00.5848697Z Alloc Granule: 4KB 2024-06-26T07:55:00.5849659Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5850638Z Accessible by all: TRUE 2024-06-26T07:55:00.5851445Z Pool 3 2024-06-26T07:55:00.5852172Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.5853126Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:55:00.5854032Z Allocatable: TRUE 2024-06-26T07:55:00.5854944Z Alloc Granule: 4KB 2024-06-26T07:55:00.5855896Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5856884Z Accessible by all: TRUE 2024-06-26T07:55:00.5857849Z ISA Info: 2024-06-26T07:55:00.5858413Z ******* 2024-06-26T07:55:00.5858947Z Agent 2 2024-06-26T07:55:00.5859464Z ******* 2024-06-26T07:55:00.5860279Z Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:55:00.5861282Z Uuid: CPU-XX 2024-06-26T07:55:00.5862334Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:55:00.5863296Z Vendor Name: CPU 2024-06-26T07:55:00.5864811Z Feature: None specified 2024-06-26T07:55:00.5865724Z Profile: FULL_PROFILE 2024-06-26T07:55:00.5866670Z Float Round Mode: NEAR 2024-06-26T07:55:00.5867637Z Max Queue Number: 0(0x0) 2024-06-26T07:55:00.5868575Z Queue Min Size: 0(0x0) 2024-06-26T07:55:00.5869483Z Queue Max Size: 0(0x0) 2024-06-26T07:55:00.5870387Z Queue Type: MULTI 2024-06-26T07:55:00.5871264Z Node: 1 2024-06-26T07:55:00.5872126Z Device Type: CPU 2024-06-26T07:55:00.5872886Z Cache Info: 2024-06-26T07:55:00.5873571Z L1: 32768(0x8000) KB 2024-06-26T07:55:00.5874463Z Chip ID: 0(0x0) 2024-06-26T07:55:00.5875340Z ASIC Revision: 0(0x0) 2024-06-26T07:55:00.5876249Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.5877184Z Max Clock Freq. (MHz): 2000 2024-06-26T07:55:00.5878058Z BDFID: 0 2024-06-26T07:55:00.5878945Z Internal Node ID: 1 2024-06-26T07:55:00.5879867Z Compute Unit: 64 2024-06-26T07:55:00.5880738Z SIMDs per CU: 0 2024-06-26T07:55:00.5881637Z Shader Engines: 0 2024-06-26T07:55:00.5882760Z Shader Arrs. per Eng.: 0 2024-06-26T07:55:00.5883892Z WatchPts on Addr. Ranges:1 2024-06-26T07:55:00.5884813Z Features: None 2024-06-26T07:55:00.5885585Z Pool Info: 2024-06-26T07:55:00.5886138Z Pool 1 2024-06-26T07:55:00.5886899Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.5888150Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:55:00.5889116Z Allocatable: TRUE 2024-06-26T07:55:00.5890079Z Alloc Granule: 4KB 2024-06-26T07:55:00.5891031Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5892092Z Accessible by all: TRUE 2024-06-26T07:55:00.5893044Z Pool 2 2024-06-26T07:55:00.5893930Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-06-26T07:55:00.5895044Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:55:00.5896122Z Allocatable: TRUE 2024-06-26T07:55:00.5897157Z Alloc Granule: 4KB 2024-06-26T07:55:00.5898100Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5899051Z Accessible by all: TRUE 2024-06-26T07:55:00.5899855Z Pool 3 2024-06-26T07:55:00.5900739Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.5901660Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:55:00.5902558Z Allocatable: TRUE 2024-06-26T07:55:00.5903478Z Alloc Granule: 4KB 2024-06-26T07:55:00.5904404Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5905353Z Accessible by all: TRUE 2024-06-26T07:55:00.5906132Z ISA Info: 2024-06-26T07:55:00.5906686Z ******* 2024-06-26T07:55:00.5907221Z Agent 3 2024-06-26T07:55:00.5907729Z ******* 2024-06-26T07:55:00.5908369Z Name: gfx90a 2024-06-26T07:55:00.5909385Z Uuid: GPU-34128c8046820a6f 2024-06-26T07:55:00.5910312Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.5911261Z Vendor Name: AMD 2024-06-26T07:55:00.5912296Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.5913375Z Profile: BASE_PROFILE 2024-06-26T07:55:00.5914454Z Float Round Mode: NEAR 2024-06-26T07:55:00.5915403Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.5916321Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.5917213Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.5918137Z Queue Type: MULTI 2024-06-26T07:55:00.5918992Z Node: 2 2024-06-26T07:55:00.5919837Z Device Type: GPU 2024-06-26T07:55:00.5920599Z Cache Info: 2024-06-26T07:55:00.5921285Z L1: 16(0x10) KB 2024-06-26T07:55:00.5922129Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.5923041Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.5923938Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.5924848Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.5925773Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.5926661Z BDFID: 12800 2024-06-26T07:55:00.5927829Z Internal Node ID: 2 2024-06-26T07:55:00.5928734Z Compute Unit: 104 2024-06-26T07:55:00.5929660Z SIMDs per CU: 4 2024-06-26T07:55:00.5930561Z Shader Engines: 8 2024-06-26T07:55:00.5931490Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.5932605Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.5933634Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.5934658Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.5935786Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.5936847Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.5937666Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.5938423Z x 1024(0x400) 2024-06-26T07:55:00.5939247Z y 1024(0x400) 2024-06-26T07:55:00.5940036Z z 1024(0x400) 2024-06-26T07:55:00.5940923Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.5942047Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.5943387Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.5944359Z Grid Max Size per Dimension: 2024-06-26T07:55:00.5945085Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.5945910Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.5946717Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.5947620Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.5970934Z Packet Processor uCode:: 78 2024-06-26T07:55:00.5972079Z SDMA engine uCode:: 8 2024-06-26T07:55:00.5973041Z IOMMU Support:: None 2024-06-26T07:55:00.5973767Z Pool Info: 2024-06-26T07:55:00.5974290Z Pool 1 2024-06-26T07:55:00.5974975Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.5975845Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.5976673Z Allocatable: TRUE 2024-06-26T07:55:00.5977535Z Alloc Granule: 4KB 2024-06-26T07:55:00.5978398Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5979279Z Accessible by all: FALSE 2024-06-26T07:55:00.5980002Z Pool 2 2024-06-26T07:55:00.5980643Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.5981445Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.5982249Z Allocatable: TRUE 2024-06-26T07:55:00.5983093Z Alloc Granule: 4KB 2024-06-26T07:55:00.5983924Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5984771Z Accessible by all: FALSE 2024-06-26T07:55:00.5985484Z Pool 3 2024-06-26T07:55:00.5986127Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.5986907Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.5987679Z Allocatable: TRUE 2024-06-26T07:55:00.5988498Z Alloc Granule: 4KB 2024-06-26T07:55:00.5989319Z Alloc Alignment: 4KB 2024-06-26T07:55:00.5990495Z Accessible by all: FALSE 2024-06-26T07:55:00.5991196Z Pool 4 2024-06-26T07:55:00.5991811Z Segment: GROUP 2024-06-26T07:55:00.5992575Z Size: 64(0x40) KB 2024-06-26T07:55:00.5993361Z Allocatable: FALSE 2024-06-26T07:55:00.5994179Z Alloc Granule: 0KB 2024-06-26T07:55:00.5994990Z Alloc Alignment: 0KB 2024-06-26T07:55:00.5995823Z Accessible by all: FALSE 2024-06-26T07:55:00.5996526Z ISA Info: 2024-06-26T07:55:00.5997019Z ISA 1 2024-06-26T07:55:00.5997899Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.5998841Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.5999731Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6000612Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6001506Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6002337Z Fast f16: TRUE 2024-06-26T07:55:00.6003317Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6004084Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6004783Z x 1024(0x400) 2024-06-26T07:55:00.6005537Z y 1024(0x400) 2024-06-26T07:55:00.6006283Z z 1024(0x400) 2024-06-26T07:55:00.6007096Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6007911Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6008572Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6009319Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6010049Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6010848Z FBarrier Max Size: 32 2024-06-26T07:55:00.6011563Z ******* 2024-06-26T07:55:00.6012071Z Agent 4 2024-06-26T07:55:00.6012609Z ******* 2024-06-26T07:55:00.6013292Z Name: gfx90a 2024-06-26T07:55:00.6014348Z Uuid: GPU-b31dd2b5485cf42f 2024-06-26T07:55:00.6015357Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6016367Z Vendor Name: AMD 2024-06-26T07:55:00.6017236Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6018034Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6018846Z Float Round Mode: NEAR 2024-06-26T07:55:00.6019693Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6020495Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6021325Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6022145Z Queue Type: MULTI 2024-06-26T07:55:00.6022904Z Node: 3 2024-06-26T07:55:00.6023668Z Device Type: GPU 2024-06-26T07:55:00.6024347Z Cache Info: 2024-06-26T07:55:00.6024946Z L1: 16(0x10) KB 2024-06-26T07:55:00.6025693Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6026468Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6027403Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6028222Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6029044Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6029812Z BDFID: 13568 2024-06-26T07:55:00.6030598Z Internal Node ID: 3 2024-06-26T07:55:00.6031398Z Compute Unit: 104 2024-06-26T07:55:00.6032192Z SIMDs per CU: 4 2024-06-26T07:55:00.6033127Z Shader Engines: 8 2024-06-26T07:55:00.6034092Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6035086Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6035844Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6036675Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6037505Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6038325Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6039063Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6039857Z x 1024(0x400) 2024-06-26T07:55:00.6040581Z y 1024(0x400) 2024-06-26T07:55:00.6041281Z z 1024(0x400) 2024-06-26T07:55:00.6042062Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6043007Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6043846Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6044592Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6045205Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6045954Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6046669Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6047473Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6048410Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6049319Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6050295Z IOMMU Support:: None 2024-06-26T07:55:00.6051138Z Pool Info: 2024-06-26T07:55:00.6051699Z Pool 1 2024-06-26T07:55:00.6052374Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6053212Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6054044Z Allocatable: TRUE 2024-06-26T07:55:00.6054939Z Alloc Granule: 4KB 2024-06-26T07:55:00.6055774Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6056668Z Accessible by all: FALSE 2024-06-26T07:55:00.6057517Z Pool 2 2024-06-26T07:55:00.6058473Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6059514Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6060484Z Allocatable: TRUE 2024-06-26T07:55:00.6061461Z Alloc Granule: 4KB 2024-06-26T07:55:00.6062277Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6063123Z Accessible by all: FALSE 2024-06-26T07:55:00.6063835Z Pool 3 2024-06-26T07:55:00.6064472Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6065444Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6066231Z Allocatable: TRUE 2024-06-26T07:55:00.6067043Z Alloc Granule: 4KB 2024-06-26T07:55:00.6067865Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6068711Z Accessible by all: FALSE 2024-06-26T07:55:00.6069414Z Pool 4 2024-06-26T07:55:00.6070028Z Segment: GROUP 2024-06-26T07:55:00.6070793Z Size: 64(0x40) KB 2024-06-26T07:55:00.6071562Z Allocatable: FALSE 2024-06-26T07:55:00.6072397Z Alloc Granule: 0KB 2024-06-26T07:55:00.6073372Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6074362Z Accessible by all: FALSE 2024-06-26T07:55:00.6075183Z ISA Info: 2024-06-26T07:55:00.6075683Z ISA 1 2024-06-26T07:55:00.6076485Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6077386Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6078689Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6079750Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6080729Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6081764Z Fast f16: TRUE 2024-06-26T07:55:00.6082713Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6083554Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6084410Z x 1024(0x400) 2024-06-26T07:55:00.6085257Z y 1024(0x400) 2024-06-26T07:55:00.6086048Z z 1024(0x400) 2024-06-26T07:55:00.6086995Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6088037Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6088775Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6090558Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6091559Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6092478Z FBarrier Max Size: 32 2024-06-26T07:55:00.6093305Z ******* 2024-06-26T07:55:00.6094034Z Agent 5 2024-06-26T07:55:00.6094640Z ******* 2024-06-26T07:55:00.6095314Z Name: gfx90a 2024-06-26T07:55:00.6096589Z Uuid: GPU-1661358e4203fb77 2024-06-26T07:55:00.6097753Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6098851Z Vendor Name: AMD 2024-06-26T07:55:00.6099971Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6101095Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6102075Z Float Round Mode: NEAR 2024-06-26T07:55:00.6103044Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6103976Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6104883Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6105948Z Queue Type: MULTI 2024-06-26T07:55:00.6106825Z Node: 4 2024-06-26T07:55:00.6107875Z Device Type: GPU 2024-06-26T07:55:00.6108685Z Cache Info: 2024-06-26T07:55:00.6109396Z L1: 16(0x10) KB 2024-06-26T07:55:00.6110300Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6111153Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6112061Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6113181Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6114242Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6115295Z BDFID: 4352 2024-06-26T07:55:00.6116367Z Internal Node ID: 4 2024-06-26T07:55:00.6117246Z Compute Unit: 104 2024-06-26T07:55:00.6118163Z SIMDs per CU: 4 2024-06-26T07:55:00.6119127Z Shader Engines: 8 2024-06-26T07:55:00.6120043Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6121058Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6121989Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6122960Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6123949Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6124948Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6125837Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6126583Z x 1024(0x400) 2024-06-26T07:55:00.6127590Z y 1024(0x400) 2024-06-26T07:55:00.6128495Z z 1024(0x400) 2024-06-26T07:55:00.6129397Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6130717Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6131825Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6132615Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6133391Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6134286Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6135093Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6136054Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6137135Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6138248Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6139164Z IOMMU Support:: None 2024-06-26T07:55:00.6139981Z Pool Info: 2024-06-26T07:55:00.6140631Z Pool 1 2024-06-26T07:55:00.6141395Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6142368Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6143491Z Allocatable: TRUE 2024-06-26T07:55:00.6144544Z Alloc Granule: 4KB 2024-06-26T07:55:00.6145505Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6146536Z Accessible by all: FALSE 2024-06-26T07:55:00.6147331Z Pool 2 2024-06-26T07:55:00.6148069Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6149077Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6149947Z Allocatable: TRUE 2024-06-26T07:55:00.6151062Z Alloc Granule: 4KB 2024-06-26T07:55:00.6152051Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6152975Z Accessible by all: FALSE 2024-06-26T07:55:00.6153811Z Pool 3 2024-06-26T07:55:00.6154627Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6155580Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6156487Z Allocatable: TRUE 2024-06-26T07:55:00.6157650Z Alloc Granule: 4KB 2024-06-26T07:55:00.6158815Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6159833Z Accessible by all: FALSE 2024-06-26T07:55:00.6160692Z Pool 4 2024-06-26T07:55:00.6161450Z Segment: GROUP 2024-06-26T07:55:00.6162309Z Size: 64(0x40) KB 2024-06-26T07:55:00.6163245Z Allocatable: FALSE 2024-06-26T07:55:00.6164193Z Alloc Granule: 0KB 2024-06-26T07:55:00.6165100Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6166224Z Accessible by all: FALSE 2024-06-26T07:55:00.6167066Z ISA Info: 2024-06-26T07:55:00.6167645Z ISA 1 2024-06-26T07:55:00.6168747Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6170508Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6171569Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6172618Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6173819Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6174954Z Fast f16: TRUE 2024-06-26T07:55:00.6176035Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6177083Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6177909Z x 1024(0x400) 2024-06-26T07:55:00.6178769Z y 1024(0x400) 2024-06-26T07:55:00.6179615Z z 1024(0x400) 2024-06-26T07:55:00.6180533Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6181381Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6182198Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6183172Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6184208Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6185263Z FBarrier Max Size: 32 2024-06-26T07:55:00.6186072Z ******* 2024-06-26T07:55:00.6186699Z Agent 6 2024-06-26T07:55:00.6187236Z ******* 2024-06-26T07:55:00.6187937Z Name: gfx90a 2024-06-26T07:55:00.6188974Z Uuid: GPU-77ea2eecb6e14f86 2024-06-26T07:55:00.6189882Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6190854Z Vendor Name: AMD 2024-06-26T07:55:00.6191814Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6192802Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6193914Z Float Round Mode: NEAR 2024-06-26T07:55:00.6195344Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6196339Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6197275Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6198224Z Queue Type: MULTI 2024-06-26T07:55:00.6199056Z Node: 5 2024-06-26T07:55:00.6199937Z Device Type: GPU 2024-06-26T07:55:00.6200757Z Cache Info: 2024-06-26T07:55:00.6201447Z L1: 16(0x10) KB 2024-06-26T07:55:00.6202314Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6203296Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6204224Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6205112Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6206096Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6207013Z BDFID: 5120 2024-06-26T07:55:00.6207971Z Internal Node ID: 5 2024-06-26T07:55:00.6208942Z Compute Unit: 104 2024-06-26T07:55:00.6210153Z SIMDs per CU: 4 2024-06-26T07:55:00.6211183Z Shader Engines: 8 2024-06-26T07:55:00.6212221Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6213191Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6214038Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6215094Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6216039Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6216936Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6217838Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6218614Z x 1024(0x400) 2024-06-26T07:55:00.6219421Z y 1024(0x400) 2024-06-26T07:55:00.6220281Z z 1024(0x400) 2024-06-26T07:55:00.6221181Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6222268Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6223220Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6224050Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6224833Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6225661Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6226507Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6227432Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6228400Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6229373Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6230307Z IOMMU Support:: None 2024-06-26T07:55:00.6231094Z Pool Info: 2024-06-26T07:55:00.6231730Z Pool 1 2024-06-26T07:55:00.6232506Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6233612Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6234690Z Allocatable: TRUE 2024-06-26T07:55:00.6235724Z Alloc Granule: 4KB 2024-06-26T07:55:00.6236812Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6238221Z Accessible by all: FALSE 2024-06-26T07:55:00.6239235Z Pool 2 2024-06-26T07:55:00.6240073Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6241008Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6242156Z Allocatable: TRUE 2024-06-26T07:55:00.6243123Z Alloc Granule: 4KB 2024-06-26T07:55:00.6244115Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6245114Z Accessible by all: FALSE 2024-06-26T07:55:00.6245948Z Pool 3 2024-06-26T07:55:00.6246834Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6247967Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6248894Z Allocatable: TRUE 2024-06-26T07:55:00.6249920Z Alloc Granule: 4KB 2024-06-26T07:55:00.6250914Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6251887Z Accessible by all: FALSE 2024-06-26T07:55:00.6252714Z Pool 4 2024-06-26T07:55:00.6253634Z Segment: GROUP 2024-06-26T07:55:00.6254540Z Size: 64(0x40) KB 2024-06-26T07:55:00.6255469Z Allocatable: FALSE 2024-06-26T07:55:00.6256414Z Alloc Granule: 0KB 2024-06-26T07:55:00.6257452Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6258381Z Accessible by all: FALSE 2024-06-26T07:55:00.6259205Z ISA Info: 2024-06-26T07:55:00.6259853Z ISA 1 2024-06-26T07:55:00.6260746Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6261790Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6262833Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6263930Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6264980Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6265951Z Fast f16: TRUE 2024-06-26T07:55:00.6266840Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6267705Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6268609Z x 1024(0x400) 2024-06-26T07:55:00.6269448Z y 1024(0x400) 2024-06-26T07:55:00.6270308Z z 1024(0x400) 2024-06-26T07:55:00.6271265Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6272106Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6272817Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6273723Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6274590Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6275449Z FBarrier Max Size: 32 2024-06-26T07:55:00.6276305Z ******* 2024-06-26T07:55:00.6276887Z Agent 7 2024-06-26T07:55:00.6277409Z ******* 2024-06-26T07:55:00.6278212Z Name: gfx90a 2024-06-26T07:55:00.6279221Z Uuid: GPU-1f7ea6a0f5503a53 2024-06-26T07:55:00.6280145Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6281309Z Vendor Name: AMD 2024-06-26T07:55:00.6282236Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6283117Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6284081Z Float Round Mode: NEAR 2024-06-26T07:55:00.6285053Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6285932Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6286887Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6288042Z Queue Type: MULTI 2024-06-26T07:55:00.6289383Z Node: 6 2024-06-26T07:55:00.6290450Z Device Type: GPU 2024-06-26T07:55:00.6291404Z Cache Info: 2024-06-26T07:55:00.6292140Z L1: 16(0x10) KB 2024-06-26T07:55:00.6292987Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6293885Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6294840Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6295767Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6296973Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6298049Z BDFID: 44544 2024-06-26T07:55:00.6299139Z Internal Node ID: 6 2024-06-26T07:55:00.6300238Z Compute Unit: 104 2024-06-26T07:55:00.6301168Z SIMDs per CU: 4 2024-06-26T07:55:00.6301725Z Shader Engines: 8 2024-06-26T07:55:00.6302259Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6302821Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6303350Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6303819Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6304366Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6304917Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6305371Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6305816Z x 1024(0x400) 2024-06-26T07:55:00.6306309Z y 1024(0x400) 2024-06-26T07:55:00.6306762Z z 1024(0x400) 2024-06-26T07:55:00.6307425Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6308177Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6308773Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6309244Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6309605Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6309987Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6310396Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6310858Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6311310Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6311774Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6312238Z IOMMU Support:: None 2024-06-26T07:55:00.6312680Z Pool Info: 2024-06-26T07:55:00.6312956Z Pool 1 2024-06-26T07:55:00.6313345Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6313898Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6314327Z Allocatable: TRUE 2024-06-26T07:55:00.6314821Z Alloc Granule: 4KB 2024-06-26T07:55:00.6315281Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6315729Z Accessible by all: FALSE 2024-06-26T07:55:00.6316142Z Pool 2 2024-06-26T07:55:00.6316501Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6316913Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6317360Z Allocatable: TRUE 2024-06-26T07:55:00.6317858Z Alloc Granule: 4KB 2024-06-26T07:55:00.6318320Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6318775Z Accessible by all: FALSE 2024-06-26T07:55:00.6319171Z Pool 3 2024-06-26T07:55:00.6319560Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6319978Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6461083Z Allocatable: TRUE 2024-06-26T07:55:00.6462352Z Alloc Granule: 4KB 2024-06-26T07:55:00.6462877Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6463384Z Accessible by all: FALSE 2024-06-26T07:55:00.6463800Z Pool 4 2024-06-26T07:55:00.6464172Z Segment: GROUP 2024-06-26T07:55:00.6464629Z Size: 64(0x40) KB 2024-06-26T07:55:00.6465070Z Allocatable: FALSE 2024-06-26T07:55:00.6465555Z Alloc Granule: 0KB 2024-06-26T07:55:00.6466034Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6466583Z Accessible by all: FALSE 2024-06-26T07:55:00.6467054Z ISA Info: 2024-06-26T07:55:00.6467361Z ISA 1 2024-06-26T07:55:00.6468069Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6468608Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6469154Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6469644Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6470141Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6470592Z Fast f16: TRUE 2024-06-26T07:55:00.6471067Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6471501Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6471892Z x 1024(0x400) 2024-06-26T07:55:00.6472318Z y 1024(0x400) 2024-06-26T07:55:00.6472731Z z 1024(0x400) 2024-06-26T07:55:00.6473220Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6473709Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6474127Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6474614Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6475056Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6475492Z FBarrier Max Size: 32 2024-06-26T07:55:00.6476009Z ******* 2024-06-26T07:55:00.6476229Z Agent 8 2024-06-26T07:55:00.6476450Z ******* 2024-06-26T07:55:00.6476721Z Name: gfx90a 2024-06-26T07:55:00.6477140Z Uuid: GPU-36bcab48b29cafe0 2024-06-26T07:55:00.6477538Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6477943Z Vendor Name: AMD 2024-06-26T07:55:00.6478317Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6478701Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6479088Z Float Round Mode: NEAR 2024-06-26T07:55:00.6479474Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6479861Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6480246Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6480626Z Queue Type: MULTI 2024-06-26T07:55:00.6480979Z Node: 7 2024-06-26T07:55:00.6481338Z Device Type: GPU 2024-06-26T07:55:00.6481660Z Cache Info: 2024-06-26T07:55:00.6482006Z L1: 16(0x10) KB 2024-06-26T07:55:00.6482385Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6482750Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6483115Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6483501Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6483888Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6484248Z BDFID: 45824 2024-06-26T07:55:00.6484623Z Internal Node ID: 7 2024-06-26T07:55:00.6485003Z Compute Unit: 104 2024-06-26T07:55:00.6485369Z SIMDs per CU: 4 2024-06-26T07:55:00.6485744Z Shader Engines: 8 2024-06-26T07:55:00.6486134Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6486538Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6486892Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6487253Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6487638Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6488098Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6488447Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6488759Z x 1024(0x400) 2024-06-26T07:55:00.6489097Z y 1024(0x400) 2024-06-26T07:55:00.6489481Z z 1024(0x400) 2024-06-26T07:55:00.6489938Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6490568Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6491125Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6491538Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6491872Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6492275Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6492675Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6493107Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6493646Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6494252Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6494714Z IOMMU Support:: None 2024-06-26T07:55:00.6495165Z Pool Info: 2024-06-26T07:55:00.6495441Z Pool 1 2024-06-26T07:55:00.6495818Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6496284Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6496746Z Allocatable: TRUE 2024-06-26T07:55:00.6497209Z Alloc Granule: 4KB 2024-06-26T07:55:00.6497669Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6498150Z Accessible by all: FALSE 2024-06-26T07:55:00.6498539Z Pool 2 2024-06-26T07:55:00.6498894Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6499329Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6499753Z Allocatable: TRUE 2024-06-26T07:55:00.6500204Z Alloc Granule: 4KB 2024-06-26T07:55:00.6500654Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6501185Z Accessible by all: FALSE 2024-06-26T07:55:00.6501577Z Pool 3 2024-06-26T07:55:00.6501943Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6502374Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6502823Z Allocatable: TRUE 2024-06-26T07:55:00.6503311Z Alloc Granule: 4KB 2024-06-26T07:55:00.6503846Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6504398Z Accessible by all: FALSE 2024-06-26T07:55:00.6504864Z Pool 4 2024-06-26T07:55:00.6505227Z Segment: GROUP 2024-06-26T07:55:00.6505631Z Size: 64(0x40) KB 2024-06-26T07:55:00.6506002Z Allocatable: FALSE 2024-06-26T07:55:00.6506382Z Alloc Granule: 0KB 2024-06-26T07:55:00.6506759Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6507148Z Accessible by all: FALSE 2024-06-26T07:55:00.6507473Z ISA Info: 2024-06-26T07:55:00.6507702Z ISA 1 2024-06-26T07:55:00.6508077Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6508502Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6508902Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6509299Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6509712Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6510100Z Fast f16: TRUE 2024-06-26T07:55:00.6510482Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6510833Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6511156Z x 1024(0x400) 2024-06-26T07:55:00.6511493Z y 1024(0x400) 2024-06-26T07:55:00.6511831Z z 1024(0x400) 2024-06-26T07:55:00.6512206Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6512542Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6512929Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6513271Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6513605Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6513980Z FBarrier Max Size: 32 2024-06-26T07:55:00.6514313Z ******* 2024-06-26T07:55:00.6514527Z Agent 9 2024-06-26T07:55:00.6514743Z ******* 2024-06-26T07:55:00.6515014Z Name: gfx90a 2024-06-26T07:55:00.6515431Z Uuid: GPU-9ab8502ae477e3b0 2024-06-26T07:55:00.6515824Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6516222Z Vendor Name: AMD 2024-06-26T07:55:00.6516610Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6516982Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6517374Z Float Round Mode: NEAR 2024-06-26T07:55:00.6517768Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6518151Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6518598Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6518975Z Queue Type: MULTI 2024-06-26T07:55:00.6519326Z Node: 8 2024-06-26T07:55:00.6519684Z Device Type: GPU 2024-06-26T07:55:00.6520006Z Cache Info: 2024-06-26T07:55:00.6520282Z L1: 16(0x10) KB 2024-06-26T07:55:00.6520635Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6521005Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6521369Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6521748Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6522136Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6522504Z BDFID: 36352 2024-06-26T07:55:00.6522866Z Internal Node ID: 8 2024-06-26T07:55:00.6523244Z Compute Unit: 104 2024-06-26T07:55:00.6523615Z SIMDs per CU: 4 2024-06-26T07:55:00.6523982Z Shader Engines: 8 2024-06-26T07:55:00.6524370Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6524776Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6525127Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6525484Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6525871Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6526249Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6526592Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6526908Z x 1024(0x400) 2024-06-26T07:55:00.6527242Z y 1024(0x400) 2024-06-26T07:55:00.6527579Z z 1024(0x400) 2024-06-26T07:55:00.6528076Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6528527Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6528928Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6529286Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6529710Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6530175Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6530653Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6531173Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6531657Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6532137Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6532590Z IOMMU Support:: None 2024-06-26T07:55:00.6532967Z Pool Info: 2024-06-26T07:55:00.6533247Z Pool 1 2024-06-26T07:55:00.6533619Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6534079Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6534556Z Allocatable: TRUE 2024-06-26T07:55:00.6535028Z Alloc Granule: 4KB 2024-06-26T07:55:00.6535490Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6535972Z Accessible by all: FALSE 2024-06-26T07:55:00.6536366Z Pool 2 2024-06-26T07:55:00.6536724Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6537238Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6537678Z Allocatable: TRUE 2024-06-26T07:55:00.6538138Z Alloc Granule: 4KB 2024-06-26T07:55:00.6538586Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6539049Z Accessible by all: FALSE 2024-06-26T07:55:00.6539439Z Pool 3 2024-06-26T07:55:00.6539793Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6540237Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6540669Z Allocatable: TRUE 2024-06-26T07:55:00.6541113Z Alloc Granule: 4KB 2024-06-26T07:55:00.6541580Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6542051Z Accessible by all: FALSE 2024-06-26T07:55:00.6542439Z Pool 4 2024-06-26T07:55:00.6542784Z Segment: GROUP 2024-06-26T07:55:00.6543221Z Size: 64(0x40) KB 2024-06-26T07:55:00.6543734Z Allocatable: FALSE 2024-06-26T07:55:00.6544269Z Alloc Granule: 0KB 2024-06-26T07:55:00.6544815Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6545324Z Accessible by all: FALSE 2024-06-26T07:55:00.6545709Z ISA Info: 2024-06-26T07:55:00.6545960Z ISA 1 2024-06-26T07:55:00.6546329Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6546753Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6547151Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6547547Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6547947Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6548330Z Fast f16: TRUE 2024-06-26T07:55:00.6548716Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6549061Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6549451Z x 1024(0x400) 2024-06-26T07:55:00.6549792Z y 1024(0x400) 2024-06-26T07:55:00.6550129Z z 1024(0x400) 2024-06-26T07:55:00.6550498Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6550840Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6551149Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6551487Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6551829Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6552205Z FBarrier Max Size: 32 2024-06-26T07:55:00.6552526Z ******* 2024-06-26T07:55:00.6552746Z Agent 10 2024-06-26T07:55:00.6552964Z ******* 2024-06-26T07:55:00.6553225Z Name: gfx90a 2024-06-26T07:55:00.6553641Z Uuid: GPU-d313938d3c8e3d86 2024-06-26T07:55:00.6554038Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:55:00.6554427Z Vendor Name: AMD 2024-06-26T07:55:00.6554813Z Feature: KERNEL_DISPATCH 2024-06-26T07:55:00.6555255Z Profile: BASE_PROFILE 2024-06-26T07:55:00.6555639Z Float Round Mode: NEAR 2024-06-26T07:55:00.6556033Z Max Queue Number: 128(0x80) 2024-06-26T07:55:00.6556418Z Queue Min Size: 64(0x40) 2024-06-26T07:55:00.6556805Z Queue Max Size: 131072(0x20000) 2024-06-26T07:55:00.6557174Z Queue Type: MULTI 2024-06-26T07:55:00.6557530Z Node: 9 2024-06-26T07:55:00.6557889Z Device Type: GPU 2024-06-26T07:55:00.6558206Z Cache Info: 2024-06-26T07:55:00.6558489Z L1: 16(0x10) KB 2024-06-26T07:55:00.6558842Z L2: 8192(0x2000) KB 2024-06-26T07:55:00.6559193Z Chip ID: 29708(0x740c) 2024-06-26T07:55:00.6559565Z ASIC Revision: 1(0x1) 2024-06-26T07:55:00.6559947Z Cacheline Size: 64(0x40) 2024-06-26T07:55:00.6560327Z Max Clock Freq. (MHz): 1700 2024-06-26T07:55:00.6560699Z BDFID: 37632 2024-06-26T07:55:00.6561066Z Internal Node ID: 9 2024-06-26T07:55:00.6561433Z Compute Unit: 104 2024-06-26T07:55:00.6561806Z SIMDs per CU: 4 2024-06-26T07:55:00.6562182Z Shader Engines: 8 2024-06-26T07:55:00.6562577Z Shader Arrs. per Eng.: 1 2024-06-26T07:55:00.6562969Z WatchPts on Addr. Ranges:4 2024-06-26T07:55:00.6563326Z Features: KERNEL_DISPATCH 2024-06-26T07:55:00.6563681Z Fast F16 Operation: TRUE 2024-06-26T07:55:00.6564060Z Wavefront Size: 64(0x40) 2024-06-26T07:55:00.6564446Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6564786Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6565088Z x 1024(0x400) 2024-06-26T07:55:00.6565424Z y 1024(0x400) 2024-06-26T07:55:00.6565758Z z 1024(0x400) 2024-06-26T07:55:00.6566184Z Max Waves Per CU: 32(0x20) 2024-06-26T07:55:00.6566631Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:55:00.6567029Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6567368Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6567652Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6568039Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6568385Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6568750Z Max fbarriers/Workgrp: 32 2024-06-26T07:55:00.6569155Z Packet Processor uCode:: 78 2024-06-26T07:55:00.6569622Z SDMA engine uCode:: 8 2024-06-26T07:55:00.6570129Z IOMMU Support:: None 2024-06-26T07:55:00.6570592Z Pool Info: 2024-06-26T07:55:00.6570924Z Pool 1 2024-06-26T07:55:00.6571313Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:55:00.6571775Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6572228Z Allocatable: TRUE 2024-06-26T07:55:00.6572780Z Alloc Granule: 4KB 2024-06-26T07:55:00.6573255Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6573732Z Accessible by all: FALSE 2024-06-26T07:55:00.6574126Z Pool 2 2024-06-26T07:55:00.6574483Z Segment: GLOBAL; FLAGS: 2024-06-26T07:55:00.6574921Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6575357Z Allocatable: TRUE 2024-06-26T07:55:00.6575817Z Alloc Granule: 4KB 2024-06-26T07:55:00.6576279Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6576748Z Accessible by all: FALSE 2024-06-26T07:55:00.6577132Z Pool 3 2024-06-26T07:55:00.6577493Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:55:00.6577933Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:55:00.6578363Z Allocatable: TRUE 2024-06-26T07:55:00.6578811Z Alloc Granule: 4KB 2024-06-26T07:55:00.6579266Z Alloc Alignment: 4KB 2024-06-26T07:55:00.6579723Z Accessible by all: FALSE 2024-06-26T07:55:00.6580106Z Pool 4 2024-06-26T07:55:00.6580455Z Segment: GROUP 2024-06-26T07:55:00.6580913Z Size: 64(0x40) KB 2024-06-26T07:55:00.6581344Z Allocatable: FALSE 2024-06-26T07:55:00.6581794Z Alloc Granule: 0KB 2024-06-26T07:55:00.6582252Z Alloc Alignment: 0KB 2024-06-26T07:55:00.6582704Z Accessible by all: FALSE 2024-06-26T07:55:00.6583093Z ISA Info: 2024-06-26T07:55:00.6583397Z ISA 1 2024-06-26T07:55:00.6583909Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:55:00.6584499Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:55:00.6585060Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:55:00.6585549Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6586069Z Default Rounding Mode: NEAR 2024-06-26T07:55:00.6586453Z Fast f16: TRUE 2024-06-26T07:55:00.6586833Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:55:00.6587182Z Workgroup Max Size per Dimension: 2024-06-26T07:55:00.6587506Z x 1024(0x400) 2024-06-26T07:55:00.6587844Z y 1024(0x400) 2024-06-26T07:55:00.6588181Z z 1024(0x400) 2024-06-26T07:55:00.6588559Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:55:00.6588904Z Grid Max Size per Dimension: 2024-06-26T07:55:00.6589201Z x 4294967295(0xffffffff) 2024-06-26T07:55:00.6589542Z y 4294967295(0xffffffff) 2024-06-26T07:55:00.6589885Z z 4294967295(0xffffffff) 2024-06-26T07:55:00.6590256Z FBarrier Max Size: 32 2024-06-26T07:55:00.6590595Z *** Done *** 2024-06-26T07:55:00.6609012Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-06-26T07:55:00.6609521Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-06-26T07:55:00.6610513Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-06-26T07:55:00.6611354Z if [[ $ngpu -eq 0 ]]; then 2024-06-26T07:55:00.6611804Z  echo "Error: Failed to detect any GPUs on the runner" 2024-06-26T07:55:00.6612244Z  echo "$msg" 2024-06-26T07:55:00.6612520Z  exit 1 2024-06-26T07:55:00.6612787Z fi 2024-06-26T07:55:00.6613083Z if [[ $ngpu -eq 1 ]]; then 2024-06-26T07:55:00.6613677Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-06-26T07:55:00.6614254Z  echo "$msg" 2024-06-26T07:55:00.6614550Z  exit 1 2024-06-26T07:55:00.6614785Z fi 2024-06-26T07:55:00.6642031Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.6642374Z env: 2024-06-26T07:55:00.6642564Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.6642848Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.6643150Z ##[endgroup] 2024-06-26T07:55:00.7839645Z Prepare all required actions 2024-06-26T07:55:00.7902858Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-06-26T07:55:00.7903594Z with: 2024-06-26T07:55:00.7904043Z diskspace-cutoff: 70 2024-06-26T07:55:00.7904556Z env: 2024-06-26T07:55:00.7904982Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.7905661Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.7906523Z ##[endgroup] 2024-06-26T07:55:00.8013855Z ##[group]Run set -ex 2024-06-26T07:55:00.8014412Z set -ex 2024-06-26T07:55:00.8014869Z diskspace_cutoff=70 2024-06-26T07:55:00.8015628Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-06-26T07:55:00.8016878Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-06-26T07:55:00.8018829Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-06-26T07:55:00.8020462Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-06-26T07:55:00.8021238Z  docker system prune -af 2024-06-26T07:55:00.8022348Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-06-26T07:55:00.8023613Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-06-26T07:55:00.8024924Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-06-26T07:55:00.8026348Z  echo "$msg" 2024-06-26T07:55:00.8026878Z  exit 1 2024-06-26T07:55:00.8027356Z  else 2024-06-26T07:55:00.8027950Z  difference=$((diskspace - diskspace_new)) 2024-06-26T07:55:00.8028819Z  echo "Diskspace saved: $difference percent" 2024-06-26T07:55:00.8029545Z  fi 2024-06-26T07:55:00.8029982Z fi 2024-06-26T07:55:00.8071091Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.8071959Z env: 2024-06-26T07:55:00.8072357Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.8072968Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.8073610Z ##[endgroup] 2024-06-26T07:55:00.8139431Z + diskspace_cutoff=70 2024-06-26T07:55:00.8144209Z ++ docker info -f '{{.DockerRootDir}}' 2024-06-26T07:55:00.8817535Z + docker_root_dir=/media/4TB/docker-rootless 2024-06-26T07:55:00.8825514Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-06-26T07:55:00.8828301Z ++ sed -n 2p 2024-06-26T07:55:00.8828789Z ++ sed s/%// 2024-06-26T07:55:00.8829474Z ++ sed 's/ //' 2024-06-26T07:55:00.8854416Z + diskspace=24 2024-06-26T07:55:00.8856108Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-06-26T07:55:00.8857756Z + [[ 24 -ge 70 ]] 2024-06-26T07:55:00.8917686Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:55:00.8918905Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:55:00.8920001Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:55:00.8966972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.8968145Z env: 2024-06-26T07:55:00.8968613Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.8969306Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.8970032Z ##[endgroup] 2024-06-26T07:55:00.9121686Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-06-26T07:55:00.9123434Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-06-26T07:55:00.9125421Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-06-26T07:55:00.9169856Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:00.9170873Z env: 2024-06-26T07:55:00.9171399Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.9172099Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.9172834Z ##[endgroup] 2024-06-26T07:55:00.9368973Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-06-26T07:55:00.9369848Z with: 2024-06-26T07:55:00.9370667Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-06-26T07:55:00.9371849Z aws-region: us-east-1 2024-06-26T07:55:00.9372561Z role-duration-seconds: 18000 2024-06-26T07:55:00.9373318Z audience: sts.amazonaws.com 2024-06-26T07:55:00.9373887Z env: 2024-06-26T07:55:00.9374308Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:00.9374978Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:00.9376255Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:00.9377397Z ##[endgroup] 2024-06-26T07:55:01.7999580Z Assuming role with OIDC 2024-06-26T07:55:02.1858864Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-06-26T07:55:02.3004705Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-06-26T07:55:02.3005567Z with: 2024-06-26T07:55:02.3006085Z mask-password: true 2024-06-26T07:55:02.3006731Z registry-type: private 2024-06-26T07:55:02.3007379Z skip-logout: false 2024-06-26T07:55:02.3008070Z env: 2024-06-26T07:55:02.3008576Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:02.3009396Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:02.3011436Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:02.3012773Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:02.3013464Z AWS_REGION: us-east-1 2024-06-26T07:55:02.3014721Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:02.3015731Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:02.3030472Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:02.3031108Z ##[endgroup] 2024-06-26T07:55:02.9634748Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:03.9857295Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:55:03.9858303Z with: 2024-06-26T07:55:03.9859654Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:03.9861202Z docker-build-dir: .ci/docker 2024-06-26T07:55:03.9861786Z working-directory: . 2024-06-26T07:55:03.9862539Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:03.9863395Z force-push: false 2024-06-26T07:55:03.9863872Z env: 2024-06-26T07:55:03.9864294Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:03.9864944Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:03.9866146Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:03.9867446Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:03.9868123Z AWS_REGION: us-east-1 2024-06-26T07:55:03.9869000Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:03.9869829Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:03.9881946Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:03.9882495Z ##[endgroup] 2024-06-26T07:55:03.9911563Z ##[group]Run set -ex 2024-06-26T07:55:03.9912117Z set -ex 2024-06-26T07:55:03.9912570Z  2024-06-26T07:55:03.9913458Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:55:03.9914932Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:55:03.9916200Z # job could then download the pre-built image as usual 2024-06-26T07:55:03.9917365Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:55:03.9918420Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9919399Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9920263Z  2024-06-26T07:55:03.9921079Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:55:03.9922056Z  exit 0 2024-06-26T07:55:03.9922530Z else 2024-06-26T07:55:03.9923106Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9923801Z fi 2024-06-26T07:55:03.9924227Z  2024-06-26T07:55:03.9924957Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:55:03.9926304Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:55:03.9927520Z  # use it as it is, but first let's extract the tag 2024-06-26T07:55:03.9928749Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:55:03.9929878Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9930962Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9931827Z else 2024-06-26T07:55:03.9932507Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:55:03.9933519Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9934900Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:03.9936065Z fi 2024-06-26T07:55:03.9982093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:03.9983105Z env: 2024-06-26T07:55:03.9983544Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:03.9984197Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:03.9985371Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:03.9986475Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:03.9987061Z AWS_REGION: us-east-1 2024-06-26T07:55:03.9987697Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:03.9988503Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:04.0000702Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:04.0001272Z REPO_NAME: pytorch 2024-06-26T07:55:04.0002679Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0004190Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:55:04.0004964Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:04.0005767Z ##[endgroup] 2024-06-26T07:55:04.0074281Z + [[ ! -d .ci/docker ]] 2024-06-26T07:55:04.0074927Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:55:04.0075548Z + echo skip=false 2024-06-26T07:55:04.0077995Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:55:04.0086406Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0088084Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:55:04.0118508Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0119527Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0123995Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0186362Z ##[group]Run set +e 2024-06-26T07:55:04.0187050Z set +e 2024-06-26T07:55:04.0187680Z set -x 2024-06-26T07:55:04.0188295Z  2024-06-26T07:55:04.0188858Z login() { 2024-06-26T07:55:04.0190050Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:55:04.0191302Z } 2024-06-26T07:55:04.0191795Z  2024-06-26T07:55:04.0192302Z retry () { 2024-06-26T07:55:04.0193037Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:55:04.0193850Z } 2024-06-26T07:55:04.0194344Z  2024-06-26T07:55:04.0194906Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:55:04.0195650Z  2024-06-26T07:55:04.0196483Z # Check if image already exists, if it does then skip building it 2024-06-26T07:55:04.0197724Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:55:04.0198601Z  exit 0 2024-06-26T07:55:04.0199184Z fi 2024-06-26T07:55:04.0199678Z  2024-06-26T07:55:04.0200576Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:55:04.0202019Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:55:04.0203318Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:55:04.0204544Z  # if we're on the base branch then use the parent commit 2024-06-26T07:55:04.0205604Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:55:04.0206358Z else 2024-06-26T07:55:04.0207181Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:55:04.0208534Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:55:04.0209409Z fi 2024-06-26T07:55:04.0209951Z  2024-06-26T07:55:04.0210619Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:55:04.0211663Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:04.0212780Z  2024-06-26T07:55:04.0213934Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:55:04.0215257Z  exit 0 2024-06-26T07:55:04.0215784Z fi 2024-06-26T07:55:04.0216260Z  2024-06-26T07:55:04.0217027Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:55:04.0218934Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:55:04.0220351Z  exit 1 2024-06-26T07:55:04.0220866Z fi 2024-06-26T07:55:04.0221338Z  2024-06-26T07:55:04.0222184Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:55:04.0223858Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:55:04.0225375Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:55:04.0227097Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:55:04.0229274Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:55:04.0230407Z fi 2024-06-26T07:55:04.0230885Z  2024-06-26T07:55:04.0231503Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:04.0283578Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:04.0284472Z env: 2024-06-26T07:55:04.0284965Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:04.0285710Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:04.0287040Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:04.0288362Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:04.0289031Z AWS_REGION: us-east-1 2024-06-26T07:55:04.0289820Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:04.0290918Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:04.0304731Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:04.0305382Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:55:04.0306193Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:55:04.0308182Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0310083Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:04.0311146Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:04.0312058Z ##[endgroup] 2024-06-26T07:55:04.0385421Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:04.0386470Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:04.0390481Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:55:04.0392517Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:05.8124429Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-06-26T07:55:05.8126120Z Configure a credential helper to remove this warning. See 2024-06-26T07:55:05.8128294Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:55:05.8129366Z 2024-06-26T07:55:05.8129791Z Login Succeeded 2024-06-26T07:55:05.8166560Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:07.7140029Z { 2024-06-26T07:55:07.7140626Z "schemaVersion": 2, 2024-06-26T07:55:07.7141663Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:55:07.7142730Z "config": { 2024-06-26T07:55:07.7143662Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:55:07.7144641Z "size": 23399, 2024-06-26T07:55:07.7145623Z "digest": "sha256:bca0f386de16405273b5d02d10f58060215ba34e910eba5e4b3c1f9ab520384f" 2024-06-26T07:55:07.7150531Z }, 2024-06-26T07:55:07.7150988Z "layers": [ 2024-06-26T07:55:07.7151476Z { 2024-06-26T07:55:07.7152253Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7153258Z "size": 28584223, 2024-06-26T07:55:07.7154265Z "digest": "sha256:560c024910bebac6b404791af28ebd48a8289303b8377d17b67ffdfe52754f2a" 2024-06-26T07:55:07.7155390Z }, 2024-06-26T07:55:07.7155823Z { 2024-06-26T07:55:07.7156784Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7157781Z "size": 1823, 2024-06-26T07:55:07.7158716Z "digest": "sha256:921e3146085a56309cf7d30ff1e12394f82fe7355735e6680b29592eb9a9e3bd" 2024-06-26T07:55:07.7159769Z }, 2024-06-26T07:55:07.7160192Z { 2024-06-26T07:55:07.7160938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7161890Z "size": 312547923, 2024-06-26T07:55:07.7162859Z "digest": "sha256:245942f677ca3199f8373d5884ee77b05071928a67fd2b44ecc4d75ffa3f0d31" 2024-06-26T07:55:07.7163933Z }, 2024-06-26T07:55:07.7164347Z { 2024-06-26T07:55:07.7165083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7166039Z "size": 804, 2024-06-26T07:55:07.7167002Z "digest": "sha256:18629a40ab9dca02fc8459fa821a7f1d946947e126cba6069b6c53ac0d3a5565" 2024-06-26T07:55:07.7168277Z }, 2024-06-26T07:55:07.7168715Z { 2024-06-26T07:55:07.7169466Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7170799Z "size": 106, 2024-06-26T07:55:07.7171793Z "digest": "sha256:7bbb4b1cb47fe104088ed1440f6bc9123fc93bcdbf88caabd6a96cc30d2d636c" 2024-06-26T07:55:07.7172947Z }, 2024-06-26T07:55:07.7173388Z { 2024-06-26T07:55:07.7174141Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7175104Z "size": 546, 2024-06-26T07:55:07.7176059Z "digest": "sha256:a3226953fcd0a3979a772aefcc9247fccc946318d7c28ec2dcda19e5c62aa866" 2024-06-26T07:55:07.7177146Z }, 2024-06-26T07:55:07.7177562Z { 2024-06-26T07:55:07.7178303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7179256Z "size": 1287, 2024-06-26T07:55:07.7180233Z "digest": "sha256:8dce30fddc30c9862f970bce0b6cb217c1cb79a824ae6f2da651a82e34081cb5" 2024-06-26T07:55:07.7181590Z + exit 0 2024-06-26T07:55:07.7182035Z }, 2024-06-26T07:55:07.7182454Z { 2024-06-26T07:55:07.7183196Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7184170Z "size": 3235, 2024-06-26T07:55:07.7185114Z "digest": "sha256:20bd81a4d13529725e256b87aab1bccbd19559f7906a9fa164a2fd2c0555651b" 2024-06-26T07:55:07.7186199Z }, 2024-06-26T07:55:07.7186609Z { 2024-06-26T07:55:07.7187410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7188534Z "size": 1906, 2024-06-26T07:55:07.7189505Z "digest": "sha256:fc467057acaae968f2bbd54965b0d2f287b6d603d9b6f2d9eb1c84439c515a09" 2024-06-26T07:55:07.7190603Z }, 2024-06-26T07:55:07.7191016Z { 2024-06-26T07:55:07.7191748Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7192703Z "size": 702, 2024-06-26T07:55:07.7193619Z "digest": "sha256:3b3a814f5f822723313bc1047b3df307e6a09c140681bfde686ce5b35590964d" 2024-06-26T07:55:07.7194670Z }, 2024-06-26T07:55:07.7195081Z { 2024-06-26T07:55:07.7195807Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7196759Z "size": 2482288483, 2024-06-26T07:55:07.7197729Z "digest": "sha256:e07f9826026326434a2744260dbbc1b553ddd6b6ee295f0e830b731100be4a8a" 2024-06-26T07:55:07.7198770Z }, 2024-06-26T07:55:07.7199190Z { 2024-06-26T07:55:07.7199933Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7200885Z "size": 379, 2024-06-26T07:55:07.7201820Z "digest": "sha256:00f54c5ba69db40d960387393fe9fa0ea0a70c748a58cc91dc7ca7da289f5787" 2024-06-26T07:55:07.7202891Z }, 2024-06-26T07:55:07.7203513Z { 2024-06-26T07:55:07.7204246Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7205192Z "size": 11135, 2024-06-26T07:55:07.7206155Z "digest": "sha256:8d28edfe9f02c23a5e5a9c08fba6413d5b3a8217ba42c6245ff5eb17ce740357" 2024-06-26T07:55:07.7207246Z }, 2024-06-26T07:55:07.7207659Z { 2024-06-26T07:55:07.7208485Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7209436Z "size": 505, 2024-06-26T07:55:07.7210691Z "digest": "sha256:e98f9d56250ad4975a04ffdde8d5e9f1f96224a55afd0985b807360c7107bc60" 2024-06-26T07:55:07.7211940Z }, 2024-06-26T07:55:07.7212353Z { 2024-06-26T07:55:07.7213075Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7214016Z "size": 121477660, 2024-06-26T07:55:07.7214984Z "digest": "sha256:bd8524a2180116eee5cc368f53ff36a8b4be9aa804b04384968ac26117deb699" 2024-06-26T07:55:07.7216056Z }, 2024-06-26T07:55:07.7216468Z { 2024-06-26T07:55:07.7217209Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7218161Z "size": 109, 2024-06-26T07:55:07.7219101Z "digest": "sha256:1cb3cdd390730bf22dec03918b4a677a1ba8c2ab93359d27bd38f806550303eb" 2024-06-26T07:55:07.7220174Z }, 2024-06-26T07:55:07.7220587Z { 2024-06-26T07:55:07.7221316Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7222261Z "size": 490, 2024-06-26T07:55:07.7223193Z "digest": "sha256:79c93fc6c46a95a3100a3eeb7e845a87bfab3d26858ad96531fac796593fc8df" 2024-06-26T07:55:07.7224250Z }, 2024-06-26T07:55:07.7224658Z { 2024-06-26T07:55:07.7225377Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7226309Z "size": 296, 2024-06-26T07:55:07.7227269Z "digest": "sha256:88f8413d04932e8f4d4733d232b1be7fd2aaaeb8df068e467f5e34ea67459c2a" 2024-06-26T07:55:07.7228539Z }, 2024-06-26T07:55:07.7228987Z { 2024-06-26T07:55:07.7229716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7230674Z "size": 103, 2024-06-26T07:55:07.7231619Z "digest": "sha256:ded58fc18fa3d2913bca5f4d5da01f5c23f3a8187a9ea49fa85a3cd9e94dff9b" 2024-06-26T07:55:07.7232702Z }, 2024-06-26T07:55:07.7233116Z { 2024-06-26T07:55:07.7233840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7234775Z "size": 1473, 2024-06-26T07:55:07.7235685Z "digest": "sha256:307295219f1b7250b249f937cd740d02bf885c87681de3d225d48f23b61cb5c4" 2024-06-26T07:55:07.7236740Z }, 2024-06-26T07:55:07.7237155Z { 2024-06-26T07:55:07.7237879Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7238841Z "size": 446755444, 2024-06-26T07:55:07.7239800Z "digest": "sha256:31160c0f6080b82bbe92996f523b3cf589b7c60a5ef4ef955844ab0c8f682130" 2024-06-26T07:55:07.7240851Z }, 2024-06-26T07:55:07.7241265Z { 2024-06-26T07:55:07.7241990Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7242941Z "size": 159, 2024-06-26T07:55:07.7243900Z "digest": "sha256:16c93ba1a8a05bf4221dd712d7026ea88aac19e4d7255cbaac2a4a5aea6ba8d8" 2024-06-26T07:55:07.7245144Z }, 2024-06-26T07:55:07.7245560Z { 2024-06-26T07:55:07.7246297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7247236Z "size": 1639, 2024-06-26T07:55:07.7248396Z "digest": "sha256:cfda4f639d5138d9cc3934ae95f0f2e2427f870473fb6f059e6606944ea87f59" 2024-06-26T07:55:07.7249460Z }, 2024-06-26T07:55:07.7249925Z { 2024-06-26T07:55:07.7250794Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7251926Z "size": 6890884018, 2024-06-26T07:55:07.7252877Z "digest": "sha256:6361109dc80e008283442038e8e7d638e3fe6a2c3cfb47feb3e85a562c67e938" 2024-06-26T07:55:07.7253908Z }, 2024-06-26T07:55:07.7254322Z { 2024-06-26T07:55:07.7255050Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7255993Z "size": 105, 2024-06-26T07:55:07.7257098Z "digest": "sha256:ba61eb599e61b89f6435f49a1d43b0f40b5688b2028232b233bed965b68e7984" 2024-06-26T07:55:07.7258146Z }, 2024-06-26T07:55:07.7258556Z { 2024-06-26T07:55:07.7259276Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7260215Z "size": 841, 2024-06-26T07:55:07.7261158Z "digest": "sha256:9a76bd9d77d08ff3ff9d3ccbbc86022416f3811ba02d3607b0f15ae64de795e2" 2024-06-26T07:55:07.7262228Z }, 2024-06-26T07:55:07.7262639Z { 2024-06-26T07:55:07.7263509Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7264462Z "size": 320864358, 2024-06-26T07:55:07.7265434Z "digest": "sha256:8c799c93ed28d727cd3502d86e4bd89c14e769129b89d21396e83b57990552b4" 2024-06-26T07:55:07.7266481Z }, 2024-06-26T07:55:07.7266891Z { 2024-06-26T07:55:07.7267717Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7268830Z "size": 111, 2024-06-26T07:55:07.7269767Z "digest": "sha256:39d4dedc3fbed7fed0f9f56b7ba60336636a16f8a85c6b6b2a2679464dd5e486" 2024-06-26T07:55:07.7270852Z }, 2024-06-26T07:55:07.7271263Z { 2024-06-26T07:55:07.7271989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7272920Z "size": 166, 2024-06-26T07:55:07.7273848Z "digest": "sha256:51d571abb22add0fa2594fae2990ac11a20a66d0d96d7de45592827a2cffcee2" 2024-06-26T07:55:07.7274907Z }, 2024-06-26T07:55:07.7275316Z { 2024-06-26T07:55:07.7276054Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7276986Z "size": 2473089, 2024-06-26T07:55:07.7277942Z "digest": "sha256:27b56b3a930360556e1a1e5ea962ada71bdbf70528119999a18aa2e7d835309e" 2024-06-26T07:55:07.7278982Z }, 2024-06-26T07:55:07.7279393Z { 2024-06-26T07:55:07.7280111Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7281042Z "size": 107, 2024-06-26T07:55:07.7281965Z "digest": "sha256:216d768f9c9fbaec4f387d047d9a5e62ea9b2c0a55bf472d5b18ff24b72a0b52" 2024-06-26T07:55:07.7283053Z }, 2024-06-26T07:55:07.7283469Z { 2024-06-26T07:55:07.7284193Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7285136Z "size": 565, 2024-06-26T07:55:07.7286061Z "digest": "sha256:4c9de5de20745bb689da6d7645d835b6d17ec564a514f600fc6730ecd7a280ae" 2024-06-26T07:55:07.7287133Z }, 2024-06-26T07:55:07.7287548Z { 2024-06-26T07:55:07.7288332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7289298Z "size": 43195735, 2024-06-26T07:55:07.7290378Z "digest": "sha256:8f8928d8aa17882f7efe0def2a390affbb9118117d922da022e18d1a1d0c23d7" 2024-06-26T07:55:07.7291646Z }, 2024-06-26T07:55:07.7292111Z { 2024-06-26T07:55:07.7292842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7293781Z "size": 106, 2024-06-26T07:55:07.7294689Z "digest": "sha256:a23f85e4431b57d03c07306d4086f22a3c113c4312f7c4ff4b0aff5afc5b7dbd" 2024-06-26T07:55:07.7295755Z }, 2024-06-26T07:55:07.7296163Z { 2024-06-26T07:55:07.7296885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7297820Z "size": 295, 2024-06-26T07:55:07.7298729Z "digest": "sha256:d91162c9504032deb0f324ff2c0627872cf8748de2b3eab2ed05656164a59c7b" 2024-06-26T07:55:07.7299779Z }, 2024-06-26T07:55:07.7300193Z { 2024-06-26T07:55:07.7300910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7301853Z "size": 88289, 2024-06-26T07:55:07.7302816Z "digest": "sha256:bd9dcf0c6e65a9d05c39742c56d4815ac6c634de25c719053e503ea85bfe867c" 2024-06-26T07:55:07.7303890Z }, 2024-06-26T07:55:07.7304297Z { 2024-06-26T07:55:07.7305022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7305961Z "size": 106, 2024-06-26T07:55:07.7306895Z "digest": "sha256:26a5181f3bebc2d99cdc5912eabd04b48a60fb8fa3a4dc13224e95bfb2e69322" 2024-06-26T07:55:07.7308108Z }, 2024-06-26T07:55:07.7308788Z { 2024-06-26T07:55:07.7309529Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7310456Z "size": 1213, 2024-06-26T07:55:07.7311388Z "digest": "sha256:9c2340adf84883a752a8173f6ea8ab72cf26ba60f7cf9f6145c54a889b97437d" 2024-06-26T07:55:07.7312442Z }, 2024-06-26T07:55:07.7312852Z { 2024-06-26T07:55:07.7313577Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7314519Z "size": 702, 2024-06-26T07:55:07.7315610Z "digest": "sha256:3b3a814f5f822723313bc1047b3df307e6a09c140681bfde686ce5b35590964d" 2024-06-26T07:55:07.7316667Z }, 2024-06-26T07:55:07.7317090Z { 2024-06-26T07:55:07.7317818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7318759Z "size": 141, 2024-06-26T07:55:07.7319670Z "digest": "sha256:4b86f74e73703501b952290eb8b880c5af00cf5132ce7d8785206f835f2ff022" 2024-06-26T07:55:07.7320709Z }, 2024-06-26T07:55:07.7321120Z { 2024-06-26T07:55:07.7321856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7322778Z "size": 120, 2024-06-26T07:55:07.7323717Z "digest": "sha256:84848c108623dac27ec9cc491baf1dfa8faddb1d56e8daffa50ead02764a6ab2" 2024-06-26T07:55:07.7324794Z }, 2024-06-26T07:55:07.7325201Z { 2024-06-26T07:55:07.7325922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7326872Z "size": 2025667418, 2024-06-26T07:55:07.7327907Z "digest": "sha256:141c28ab8d763cb117a87f4e404572516db272fc7c3a68288a47fa3ecd9c0a16" 2024-06-26T07:55:07.7328962Z }, 2024-06-26T07:55:07.7329375Z { 2024-06-26T07:55:07.7330200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7331299Z "size": 177, 2024-06-26T07:55:07.7332315Z "digest": "sha256:4029cc9e175af35760376cb5fc47f06bc2012f7cc3c4cf9f35ed84d632d50309" 2024-06-26T07:55:07.7333369Z }, 2024-06-26T07:55:07.7333777Z { 2024-06-26T07:55:07.7334499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7335442Z "size": 208, 2024-06-26T07:55:07.7336371Z "digest": "sha256:69a8d39d41daa51bd2beaa2d776d56a9c17a35591544c5b6815810358b72bec1" 2024-06-26T07:55:07.7337433Z }, 2024-06-26T07:55:07.7337843Z { 2024-06-26T07:55:07.7338570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7339502Z "size": 702, 2024-06-26T07:55:07.7340416Z "digest": "sha256:3b3a814f5f822723313bc1047b3df307e6a09c140681bfde686ce5b35590964d" 2024-06-26T07:55:07.7341474Z }, 2024-06-26T07:55:07.7341890Z { 2024-06-26T07:55:07.7342620Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7343570Z "size": 629, 2024-06-26T07:55:07.7344494Z "digest": "sha256:21a593d3be236b543d1e00144a09ce0d286d9d3accc1998bc069fba60c59b5f7" 2024-06-26T07:55:07.7345545Z }, 2024-06-26T07:55:07.7345953Z { 2024-06-26T07:55:07.7346671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7347688Z "size": 232636059, 2024-06-26T07:55:07.7348869Z "digest": "sha256:eeacdb5e606ef76a1d929d60a47059dfde9b9de94fb0afdb79a7624f32de4626" 2024-06-26T07:55:07.7349977Z }, 2024-06-26T07:55:07.7350388Z { 2024-06-26T07:55:07.7351103Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7352041Z "size": 1842, 2024-06-26T07:55:07.7352962Z "digest": "sha256:a7f852cf8f06f914c6453053c90158db80418d335c1d45d3428ad9e046ceb406" 2024-06-26T07:55:07.7354010Z }, 2024-06-26T07:55:07.7354421Z { 2024-06-26T07:55:07.7355156Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:55:07.7356097Z "size": 15983016, 2024-06-26T07:55:07.7357068Z "digest": "sha256:4a199161edb3cde63797dfbba73b8e820fd8d68e9537e7f4706f278dea01fdac" 2024-06-26T07:55:07.7358121Z } 2024-06-26T07:55:07.7358530Z ] 2024-06-26T07:55:07.7358947Z } 2024-06-26T07:55:07.7567692Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:55:07.7568944Z with: 2024-06-26T07:55:07.7570253Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:07.7571947Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:07.7572735Z env: 2024-06-26T07:55:07.7573161Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:07.7573801Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:07.7574961Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:07.7576063Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:07.7576650Z AWS_REGION: us-east-1 2024-06-26T07:55:07.7577301Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:07.7578115Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:07.7590185Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:07.7590719Z ##[endgroup] 2024-06-26T07:55:07.7616290Z ##[group]Run set -x 2024-06-26T07:55:07.7616807Z set -x 2024-06-26T07:55:07.7617278Z set +e 2024-06-26T07:55:07.7617713Z  2024-06-26T07:55:07.7618135Z login() { 2024-06-26T07:55:07.7619166Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:55:07.7620263Z } 2024-06-26T07:55:07.7620685Z  2024-06-26T07:55:07.7621111Z retry () { 2024-06-26T07:55:07.7621714Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:55:07.7622402Z } 2024-06-26T07:55:07.7622822Z  2024-06-26T07:55:07.7623312Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:55:07.7623950Z  2024-06-26T07:55:07.7624355Z set -e 2024-06-26T07:55:07.7625079Z # ignore output since only exit code is used for conditional 2024-06-26T07:55:07.7626154Z # only pull docker image if it's not available locally 2024-06-26T07:55:07.7627353Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:55:07.7628437Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:55:07.7629108Z fi 2024-06-26T07:55:07.7668601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:07.7668958Z env: 2024-06-26T07:55:07.7669156Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:07.7669456Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:07.7669994Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:07.7670503Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:07.7670778Z AWS_REGION: us-east-1 2024-06-26T07:55:07.7671073Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:07.7671440Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:07.7676978Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:07.7677639Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:07.7678403Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:07.7678776Z ##[endgroup] 2024-06-26T07:55:07.7721144Z + set +e 2024-06-26T07:55:07.7722390Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:07.7723592Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:07.7727623Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:55:07.7729503Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:55:09.5175363Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-06-26T07:55:09.5176902Z Configure a credential helper to remove this warning. See 2024-06-26T07:55:09.5178555Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:55:09.5179502Z 2024-06-26T07:55:09.5179751Z Login Succeeded 2024-06-26T07:55:09.5212473Z + set -e 2024-06-26T07:55:09.5215252Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:55:09.5537382Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:55:09.5538789Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:55:09.5539997Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:55:09.5541163Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:55:09.5592150Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:09.5593048Z env: 2024-06-26T07:55:09.5593548Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:09.5594297Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:09.5595637Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:09.5596883Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:09.5597550Z AWS_REGION: us-east-1 2024-06-26T07:55:09.5598465Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:09.5599491Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:09.5613481Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:09.5614096Z ##[endgroup] 2024-06-26T07:55:10.0265601Z Requirement already satisfied: psutil==5.9.1 in /opt/anaconda3/lib/python3.11/site-packages (5.9.1) 2024-06-26T07:55:10.0273492Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /opt/anaconda3/lib/python3.11/site-packages (11.525.84) 2024-06-26T07:55:11.5741455Z Prepare all required actions 2024-06-26T07:55:11.5742595Z Getting action download info 2024-06-26T07:55:11.7705657Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:55:12.8807453Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:55:13.6984083Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:55:13.6984936Z with: 2024-06-26T07:55:13.6985461Z name: linux-focal-rocm6.1-py3.8 2024-06-26T07:55:13.6986167Z s3-bucket: gha-artifacts 2024-06-26T07:55:13.6986771Z env: 2024-06-26T07:55:13.6987245Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:13.6987967Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:13.6989299Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:13.6990548Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:13.6991273Z AWS_REGION: us-east-1 2024-06-26T07:55:13.6991999Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:13.6993004Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:13.7006658Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:13.7007262Z ##[endgroup] 2024-06-26T07:55:13.7066101Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:55:13.7066795Z with: 2024-06-26T07:55:13.7067261Z name: linux-focal-rocm6.1-py3.8 2024-06-26T07:55:13.7067889Z s3-bucket: gha-artifacts 2024-06-26T07:55:13.7068443Z region: us-east-1 2024-06-26T07:55:13.7068906Z env: 2024-06-26T07:55:13.7069334Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:13.7069994Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:13.7071360Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:13.7072560Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:13.7073157Z AWS_REGION: us-east-1 2024-06-26T07:55:13.7073787Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:13.7074604Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:13.7086653Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:13.7087192Z ##[endgroup] 2024-06-26T07:55:14.0899084Z (node:2112790) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:55:14.0900234Z 2024-06-26T07:55:14.0900748Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:55:14.0902093Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:55:14.0903627Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:55:14.5000138Z Found 1 objects with prefix pytorch/pytorch/9673646592/linux-focal-rocm6.1-py3.8/ 2024-06-26T07:55:14.5002242Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:55:51.0465162Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:55:51.0486210Z Artifact download has finished successfully 2024-06-26T07:55:51.0823258Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:55:51.0824030Z unzip -o artifacts.zip 2024-06-26T07:55:51.0874847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:55:51.0875762Z env: 2024-06-26T07:55:51.0876256Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:55:51.0877000Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:55:51.0878322Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:55:51.0879557Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:55:51.0880235Z AWS_REGION: us-east-1 2024-06-26T07:55:51.0880987Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:55:51.0881915Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:55:51.0896132Z AWS_SESSION_TOKEN: *** 2024-06-26T07:55:51.0896758Z ##[endgroup] 2024-06-26T07:55:51.0992448Z Archive: artifacts.zip 2024-06-26T07:55:51.0993162Z creating: dist/ 2024-06-26T07:55:53.7333947Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp38-cp38-linux_x86_64.whl 2024-06-26T07:55:53.7451501Z inflating: dist/.ninja_log 2024-06-26T07:55:53.7454800Z creating: build/custom_test_artifacts/ 2024-06-26T07:55:53.7456258Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:55:53.7457550Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:55:53.7459064Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:55:53.7460855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:55:53.7462842Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:55:53.7464559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:55:53.7466325Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:55:53.7468066Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:55:53.7470055Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:55:53.7472050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:55:53.7473816Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:55:53.7475583Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:55:53.7477648Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:55:53.7479716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:55:53.7481724Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:55:53.7483748Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:55:53.7485794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:55:53.7488222Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:55:53.7489988Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:55:53.7491519Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:55:53.7493117Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:55:53.7495480Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:55:53.7497313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:55:53.7499352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:55:53.7501337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:55:53.7503176Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:55:53.7505072Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:55:53.7506980Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:55:53.7509026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:55:53.7511195Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:55:53.7513060Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:55:53.7514923Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:55:53.7611211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:55:53.7613450Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:55:53.7615452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:55:53.7617685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:55:53.7619784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:55:53.7621736Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:55:53.7623740Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:55:53.7625755Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:55:53.7627800Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:55:53.7630065Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:55:53.7632183Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:55:53.7634340Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:55:53.7708544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:55:53.7711297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:55:53.7713270Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:55:53.7715040Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:55:53.7716663Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:55:53.7718256Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:55:53.7719862Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2024-06-26T07:55:53.7721387Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-06-26T07:55:53.7722863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:55:53.7724228Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:55:53.7726036Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:55:53.7805110Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:55:53.7856574Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:55:53.7858245Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:55:53.7859746Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:55:53.7861224Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:55:53.7862983Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:55:53.7864636Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:55:53.7866288Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:55:53.7868013Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:55:53.7869735Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:55:53.7871678Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:55:53.7873999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:55:53.7875786Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:55:53.7877528Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:55:53.7879705Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:55:53.7881984Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:55:53.7883968Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:55:53.7885960Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:55:53.7888300Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:55:53.7890362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:55:53.7892082Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:55:53.7893598Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:55:53.7895174Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:55:53.7896848Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:55:53.7898708Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:55:53.7900815Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:55:53.7902826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:55:53.7904722Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:55:53.7906673Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:55:53.7908635Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:55:53.7910606Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:55:53.7912557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:55:53.7914488Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:55:53.7916756Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:55:53.7946370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:55:53.7948547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:55:53.7950500Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:55:53.7952220Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:55:53.7953814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:55:53.7955384Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:55:53.7957000Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2024-06-26T07:55:53.7958568Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-06-26T07:55:53.7960014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:55:53.7961371Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:55:53.7962990Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:55:53.7991045Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:55:53.7992456Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:55:53.7994030Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:55:53.7995576Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:55:53.7997413Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:55:53.7999133Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:55:53.8000905Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:55:53.8002743Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:55:53.8004578Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:55:53.8006662Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:55:53.8008822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:55:53.8011003Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:55:53.8012936Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:55:53.8015371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:55:53.8017592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:55:53.8019715Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:55:53.8021836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:55:53.8023986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:55:53.8026130Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:55:53.8027980Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:55:53.8029723Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:55:53.8031698Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:55:53.8033699Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:55:53.8035699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:55:53.8038089Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:55:53.8040430Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:55:53.8042469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:55:53.8044567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:55:53.8046653Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:55:53.8048844Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:55:53.8050934Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:55:53.8053006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:55:53.8055372Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:55:53.8128738Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:55:53.8130872Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:55:53.8133178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:55:53.8135617Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:55:53.8137868Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:55:53.8140001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:55:53.8142203Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:55:53.8144403Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:55:53.8146613Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:55:53.8148821Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:55:53.8151324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:55:53.8153819Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:55:53.8200434Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:55:53.8202737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:55:53.8204766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:55:53.8206602Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:55:53.8208361Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:55:53.8210061Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:55:53.8211749Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2024-06-26T07:55:53.8213586Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-06-26T07:55:53.8215138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:55:53.8216612Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:55:53.8218129Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:55:53.8290431Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:55:53.8325274Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:55:53.8326403Z creating: build/lib/ 2024-06-26T07:55:53.8398077Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:55:53.8773859Z inflating: build/lib/libprotobuf.a 2024-06-26T07:55:53.9189488Z inflating: build/lib/libprotoc.a 2024-06-26T07:55:53.9197226Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:55:53.9203769Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:55:53.9210245Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:55:53.9211194Z inflating: build/lib/libclog.a 2024-06-26T07:55:53.9226060Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:55:53.9227556Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:55:53.9242668Z inflating: build/lib/libnnpack.a 2024-06-26T07:55:53.9296049Z inflating: build/lib/libgtest.a 2024-06-26T07:55:53.9310662Z inflating: build/lib/libgmock.a 2024-06-26T07:55:53.9311453Z inflating: build/lib/libgtest_main.a 2024-06-26T07:55:53.9312253Z inflating: build/lib/libgmock_main.a 2024-06-26T07:55:53.9373074Z inflating: build/lib/libbenchmark.a 2024-06-26T07:55:53.9373918Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:55:53.9525438Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:55:53.9531622Z inflating: build/lib/libittnotify.a 2024-06-26T07:55:53.9583902Z inflating: build/lib/libasmjit.a 2024-06-26T07:55:54.0619635Z inflating: build/lib/libfbgemm.a 2024-06-26T07:55:54.0642468Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:55:54.1115389Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:55:54.1116334Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:55:54.1209404Z inflating: build/lib/libgloo.a 2024-06-26T07:55:54.1243839Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:55:54.1562912Z inflating: build/lib/libgloo_hip.a 2024-06-26T07:55:54.2135027Z inflating: build/lib/libonnx.a 2024-06-26T07:55:55.0498594Z inflating: build/lib/libdnnl.a 2024-06-26T07:55:55.0516101Z inflating: build/lib/libfmt.a 2024-06-26T07:55:55.0788824Z inflating: build/lib/libkineto.a 2024-06-26T07:55:55.0863641Z inflating: build/lib/libc10.so 2024-06-26T07:55:55.0864606Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:55:55.0865539Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:55:55.0901766Z inflating: build/lib/libc10_hip.so 2024-06-26T07:55:57.3188624Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:55:57.3192163Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:55:57.3195558Z inflating: build/lib/libshm.so 2024-06-26T07:56:03.3421033Z inflating: build/lib/libtorch_hip.so 2024-06-26T07:56:03.3422024Z inflating: build/lib/libtorch.so 2024-06-26T07:56:03.3438309Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:56:03.3494518Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:56:03.3515474Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:56:03.5118554Z inflating: build/lib/libtorch_python.so 2024-06-26T07:56:03.5146664Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:56:03.5147569Z creating: build/bin/ 2024-06-26T07:56:03.5148234Z creating: build/bin/CMakeFiles/ 2024-06-26T07:56:03.5191572Z inflating: build/bin/memory_format_test 2024-06-26T07:56:03.5193033Z inflating: build/bin/cmake_install.cmake 2024-06-26T07:56:03.5235649Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:56:03.5236669Z inflating: build/bin/CTestTestfile.cmake 2024-06-26T07:56:03.5602100Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:56:03.5967622Z inflating: build/bin/protoc 2024-06-26T07:56:03.6009887Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:56:03.6051920Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:56:03.6092846Z inflating: build/bin/c10_Device_test 2024-06-26T07:56:03.6133132Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:56:03.6181455Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:56:03.6223275Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:56:03.6267112Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:56:03.6311919Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:56:03.6357440Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:56:03.6403659Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:56:03.6461607Z inflating: build/bin/c10_cow_test 2024-06-26T07:56:03.6501899Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:56:03.6545102Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:56:03.6586656Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:56:03.6632272Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:56:03.6676766Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:56:03.6718232Z inflating: build/bin/c10_Half_test 2024-06-26T07:56:03.6759251Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:56:03.6804821Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:56:03.6847001Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:56:03.6888941Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:56:03.6929140Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:56:03.6971469Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:56:03.7017016Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:56:03.7058298Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:56:03.7104616Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:56:03.7150003Z inflating: build/bin/c10_complex_test 2024-06-26T07:56:03.7194067Z inflating: build/bin/c10_exception_test 2024-06-26T07:56:03.7235140Z inflating: build/bin/c10_flags_test 2024-06-26T07:56:03.7276384Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:56:03.7317817Z inflating: build/bin/c10_irange_test 2024-06-26T07:56:03.7361793Z inflating: build/bin/c10_lazy_test 2024-06-26T07:56:03.7408354Z inflating: build/bin/c10_logging_test 2024-06-26T07:56:03.7544158Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:56:03.7595287Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:56:03.7656830Z inflating: build/bin/c10_optional_test 2024-06-26T07:56:03.7701362Z inflating: build/bin/c10_registry_test 2024-06-26T07:56:03.7743326Z inflating: build/bin/c10_ssize_test 2024-06-26T07:56:03.7871073Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:56:03.7913462Z inflating: build/bin/c10_string_util_test 2024-06-26T07:56:03.7955076Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:56:03.8003133Z inflating: build/bin/c10_string_view_test 2024-06-26T07:56:03.8049609Z inflating: build/bin/c10_typeid_test 2024-06-26T07:56:03.8089028Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:56:03.8129398Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-06-26T07:56:03.8169400Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-06-26T07:56:03.8209770Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:56:03.8250523Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-06-26T07:56:03.8290662Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:56:03.8330818Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:56:03.8370760Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:56:03.8410707Z inflating: build/bin/c10_hip_HIPTest 2024-06-26T07:56:03.8692650Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:56:03.8963500Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:56:03.9258396Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:56:03.9302481Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:56:03.9345993Z inflating: build/bin/FileStoreTest 2024-06-26T07:56:03.9391491Z inflating: build/bin/TCPStoreTest 2024-06-26T07:56:03.9435376Z inflating: build/bin/HashStoreTest 2024-06-26T07:56:03.9489490Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:56:03.9490607Z inflating: build/bin/example_allreduce 2024-06-26T07:56:03.9494724Z inflating: build/bin/torch_shm_manager 2024-06-26T07:56:03.9702697Z inflating: build/bin/static_runtime_test 2024-06-26T07:56:03.9751522Z inflating: build/bin/static_runtime_bench 2024-06-26T07:56:03.9812249Z inflating: build/bin/Dict_test 2024-06-26T07:56:03.9854882Z inflating: build/bin/Dimname_test 2024-06-26T07:56:03.9907998Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:56:03.9955609Z inflating: build/bin/NamedTensor_test 2024-06-26T07:56:04.0003948Z inflating: build/bin/apply_utils_test 2024-06-26T07:56:04.0051962Z inflating: build/bin/atest 2024-06-26T07:56:04.0103649Z inflating: build/bin/basic 2024-06-26T07:56:04.0149453Z inflating: build/bin/broadcast_test 2024-06-26T07:56:04.0191403Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:56:04.0240071Z inflating: build/bin/cpu_generator_test 2024-06-26T07:56:04.0283970Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:56:04.0360685Z inflating: build/bin/cpu_rng_test 2024-06-26T07:56:04.0401903Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:56:04.0443276Z inflating: build/bin/dlconvertor_test 2024-06-26T07:56:04.0491023Z inflating: build/bin/extension_backend_test 2024-06-26T07:56:04.0536450Z inflating: build/bin/half_test 2024-06-26T07:56:04.0615297Z inflating: build/bin/ivalue_test 2024-06-26T07:56:04.0656336Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:56:04.0700966Z inflating: build/bin/math_kernel_test 2024-06-26T07:56:04.0744543Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:56:04.0790273Z inflating: build/bin/native_test 2024-06-26T07:56:04.0832116Z inflating: build/bin/operator_name_test 2024-06-26T07:56:04.0874305Z inflating: build/bin/operators_test 2024-06-26T07:56:04.0917112Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:56:04.0972221Z inflating: build/bin/pow_test 2024-06-26T07:56:04.1019412Z inflating: build/bin/quantized_test 2024-06-26T07:56:04.1060359Z inflating: build/bin/reduce_ops_test 2024-06-26T07:56:04.1102426Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:56:04.1148732Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:56:04.1197000Z inflating: build/bin/scalar_test 2024-06-26T07:56:04.1239483Z inflating: build/bin/StorageUtils_test 2024-06-26T07:56:04.1282951Z inflating: build/bin/stride_properties_test 2024-06-26T07:56:04.1348695Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:56:04.1393399Z inflating: build/bin/test_parallel 2024-06-26T07:56:04.1394625Z inflating: build/bin/thread_init_test 2024-06-26T07:56:04.1441147Z inflating: build/bin/type_ptr_test 2024-06-26T07:56:04.1490525Z inflating: build/bin/type_test 2024-06-26T07:56:04.1533393Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:56:04.1534396Z inflating: build/bin/verify_api_visibility 2024-06-26T07:56:04.1591859Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:56:04.1634144Z inflating: build/bin/weakref_test 2024-06-26T07:56:04.1676541Z inflating: build/bin/wrapdim_test 2024-06-26T07:56:04.1725758Z inflating: build/bin/IListRef_test 2024-06-26T07:56:04.1768551Z inflating: build/bin/xla_tensor_test 2024-06-26T07:56:04.1856025Z inflating: build/bin/List_test 2024-06-26T07:56:04.1956365Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:56:04.2011011Z inflating: build/bin/KernelFunction_test 2024-06-26T07:56:04.2090409Z inflating: build/bin/kernel_function_test 2024-06-26T07:56:04.2195490Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:56:04.2280723Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:56:04.2331038Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:56:04.2411132Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:56:04.2452881Z inflating: build/bin/CppSignature_test 2024-06-26T07:56:04.2493170Z inflating: build/bin/op_allowlist_test 2024-06-26T07:56:04.2538912Z inflating: build/bin/backend_fallback_test 2024-06-26T07:56:04.2591376Z inflating: build/bin/inline_container_test 2024-06-26T07:56:04.2839805Z inflating: build/bin/op_registration_test 2024-06-26T07:56:04.2883163Z inflating: build/bin/hip_apply_test 2024-06-26T07:56:04.2923682Z inflating: build/bin/hip_complex_math_test 2024-06-26T07:56:04.2964134Z inflating: build/bin/hip_complex_test 2024-06-26T07:56:04.3004197Z inflating: build/bin/hip_distributions_test 2024-06-26T07:56:04.3045746Z inflating: build/bin/hip_dlconvertor_test 2024-06-26T07:56:04.3086041Z inflating: build/bin/hip_generator_test 2024-06-26T07:56:04.3125984Z inflating: build/bin/hip_half_test 2024-06-26T07:56:04.3166432Z inflating: build/bin/hip_integer_divider_test 2024-06-26T07:56:04.3206375Z inflating: build/bin/hip_optional_test 2024-06-26T07:56:04.3246807Z inflating: build/bin/hip_packedtensoraccessor_test 2024-06-26T07:56:04.3287185Z inflating: build/bin/hip_vectorized_test 2024-06-26T07:56:04.3763761Z inflating: build/bin/test_jit 2024-06-26T07:56:04.4419329Z inflating: build/bin/test_tensorexpr 2024-06-26T07:56:04.4431450Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:56:04.4477280Z inflating: build/bin/test_dist_autograd 2024-06-26T07:56:04.4534221Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:56:04.5508059Z inflating: build/bin/test_api 2024-06-26T07:56:04.5509205Z inflating: build/bin/parallel_benchmark 2024-06-26T07:56:04.5565109Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:56:04.5572637Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:56:04.5857120Z inflating: build/bin/test_lazy 2024-06-26T07:56:04.5857978Z creating: .additional_ci_files/ 2024-06-26T07:56:04.5903420Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:56:04.6082893Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:56:04.6128853Z ##[group]Run rm artifacts.zip 2024-06-26T07:56:04.6129563Z rm artifacts.zip 2024-06-26T07:56:04.6180875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:04.6181768Z env: 2024-06-26T07:56:04.6182263Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:04.6182996Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:04.6184341Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:04.6185594Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:04.6186377Z AWS_REGION: us-east-1 2024-06-26T07:56:04.6187228Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:04.6188302Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:04.6202067Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:04.6202673Z ##[endgroup] 2024-06-26T07:56:04.7557158Z ##[group]Run df -H 2024-06-26T07:56:04.7557732Z df -H 2024-06-26T07:56:04.7611786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:04.7612694Z env: 2024-06-26T07:56:04.7613194Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:04.7614238Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:04.7615566Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:04.7616864Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:04.7617655Z AWS_REGION: us-east-1 2024-06-26T07:56:04.7618543Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:04.7619667Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:04.7634397Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:04.7635022Z ##[endgroup] 2024-06-26T07:56:04.7735358Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:56:04.7736262Z tmpfs 109G 33M 109G 1% /run 2024-06-26T07:56:04.7737109Z /dev/nvme0n1p2 944G 79G 817G 9% / 2024-06-26T07:56:04.7737915Z tmpfs 542G 33k 542G 1% /dev/shm 2024-06-26T07:56:04.7738733Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-06-26T07:56:04.7739920Z /dev/nvme0n1p1 536M 6.4M 530M 2% /boot/efi 2024-06-26T07:56:04.7740851Z /dev/nvme1n1p1 3.8T 836G 2.8T 24% /media/4TB 2024-06-26T07:56:04.7741729Z tmpfs 109G 29k 109G 1% /run/user/4001 2024-06-26T07:56:04.7742598Z 172.18.144.8:/export/amd 26T 21T 4.6T 82% /mnt 2024-06-26T07:56:04.7802763Z Prepare all required actions 2024-06-26T07:56:04.7803481Z Getting action download info 2024-06-26T07:56:04.9745856Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:56:04.9746578Z with: 2024-06-26T07:56:04.9746964Z env: 2024-06-26T07:56:04.9747386Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:04.9748029Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:04.9749196Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:04.9750288Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:04.9750875Z AWS_REGION: us-east-1 2024-06-26T07:56:04.9751518Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:04.9752331Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:04.9764359Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:04.9764895Z ##[endgroup] 2024-06-26T07:56:04.9816313Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:56:04.9817007Z with: 2024-06-26T07:56:04.9817421Z name: td_results 2024-06-26T07:56:04.9817963Z s3-bucket: gha-artifacts 2024-06-26T07:56:04.9818499Z region: us-east-1 2024-06-26T07:56:04.9818956Z env: 2024-06-26T07:56:04.9819377Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:04.9820015Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:04.9821184Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:04.9822290Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:04.9822879Z AWS_REGION: us-east-1 2024-06-26T07:56:04.9823509Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:04.9824301Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:04.9836311Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:04.9836849Z ##[endgroup] 2024-06-26T07:56:05.3664398Z (node:2112837) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:56:05.3665056Z 2024-06-26T07:56:05.3665300Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:56:05.3666049Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:56:05.3667158Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:56:05.6898680Z Found 1 objects with prefix pytorch/pytorch/9673646592/td_results/ 2024-06-26T07:56:05.6899762Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T07:56:06.1241081Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-06-26T07:56:06.1251922Z Artifact download has finished successfully 2024-06-26T07:56:06.1635930Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:56:06.1636870Z mkdir -p .additional_ci_files 2024-06-26T07:56:06.1638197Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:56:06.1692881Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:06.1693816Z env: 2024-06-26T07:56:06.1694334Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:06.1695137Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:06.1696572Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:06.1697897Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:06.1698604Z AWS_REGION: us-east-1 2024-06-26T07:56:06.1699384Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:06.1700345Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:06.1715041Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:06.1715693Z ##[endgroup] 2024-06-26T07:56:06.3427263Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:56:06.3428173Z .github/scripts/parse_ref.py 2024-06-26T07:56:06.3481224Z shell: /usr/bin/bash -e {0} 2024-06-26T07:56:06.3481899Z env: 2024-06-26T07:56:06.3482431Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:06.3483213Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:06.3484615Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:06.3485926Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:06.3486636Z AWS_REGION: us-east-1 2024-06-26T07:56:06.3487429Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:06.3488655Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:06.3542865Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:06.3543634Z ##[endgroup] 2024-06-26T07:56:06.4050878Z Prepare all required actions 2024-06-26T07:56:06.4159725Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:56:06.4160532Z with: 2024-06-26T07:56:06.4161407Z github-token: *** 2024-06-26T07:56:06.4161948Z env: 2024-06-26T07:56:06.4162427Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:06.4163153Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:06.4164498Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:06.4165779Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:06.4166445Z AWS_REGION: us-east-1 2024-06-26T07:56:06.4167123Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:06.4168318Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:06.4182000Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:06.4182606Z ##[endgroup] 2024-06-26T07:56:06.4212383Z ##[group]Run set -eux 2024-06-26T07:56:06.4212995Z set -eux 2024-06-26T07:56:06.4214085Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:56:06.4258928Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:06.4259808Z env: 2024-06-26T07:56:06.4260302Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:06.4261053Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:06.4262387Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:06.4263656Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:06.4264317Z AWS_REGION: us-east-1 2024-06-26T07:56:06.4265027Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:06.4265935Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:06.4279729Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:06.4280718Z GITHUB_TOKEN: *** 2024-06-26T07:56:06.4281273Z ##[endgroup] 2024-06-26T07:56:06.4347264Z + python3 .github/scripts/get_workflow_job_id.py 9673646592 GPU9449 2024-06-26T07:56:08.2481209Z setting job-id=26688598647 2024-06-26T07:56:08.2482612Z setting job-name=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:08.2893638Z Prepare all required actions 2024-06-26T07:56:08.2894359Z Getting action download info 2024-06-26T07:56:09.0554703Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T07:56:09.9924924Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:56:09.9925644Z with: 2024-06-26T07:56:09.9926596Z github-token: *** 2024-06-26T07:56:09.9928830Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-06-26T07:56:09.9931570Z job-name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:09.9932535Z env: 2024-06-26T07:56:09.9932965Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:09.9933610Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:09.9934830Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:09.9935933Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:09.9936527Z AWS_REGION: us-east-1 2024-06-26T07:56:09.9937132Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:09.9937932Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:09.9950255Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:09.9950805Z ##[endgroup] 2024-06-26T07:56:10.0018461Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:56:10.0019317Z with: 2024-06-26T07:56:10.0019725Z shell: bash 2024-06-26T07:56:10.0020188Z timeout_minutes: 10 2024-06-26T07:56:10.0020692Z max_attempts: 5 2024-06-26T07:56:10.0021181Z retry_wait_seconds: 30 2024-06-26T07:56:10.0022954Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:56:10.0024793Z polling_interval_seconds: 1 2024-06-26T07:56:10.0025373Z warning_on_retry: true 2024-06-26T07:56:10.0025938Z continue_on_error: false 2024-06-26T07:56:10.0026547Z env: 2024-06-26T07:56:10.0027040Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:10.0027744Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:10.0028913Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:10.0030024Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:10.0030610Z AWS_REGION: us-east-1 2024-06-26T07:56:10.0031237Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:10.0032031Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:10.0044027Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:10.0044791Z GITHUB_TOKEN: *** 2024-06-26T07:56:10.0045277Z ##[endgroup] 2024-06-26T07:56:10.0791743Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:56:10.5608124Z Requirement already satisfied: requests==2.27.1 in /opt/anaconda3/lib/python3.11/site-packages (2.27.1) 2024-06-26T07:56:10.5613971Z Requirement already satisfied: pyyaml==6.0.1 in /opt/anaconda3/lib/python3.11/site-packages (6.0.1) 2024-06-26T07:56:10.5695757Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/anaconda3/lib/python3.11/site-packages (from requests==2.27.1) (1.26.18) 2024-06-26T07:56:10.5702888Z Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.11/site-packages (from requests==2.27.1) (2024.2.2) 2024-06-26T07:56:10.5714961Z Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/anaconda3/lib/python3.11/site-packages (from requests==2.27.1) (2.0.4) 2024-06-26T07:56:10.5727088Z Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/lib/python3.11/site-packages (from requests==2.27.1) (3.4) 2024-06-26T07:56:13.0810930Z Command completed after 1 attempt(s). 2024-06-26T07:56:13.0892933Z ##[group]Run set -x 2024-06-26T07:56:13.0893533Z set -x 2024-06-26T07:56:13.0894064Z  2024-06-26T07:56:13.0895359Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:56:13.0896628Z # in runner workspace 2024-06-26T07:56:13.0897608Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:56:13.0951189Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:13.0952122Z env: 2024-06-26T07:56:13.0952902Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:13.0953691Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:13.0955117Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:13.0956447Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:13.0957160Z AWS_REGION: us-east-1 2024-06-26T07:56:13.0957927Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:13.0958904Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:13.0973343Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:13.0973992Z ##[endgroup] 2024-06-26T07:56:13.1052421Z + python3 /home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:56:13.1444864Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:56:13.1445808Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:56:13.1446609Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:56:13.1447296Z  2024-06-26T07:56:13.1448607Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:56:13.1449822Z # in runner workspace 2024-06-26T07:56:13.1450834Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:56:13.1451979Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:56:13.1452786Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:56:13.1455192Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-06-26T07:56:13.1457624Z  --selected-test-configs "" \ 2024-06-26T07:56:13.1458432Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:56:13.1459151Z  --tag "${TAG}" \ 2024-06-26T07:56:13.1459841Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:56:13.1460620Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:56:13.1461368Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:56:13.1512108Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:13.1512988Z env: 2024-06-26T07:56:13.1513477Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:13.1514210Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:13.1515529Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:13.1516759Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:13.1517426Z AWS_REGION: us-east-1 2024-06-26T07:56:13.1518168Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:13.1519097Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:13.1532850Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:13.1533715Z GITHUB_TOKEN: *** 2024-06-26T07:56:13.1534684Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:13.1535779Z PR_NUMBER: 2024-06-26T07:56:13.1536315Z TAG: ciflow/trunk/129470 2024-06-26T07:56:13.1536936Z EVENT_NAME: push 2024-06-26T07:56:13.1537466Z SCHEDULE: 2024-06-26T07:56:13.1537962Z HEAD_BRANCH: 2024-06-26T07:56:13.1538477Z ##[endgroup] 2024-06-26T07:56:13.1615098Z Workflow: trunk 2024-06-26T07:56:13.1616529Z Job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:13.6409190Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:56:14.4856628Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:56:14.4857450Z echo "Filtered matrix:" 2024-06-26T07:56:14.4860131Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-06-26T07:56:14.4862853Z  2024-06-26T07:56:14.4863441Z echo 2024-06-26T07:56:14.4864245Z echo "Is the current job unstable? False" 2024-06-26T07:56:14.4865525Z  2024-06-26T07:56:14.4866098Z echo 2024-06-26T07:56:14.4866864Z echo "Is keep-going label set? False" 2024-06-26T07:56:14.4867689Z  2024-06-26T07:56:14.4868171Z echo 2024-06-26T07:56:14.4868726Z echo "Renabled issues? " 2024-06-26T07:56:14.4921709Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:14.4922619Z env: 2024-06-26T07:56:14.4923127Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:14.4923906Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:14.4925325Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:14.4926628Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:14.4927319Z AWS_REGION: us-east-1 2024-06-26T07:56:14.4928500Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:14.4929481Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:14.4944482Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:14.4945248Z ##[endgroup] 2024-06-26T07:56:14.5026183Z Filtered matrix: 2024-06-26T07:56:14.5028656Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-06-26T07:56:14.5030828Z 2024-06-26T07:56:14.5031149Z Is the current job unstable? False 2024-06-26T07:56:14.5031687Z 2024-06-26T07:56:14.5032290Z Is keep-going label set? False 2024-06-26T07:56:14.5032749Z 2024-06-26T07:56:14.5033034Z Renabled issues? 2024-06-26T07:56:14.5118267Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:56:14.5119612Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:56:14.5167154Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:56:14.5168193Z env: 2024-06-26T07:56:14.5168711Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:14.5169487Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:14.5170890Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:14.5172193Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:14.5172889Z AWS_REGION: us-east-1 2024-06-26T07:56:14.5173632Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:14.5174617Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:14.5190016Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:14.5190658Z JOB_TIMEOUT: 300 2024-06-26T07:56:14.5191225Z ##[endgroup] 2024-06-26T07:56:14.5370116Z ##[group]Run set -x 2024-06-26T07:56:14.5370832Z set -x 2024-06-26T07:56:14.5371343Z  2024-06-26T07:56:14.5371957Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:56:14.5372925Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:56:14.5373918Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:56:14.5374823Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-06-26T07:56:14.5375540Z else 2024-06-26T07:56:14.5376153Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:56:14.5376871Z fi 2024-06-26T07:56:14.5377330Z  2024-06-26T07:56:14.5378134Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:56:14.5379456Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:56:14.5380621Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:56:14.5381608Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:56:14.5382416Z container_name=$(docker run \ 2024-06-26T07:56:14.5383143Z  ${GPU_FLAG:-} \ 2024-06-26T07:56:14.5383782Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:56:14.5384470Z  -e PR_NUMBER \ 2024-06-26T07:56:14.5385086Z  -e GITHUB_ACTIONS \ 2024-06-26T07:56:14.5385769Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:56:14.5386477Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:56:14.5387139Z  -e GITHUB_JOB \ 2024-06-26T07:56:14.5388066Z  -e GITHUB_RUN_ID \ 2024-06-26T07:56:14.5388720Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:56:14.5389413Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:56:14.5390091Z  -e JOB_ID \ 2024-06-26T07:56:14.5390674Z  -e JOB_NAME \ 2024-06-26T07:56:14.5391258Z  -e BRANCH \ 2024-06-26T07:56:14.5391816Z  -e SHA1 \ 2024-06-26T07:56:14.5392407Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:56:14.5393112Z  -e IN_WHEEL_TEST \ 2024-06-26T07:56:14.5393752Z  -e SHARD_NUMBER \ 2024-06-26T07:56:14.5394381Z  -e TEST_CONFIG \ 2024-06-26T07:56:14.5395018Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:56:14.5395689Z  -e REENABLED_ISSUES \ 2024-06-26T07:56:14.5396406Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:56:14.5397135Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:56:14.5397815Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:56:14.5398502Z  -e NO_TD \ 2024-06-26T07:56:14.5399202Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:56:14.5399999Z  -e SCCACHE_BUCKET \ 2024-06-26T07:56:14.5400740Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:56:14.5401605Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:56:14.5402497Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:56:14.5403297Z  -e TESTS_TO_INCLUDE \ 2024-06-26T07:56:14.5404150Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:56:14.5405089Z  --ulimit stack=10485760:83886080 \ 2024-06-26T07:56:14.5405871Z  --ulimit core=0 \ 2024-06-26T07:56:14.5406606Z  --security-opt seccomp=unconfined \ 2024-06-26T07:56:14.5407402Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:56:14.5408145Z  --shm-size="8g" \ 2024-06-26T07:56:14.5408741Z  --tty \ 2024-06-26T07:56:14.5409268Z  --detach \ 2024-06-26T07:56:14.5409877Z  --name="${container_name}" \ 2024-06-26T07:56:14.5410609Z  --user jenkins \ 2024-06-26T07:56:14.5411449Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:56:14.5412439Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:56:14.5413172Z  "${DOCKER_IMAGE}" 2024-06-26T07:56:14.5413753Z ) 2024-06-26T07:56:14.5414331Z # save container name for later step 2024-06-26T07:56:14.5415581Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-06-26T07:56:14.5417394Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-06-26T07:56:14.5419747Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-06-26T07:56:14.5469753Z shell: /usr/bin/bash -e {0} 2024-06-26T07:56:14.5470378Z env: 2024-06-26T07:56:14.5470850Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:56:14.5471588Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T07:56:14.5472902Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T07:56:14.5474133Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:56:14.5474776Z AWS_REGION: us-east-1 2024-06-26T07:56:14.5475490Z AWS_ACCESS_KEY_ID: *** 2024-06-26T07:56:14.5476384Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T07:56:14.5490215Z AWS_SESSION_TOKEN: *** 2024-06-26T07:56:14.5490956Z BUILD_ENVIRONMENT: linux-focal-rocm6.1-py3.8 2024-06-26T07:56:14.5491733Z PR_NUMBER: 2024-06-26T07:56:14.5492314Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:56:14.5492997Z GITHUB_WORKFLOW: trunk 2024-06-26T07:56:14.5493511Z GITHUB_JOB: test 2024-06-26T07:56:14.5493986Z GITHUB_RUN_ID: 9673646592 2024-06-26T07:56:14.5494524Z GITHUB_RUN_NUMBER: 83498 2024-06-26T07:56:14.5495056Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:56:14.5495553Z JOB_ID: 26688598647 2024-06-26T07:56:14.5496406Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:14.5497555Z BRANCH: 2024-06-26T07:56:14.5498064Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:14.5498768Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:56:14.5499356Z VERBOSE_TEST_LOGS: False 2024-06-26T07:56:14.5499891Z NO_TEST_TIMEOUT: False 2024-06-26T07:56:14.5500393Z NO_TD: False 2024-06-26T07:56:14.5500848Z TEST_CONFIG: distributed 2024-06-26T07:56:14.5501370Z SHARD_NUMBER: 1 2024-06-26T07:56:14.5501835Z NUM_TEST_SHARDS: 1 2024-06-26T07:56:14.5502319Z REENABLED_ISSUES: 2024-06-26T07:56:14.5502932Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:56:14.5504535Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:56:14.5506280Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:56:14.5507226Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:56:14.5507895Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:56:14.5509682Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-06-26T07:56:14.5511433Z ##[endgroup] 2024-06-26T07:56:14.5580795Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:56:14.5582122Z + [[ linux-focal-rocm6.1-py3.8 == *onnx* ]] 2024-06-26T07:56:14.5583165Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:56:14.5595239Z +++ nproc --ignore=2 2024-06-26T07:56:14.5619717Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_9673646592 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:56:14.8108800Z + container_name=bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T07:56:14.8110560Z + echo CONTAINER_NAME=bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T07:56:14.8120837Z + docker exec -t bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-06-26T07:56:21.2153224Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp38-cp38-linux_x86_64.whl 2024-06-26T07:56:21.5738385Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:56:21.5741518Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:56:21.5744297Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:56:21.5747163Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:56:21.5749773Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:56:21.5753150Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:56:21.6100419Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:56:21.6226989Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:56:22.3694770Z Installing collected packages: torch 2024-06-26T07:56:31.8218753Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:56:31.8692127Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:56:31.8708414Z + source .ci/pytorch/common.sh 2024-06-26T07:56:31.8712564Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:56:31.8724542Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:56:31.8728705Z +++ declare -f -t trap_add 2024-06-26T07:56:31.8734413Z ++ set -ex 2024-06-26T07:56:31.8735372Z ++ [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-06-26T07:56:31.8736259Z ++ unset HIP_PLATFORM 2024-06-26T07:56:31.8736954Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-06-26T07:56:31.8737756Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-06-26T07:56:31.8738588Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-06-26T07:56:31.8739262Z ++ HSAKMT_DEBUG_LEVEL=4 2024-06-26T07:56:31.8739961Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-06-26T07:56:31.8740745Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-06-26T07:56:31.8741439Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:56:31.8742247Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-06-26T07:56:31.8743168Z + echo 'Environment variables:' 2024-06-26T07:56:31.8743867Z Environment variables: 2024-06-26T07:56:31.8744432Z + env 2024-06-26T07:56:31.8749929Z INSTALLED_DB=yes 2024-06-26T07:56:31.8751464Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T07:56:31.8752878Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-06-26T07:56:31.8753742Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:56:31.8754719Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-06-26T07:56:31.8755541Z HOSTNAME=bd5a72183d56 2024-06-26T07:56:31.8757433Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.8758998Z GITHUB_ACTION=__self 2024-06-26T07:56:31.8759659Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:56:31.8760385Z GITHUB_RUN_NUMBER=83498 2024-06-26T07:56:31.8761589Z TEST_CONFIG=distributed 2024-06-26T07:56:31.8762386Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:56:31.8763234Z AWS_DEFAULT_REGION=us-east-1 2024-06-26T07:56:31.8764058Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:56:31.8764825Z GITHUB_REF_TYPE=tag 2024-06-26T07:56:31.8768534Z *** 2024-06-26T07:56:31.8769142Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:56:31.8769864Z GITHUB_ACTIONS=true 2024-06-26T07:56:31.8770574Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.8771499Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.8772940Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/129470 2024-06-26T07:56:31.8774234Z VERBOSE_TEST_LOGS=False 2024-06-26T07:56:31.8774929Z GITHUB_REF=refs/tags/ciflow/trunk/129470 2024-06-26T07:56:31.8775685Z SHARD_NUMBER=1 2024-06-26T07:56:31.8776254Z GITHUB_REF_PROTECTED=false 2024-06-26T07:56:31.8776895Z HOME=/var/lib/jenkins 2024-06-26T07:56:31.8777573Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:56:31.8778403Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:56:31.8779207Z LANG=C.UTF-8 2024-06-26T07:56:31.8779822Z PYTORCH_TEST_WITH_ROCM=1 2024-06-26T07:56:31.8780549Z NUM_TEST_SHARDS=1 2024-06-26T07:56:31.8782457Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.8784850Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:31.8786009Z MAGMA_HOME=/opt/rocm/magma 2024-06-26T07:56:31.8788043Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.8789555Z HSAKMT_DEBUG_LEVEL=4 2024-06-26T07:56:31.8791039Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:56:31.8792463Z GITHUB_EVENT_NAME=push 2024-06-26T07:56:31.8793077Z GITHUB_RUN_ID=9673646592 2024-06-26T07:56:31.8794832Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.8796545Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:56:31.8797207Z PR_NUMBER= 2024-06-26T07:56:31.8797728Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:56:31.8798342Z ANACONDA_PYTHON_VERSION=3.8 2024-06-26T07:56:31.8799154Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:56:31.8799992Z TERM=xterm 2024-06-26T07:56:31.8800504Z INSTALLED_VISION=yes 2024-06-26T07:56:31.8801068Z BRANCH= 2024-06-26T07:56:31.8802955Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-06-26T07:56:31.8805893Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-06-26T07:56:31.8807319Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:56:31.8808200Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-06-26T07:56:31.8808896Z REENABLED_ISSUES= 2024-06-26T07:56:31.8809446Z SHLVL=1 2024-06-26T07:56:31.8809915Z MAX_JOBS=126 2024-06-26T07:56:31.8810465Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:56:31.8811297Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.8812276Z GITHUB_REF_NAME=ciflow/trunk/129470 2024-06-26T07:56:31.8812999Z ROCM_PATH=/opt/rocm 2024-06-26T07:56:31.8814053Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:56:31.8815116Z GITHUB_JOB=test 2024-06-26T07:56:31.8815665Z NO_TEST_TIMEOUT=False 2024-06-26T07:56:31.8816326Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:56:31.8817060Z LC_ALL=C.UTF-8 2024-06-26T07:56:31.8817618Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:56:31.8818258Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:56:31.8820877Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:56:31.8823968Z GITHUB_BASE_REF= 2024-06-26T07:56:31.8824590Z CI=true 2024-06-26T07:56:31.8825100Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-06-26T07:56:31.8825794Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:56:31.8826469Z JOB_ID=26688598647 2024-06-26T07:56:31.8827038Z INSTALLED_PROTOBUF=yes 2024-06-26T07:56:31.8827630Z GITHUB_HEAD_REF= 2024-06-26T07:56:31.8828169Z GITHUB_ACTION_REF= 2024-06-26T07:56:31.8828969Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:56:31.8829884Z GITHUB_WORKFLOW=trunk 2024-06-26T07:56:31.8830629Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:56:31.8832660Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.8834199Z NO_TD=False 2024-06-26T07:56:31.8834727Z OLDPWD=/var/lib/jenkins 2024-06-26T07:56:31.8835328Z _=/usr/bin/env 2024-06-26T07:56:31.8836238Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:56:31.9051990Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-06-26T07:56:31.9053908Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T07:56:31.9055819Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-06-26T07:56:31.9057762Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-06-26T07:56:31.9058956Z + BUILD_DIR=build 2024-06-26T07:56:31.9059569Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:56:31.9060298Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:56:31.9061301Z + SHARD_NUMBER=1 2024-06-26T07:56:31.9061852Z + NUM_TEST_SHARDS=1 2024-06-26T07:56:31.9062429Z + export VALGRIND=ON 2024-06-26T07:56:31.9063009Z + VALGRIND=ON 2024-06-26T07:56:31.9063862Z + [[ linux-focal-rocm6.1-py3.8 == *clang9* ]] 2024-06-26T07:56:31.9064828Z + [[ 0 == \1 ]] 2024-06-26T07:56:31.9065465Z + [[ False == \1 ]] 2024-06-26T07:56:31.9066328Z + [[ linux-focal-rocm6.1-py3.8 != *bazel* ]] 2024-06-26T07:56:31.9067205Z ++ realpath build/custom_test_artifacts 2024-06-26T07:56:31.9079922Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-06-26T07:56:31.9082926Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-06-26T07:56:31.9085056Z + echo 'Setting INCLUDE_CLAUSE' 2024-06-26T07:56:31.9085791Z Setting INCLUDE_CLAUSE 2024-06-26T07:56:31.9088326Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-06-26T07:56:31.9090606Z + echo 'Environment variables' 2024-06-26T07:56:31.9091301Z Environment variables 2024-06-26T07:56:31.9091872Z + env 2024-06-26T07:56:31.9095043Z INSTALLED_DB=yes 2024-06-26T07:56:31.9096157Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T07:56:31.9097330Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-06-26T07:56:31.9098176Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:56:31.9099044Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-06-26T07:56:31.9099855Z HOSTNAME=bd5a72183d56 2024-06-26T07:56:31.9101860Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.9103688Z GITHUB_ACTION=__self 2024-06-26T07:56:31.9104446Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:56:31.9105305Z GITHUB_RUN_NUMBER=83498 2024-06-26T07:56:31.9105991Z TEST_CONFIG=distributed 2024-06-26T07:56:31.9106635Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:56:31.9107460Z AWS_DEFAULT_REGION=us-east-1 2024-06-26T07:56:31.9108273Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:56:31.9109033Z GITHUB_REF_TYPE=tag 2024-06-26T07:56:31.9109703Z *** 2024-06-26T07:56:31.9110202Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:56:31.9110926Z GITHUB_ACTIONS=true 2024-06-26T07:56:31.9111703Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.9113181Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.9114632Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/129470 2024-06-26T07:56:31.9115905Z VERBOSE_TEST_LOGS=False 2024-06-26T07:56:31.9116572Z GITHUB_REF=refs/tags/ciflow/trunk/129470 2024-06-26T07:56:31.9117312Z SHARD_NUMBER=1 2024-06-26T07:56:31.9117869Z GITHUB_REF_PROTECTED=false 2024-06-26T07:56:31.9118501Z HOME=/var/lib/jenkins 2024-06-26T07:56:31.9119162Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:56:31.9119976Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:56:31.9120732Z LANG=C.UTF-8 2024-06-26T07:56:31.9121283Z PYTORCH_TEST_WITH_ROCM=1 2024-06-26T07:56:31.9121898Z NUM_TEST_SHARDS=1 2024-06-26T07:56:31.9123506Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.9125601Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-06-26T07:56:31.9126758Z MAGMA_HOME=/opt/rocm/magma 2024-06-26T07:56:31.9128490Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.9129963Z HSAKMT_DEBUG_LEVEL=4 2024-06-26T07:56:31.9131186Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:56:31.9132414Z GITHUB_EVENT_NAME=push 2024-06-26T07:56:31.9133017Z GITHUB_RUN_ID=9673646592 2024-06-26T07:56:31.9134747Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.9136676Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:56:31.9137328Z PR_NUMBER= 2024-06-26T07:56:31.9137846Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:56:31.9138413Z VALGRIND=ON 2024-06-26T07:56:31.9138953Z ANACONDA_PYTHON_VERSION=3.8 2024-06-26T07:56:31.9139809Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:56:31.9140783Z TERM=xterm 2024-06-26T07:56:31.9141384Z INSTALLED_VISION=yes 2024-06-26T07:56:31.9142052Z BRANCH= 2024-06-26T07:56:31.9144279Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-06-26T07:56:31.9147295Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-06-26T07:56:31.9148704Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:56:31.9149474Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-06-26T07:56:31.9150171Z REENABLED_ISSUES= 2024-06-26T07:56:31.9150797Z SHLVL=1 2024-06-26T07:56:31.9151354Z MAX_JOBS=126 2024-06-26T07:56:31.9151985Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:56:31.9152972Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:56:31.9153954Z GITHUB_REF_NAME=ciflow/trunk/129470 2024-06-26T07:56:31.9154668Z ROCM_PATH=/opt/rocm 2024-06-26T07:56:31.9155711Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:56:31.9156840Z GITHUB_JOB=test 2024-06-26T07:56:31.9157386Z NO_TEST_TIMEOUT=False 2024-06-26T07:56:31.9158030Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:56:31.9158748Z LC_ALL=C.UTF-8 2024-06-26T07:56:31.9159292Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:56:31.9159930Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:56:31.9162309Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:56:31.9164711Z GITHUB_BASE_REF= 2024-06-26T07:56:31.9165232Z CI=true 2024-06-26T07:56:31.9165737Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-06-26T07:56:31.9166419Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:56:31.9167088Z JOB_ID=26688598647 2024-06-26T07:56:31.9167646Z INSTALLED_PROTOBUF=yes 2024-06-26T07:56:31.9168553Z GITHUB_HEAD_REF= 2024-06-26T07:56:31.9169090Z GITHUB_ACTION_REF= 2024-06-26T07:56:31.9169900Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:56:31.9170980Z GITHUB_WORKFLOW=trunk 2024-06-26T07:56:31.9171621Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:56:31.9173166Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_d56b091e-a346-4c11-8060-2026eb4402ad 2024-06-26T07:56:31.9174439Z NO_TD=False 2024-06-26T07:56:31.9174873Z OLDPWD=/var/lib/jenkins 2024-06-26T07:56:31.9175367Z _=/usr/bin/env 2024-06-26T07:56:31.9175874Z + echo 'Testing pytorch' 2024-06-26T07:56:31.9176393Z Testing pytorch 2024-06-26T07:56:31.9176876Z + export LANG=C.UTF-8 2024-06-26T07:56:31.9177385Z + LANG=C.UTF-8 2024-06-26T07:56:31.9177815Z + PR_NUMBER= 2024-06-26T07:56:31.9178298Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-06-26T07:56:31.9178985Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:56:31.9179791Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-06-26T07:56:31.9180483Z + export HIP_VISIBLE_DEVICES=0,1 2024-06-26T07:56:31.9181092Z + HIP_VISIBLE_DEVICES=0,1 2024-06-26T07:56:31.9181630Z + [[ distributed == \s\l\o\w ]] 2024-06-26T07:56:31.9182419Z + [[ linux-focal-rocm6.1-py3.8 == *slow-gradcheck* ]] 2024-06-26T07:56:31.9183286Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-06-26T07:56:31.9184053Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-06-26T07:56:31.9184792Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:56:31.9185508Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:56:31.9186166Z + [[ distributed == *crossref* ]] 2024-06-26T07:56:31.9186880Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-06-26T07:56:31.9187674Z + export VALGRIND=OFF 2024-06-26T07:56:31.9188154Z + VALGRIND=OFF 2024-06-26T07:56:31.9188581Z + rocminfo 2024-06-26T07:56:31.9252845Z ROCk module version 6.2.4 is loaded 2024-06-26T07:56:32.0041030Z ===================== 2024-06-26T07:56:32.0041675Z HSA System Attributes 2024-06-26T07:56:32.0042224Z ===================== 2024-06-26T07:56:32.0042770Z Runtime Version: 1.13 2024-06-26T07:56:32.0043363Z Runtime Ext Version: 1.4 2024-06-26T07:56:32.0044068Z System Timestamp Freq.: 1000.000000MHz 2024-06-26T07:56:32.0045098Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-06-26T07:56:32.0046269Z Machine Model: LARGE 2024-06-26T07:56:32.0047196Z System Endianness: LITTLE 2024-06-26T07:56:32.0048376Z Mwaitx: DISABLED 2024-06-26T07:56:32.0049025Z DMAbuf Support: YES 2024-06-26T07:56:32.0049463Z 2024-06-26T07:56:32.0049663Z ========== 2024-06-26T07:56:32.0050179Z HSA Agents 2024-06-26T07:56:32.0050674Z ========== 2024-06-26T07:56:32.0051299Z ******* 2024-06-26T07:56:32.0051777Z Agent 1 2024-06-26T07:56:32.0052253Z ******* 2024-06-26T07:56:32.0053116Z Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.0054081Z Uuid: CPU-XX 2024-06-26T07:56:32.0055066Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.0055996Z Vendor Name: CPU 2024-06-26T07:56:32.0056853Z Feature: None specified 2024-06-26T07:56:32.0057719Z Profile: FULL_PROFILE 2024-06-26T07:56:32.0058607Z Float Round Mode: NEAR 2024-06-26T07:56:32.0059541Z Max Queue Number: 0(0x0) 2024-06-26T07:56:32.0060408Z Queue Min Size: 0(0x0) 2024-06-26T07:56:32.0061277Z Queue Max Size: 0(0x0) 2024-06-26T07:56:32.0062106Z Queue Type: MULTI 2024-06-26T07:56:32.0062965Z Node: 0 2024-06-26T07:56:32.0063843Z Device Type: CPU 2024-06-26T07:56:32.0064923Z Cache Info: 2024-06-26T07:56:32.0065619Z L1: 32768(0x8000) KB 2024-06-26T07:56:32.0066457Z Chip ID: 0(0x0) 2024-06-26T07:56:32.0067335Z ASIC Revision: 0(0x0) 2024-06-26T07:56:32.0068250Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0069181Z Max Clock Freq. (MHz): 2000 2024-06-26T07:56:32.0070005Z BDFID: 0 2024-06-26T07:56:32.0070862Z Internal Node ID: 0 2024-06-26T07:56:32.0071703Z Compute Unit: 64 2024-06-26T07:56:32.0072552Z SIMDs per CU: 0 2024-06-26T07:56:32.0074164Z Shader Engines: 0 2024-06-26T07:56:32.0075063Z Shader Arrs. per Eng.: 0 2024-06-26T07:56:32.0075996Z WatchPts on Addr. Ranges:1 2024-06-26T07:56:32.0076778Z Features: None 2024-06-26T07:56:32.0077370Z Pool Info: 2024-06-26T07:56:32.0077914Z Pool 1 2024-06-26T07:56:32.0078646Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0079544Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:56:32.0080449Z Allocatable: TRUE 2024-06-26T07:56:32.0081526Z Alloc Granule: 4KB 2024-06-26T07:56:32.0082514Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0083472Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0084389Z Accessible by all: TRUE 2024-06-26T07:56:32.0085173Z Pool 2 2024-06-26T07:56:32.0085889Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-06-26T07:56:32.0086791Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:56:32.0087647Z Allocatable: TRUE 2024-06-26T07:56:32.0088636Z Alloc Granule: 4KB 2024-06-26T07:56:32.0089577Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0090539Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0091488Z Accessible by all: TRUE 2024-06-26T07:56:32.0092262Z Pool 3 2024-06-26T07:56:32.0092967Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0093842Z Size: 528200892(0x1f7bb4bc) KB 2024-06-26T07:56:32.0094702Z Allocatable: TRUE 2024-06-26T07:56:32.0095577Z Alloc Granule: 4KB 2024-06-26T07:56:32.0096514Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0097439Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0098332Z Accessible by all: TRUE 2024-06-26T07:56:32.0099083Z ISA Info: 2024-06-26T07:56:32.0099613Z ******* 2024-06-26T07:56:32.0100119Z Agent 2 2024-06-26T07:56:32.0100622Z ******* 2024-06-26T07:56:32.0101376Z Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.0102310Z Uuid: CPU-XX 2024-06-26T07:56:32.0103297Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.0104202Z Vendor Name: CPU 2024-06-26T07:56:32.0105199Z Feature: None specified 2024-06-26T07:56:32.0106051Z Profile: FULL_PROFILE 2024-06-26T07:56:32.0106898Z Float Round Mode: NEAR 2024-06-26T07:56:32.0107775Z Max Queue Number: 0(0x0) 2024-06-26T07:56:32.0108629Z Queue Min Size: 0(0x0) 2024-06-26T07:56:32.0109464Z Queue Max Size: 0(0x0) 2024-06-26T07:56:32.0110293Z Queue Type: MULTI 2024-06-26T07:56:32.0111093Z Node: 1 2024-06-26T07:56:32.0111884Z Device Type: CPU 2024-06-26T07:56:32.0112596Z Cache Info: 2024-06-26T07:56:32.0113239Z L1: 32768(0x8000) KB 2024-06-26T07:56:32.0114047Z Chip ID: 0(0x0) 2024-06-26T07:56:32.0114898Z ASIC Revision: 0(0x0) 2024-06-26T07:56:32.0115762Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0116629Z Max Clock Freq. (MHz): 2000 2024-06-26T07:56:32.0117458Z BDFID: 0 2024-06-26T07:56:32.0118274Z Internal Node ID: 1 2024-06-26T07:56:32.0119116Z Compute Unit: 64 2024-06-26T07:56:32.0120091Z SIMDs per CU: 0 2024-06-26T07:56:32.0120949Z Shader Engines: 0 2024-06-26T07:56:32.0121816Z Shader Arrs. per Eng.: 0 2024-06-26T07:56:32.0122699Z WatchPts on Addr. Ranges:1 2024-06-26T07:56:32.0123453Z Features: None 2024-06-26T07:56:32.0124021Z Pool Info: 2024-06-26T07:56:32.0124561Z Pool 1 2024-06-26T07:56:32.0125283Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0126268Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:56:32.0127141Z Allocatable: TRUE 2024-06-26T07:56:32.0128253Z Alloc Granule: 4KB 2024-06-26T07:56:32.0129198Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0130157Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0131054Z Accessible by all: TRUE 2024-06-26T07:56:32.0131815Z Pool 2 2024-06-26T07:56:32.0132529Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-06-26T07:56:32.0133403Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:56:32.0134256Z Allocatable: TRUE 2024-06-26T07:56:32.0135131Z Alloc Granule: 4KB 2024-06-26T07:56:32.0136047Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0136967Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0137850Z Accessible by all: TRUE 2024-06-26T07:56:32.0138591Z Pool 3 2024-06-26T07:56:32.0139293Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0140148Z Size: 528456664(0x1f7f9bd8) KB 2024-06-26T07:56:32.0140995Z Allocatable: TRUE 2024-06-26T07:56:32.0141847Z Alloc Granule: 4KB 2024-06-26T07:56:32.0142763Z Alloc Recommended Granule:4KB 2024-06-26T07:56:32.0143837Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0144741Z Accessible by all: TRUE 2024-06-26T07:56:32.0145479Z ISA Info: 2024-06-26T07:56:32.0146011Z ******* 2024-06-26T07:56:32.0146513Z Agent 3 2024-06-26T07:56:32.0147004Z ******* 2024-06-26T07:56:32.0147602Z Name: gfx90a 2024-06-26T07:56:32.0148528Z Uuid: GPU-34128c8046820a6f 2024-06-26T07:56:32.0149407Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0150269Z Vendor Name: AMD 2024-06-26T07:56:32.0151118Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0151945Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0152787Z Float Round Mode: NEAR 2024-06-26T07:56:32.0153657Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0154512Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0155351Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0156184Z Queue Type: MULTI 2024-06-26T07:56:32.0157009Z Node: 2 2024-06-26T07:56:32.0157822Z Device Type: GPU 2024-06-26T07:56:32.0158688Z Cache Info: 2024-06-26T07:56:32.0159332Z L1: 16(0x10) KB 2024-06-26T07:56:32.0160124Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0160948Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0161787Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0162651Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0163508Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0164323Z BDFID: 12800 2024-06-26T07:56:32.0165135Z Internal Node ID: 2 2024-06-26T07:56:32.0165967Z Compute Unit: 104 2024-06-26T07:56:32.0166790Z SIMDs per CU: 4 2024-06-26T07:56:32.0167621Z Shader Engines: 8 2024-06-26T07:56:32.0168571Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0169449Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0170333Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0171131Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0171945Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0172833Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0173717Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0174490Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0175201Z x 1024(0x400) 2024-06-26T07:56:32.0175966Z y 1024(0x400) 2024-06-26T07:56:32.0176719Z z 1024(0x400) 2024-06-26T07:56:32.0177545Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0178554Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0179454Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0180216Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0180861Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0181779Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0182542Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0183383Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0184324Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0185224Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0186091Z IOMMU Support:: None 2024-06-26T07:56:32.0186816Z Pool Info: 2024-06-26T07:56:32.0187374Z Pool 1 2024-06-26T07:56:32.0188088Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0188977Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0189846Z Allocatable: TRUE 2024-06-26T07:56:32.0190749Z Alloc Granule: 4KB 2024-06-26T07:56:32.0191679Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0192628Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0193536Z Accessible by all: FALSE 2024-06-26T07:56:32.0194303Z Pool 2 2024-06-26T07:56:32.0195012Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0195884Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0200264Z Allocatable: TRUE 2024-06-26T07:56:32.0201142Z Alloc Granule: 4KB 2024-06-26T07:56:32.0202055Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0202977Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0203879Z Accessible by all: FALSE 2024-06-26T07:56:32.0204635Z Pool 3 2024-06-26T07:56:32.0205322Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0206189Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0207025Z Allocatable: TRUE 2024-06-26T07:56:32.0208115Z Alloc Granule: 4KB 2024-06-26T07:56:32.0209038Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0209958Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0210845Z Accessible by all: FALSE 2024-06-26T07:56:32.0211588Z Pool 4 2024-06-26T07:56:32.0212248Z Segment: GROUP 2024-06-26T07:56:32.0213078Z Size: 64(0x40) KB 2024-06-26T07:56:32.0213954Z Allocatable: FALSE 2024-06-26T07:56:32.0214825Z Alloc Granule: 0KB 2024-06-26T07:56:32.0215728Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0216664Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0217559Z Accessible by all: FALSE 2024-06-26T07:56:32.0218303Z ISA Info: 2024-06-26T07:56:32.0218844Z ISA 1 2024-06-26T07:56:32.0219701Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0220660Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0221560Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0222469Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0223396Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0224435Z Fast f16: TRUE 2024-06-26T07:56:32.0225312Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0226107Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0226841Z x 1024(0x400) 2024-06-26T07:56:32.0227630Z y 1024(0x400) 2024-06-26T07:56:32.0228411Z z 1024(0x400) 2024-06-26T07:56:32.0229262Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0230041Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0230726Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0231513Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0232300Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0233150Z FBarrier Max Size: 32 2024-06-26T07:56:32.0233884Z ******* 2024-06-26T07:56:32.0234385Z Agent 4 2024-06-26T07:56:32.0234877Z ******* 2024-06-26T07:56:32.0235485Z Name: gfx90a 2024-06-26T07:56:32.0236426Z Uuid: GPU-b31dd2b5485cf42f 2024-06-26T07:56:32.0237325Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0238358Z Vendor Name: AMD 2024-06-26T07:56:32.0239196Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0240023Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0240864Z Float Round Mode: NEAR 2024-06-26T07:56:32.0241732Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0242595Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0243433Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0244264Z Queue Type: MULTI 2024-06-26T07:56:32.0245052Z Node: 3 2024-06-26T07:56:32.0245841Z Device Type: GPU 2024-06-26T07:56:32.0246552Z Cache Info: 2024-06-26T07:56:32.0247192Z L1: 16(0x10) KB 2024-06-26T07:56:32.0248083Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0248906Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0249724Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0250573Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0251424Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0252250Z BDFID: 13568 2024-06-26T07:56:32.0253059Z Internal Node ID: 3 2024-06-26T07:56:32.0253894Z Compute Unit: 104 2024-06-26T07:56:32.0254712Z SIMDs per CU: 4 2024-06-26T07:56:32.0255543Z Shader Engines: 8 2024-06-26T07:56:32.0256397Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0257284Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0258168Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0258959Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0259756Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0260649Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0261667Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0262456Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0263273Z x 1024(0x400) 2024-06-26T07:56:32.0264153Z y 1024(0x400) 2024-06-26T07:56:32.0264913Z z 1024(0x400) 2024-06-26T07:56:32.0265725Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0266719Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0267626Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0268379Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0269028Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0269801Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0270567Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0271410Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0272336Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0273230Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0274107Z IOMMU Support:: None 2024-06-26T07:56:32.0274830Z Pool Info: 2024-06-26T07:56:32.0275369Z Pool 1 2024-06-26T07:56:32.0276086Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0277122Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0277989Z Allocatable: TRUE 2024-06-26T07:56:32.0278886Z Alloc Granule: 4KB 2024-06-26T07:56:32.0279827Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0280780Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0281687Z Accessible by all: FALSE 2024-06-26T07:56:32.0282445Z Pool 2 2024-06-26T07:56:32.0283140Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0284020Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0284867Z Allocatable: TRUE 2024-06-26T07:56:32.0285747Z Alloc Granule: 4KB 2024-06-26T07:56:32.0286668Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0287585Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0288554Z Accessible by all: FALSE 2024-06-26T07:56:32.0289291Z Pool 3 2024-06-26T07:56:32.0289973Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0290865Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0291859Z Allocatable: TRUE 2024-06-26T07:56:32.0292864Z Alloc Granule: 4KB 2024-06-26T07:56:32.0293817Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0294729Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0295612Z Accessible by all: FALSE 2024-06-26T07:56:32.0296353Z Pool 4 2024-06-26T07:56:32.0297004Z Segment: GROUP 2024-06-26T07:56:32.0297830Z Size: 64(0x40) KB 2024-06-26T07:56:32.0298660Z Allocatable: FALSE 2024-06-26T07:56:32.0299511Z Alloc Granule: 0KB 2024-06-26T07:56:32.0300551Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0301491Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0302384Z Accessible by all: FALSE 2024-06-26T07:56:32.0303126Z ISA Info: 2024-06-26T07:56:32.0303658Z ISA 1 2024-06-26T07:56:32.0304495Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0305442Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0306352Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0307263Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0308182Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0309059Z Fast f16: TRUE 2024-06-26T07:56:32.0309934Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0310733Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0311466Z x 1024(0x400) 2024-06-26T07:56:32.0312257Z y 1024(0x400) 2024-06-26T07:56:32.0313104Z z 1024(0x400) 2024-06-26T07:56:32.0314099Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0315070Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0315758Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0316538Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0317314Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0318167Z FBarrier Max Size: 32 2024-06-26T07:56:32.0318900Z ******* 2024-06-26T07:56:32.0319417Z Agent 5 2024-06-26T07:56:32.0319912Z ******* 2024-06-26T07:56:32.0320521Z Name: gfx90a 2024-06-26T07:56:32.0321442Z Uuid: GPU-1661358e4203fb77 2024-06-26T07:56:32.0322317Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0323180Z Vendor Name: AMD 2024-06-26T07:56:32.0324017Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0324853Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0325695Z Float Round Mode: NEAR 2024-06-26T07:56:32.0326556Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0327410Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0328437Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0329306Z Queue Type: MULTI 2024-06-26T07:56:32.0330251Z Node: 4 2024-06-26T07:56:32.0331185Z Device Type: GPU 2024-06-26T07:56:32.0331934Z Cache Info: 2024-06-26T07:56:32.0332578Z L1: 16(0x10) KB 2024-06-26T07:56:32.0333368Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0334197Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0335078Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0335952Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0336913Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0337892Z BDFID: 4352 2024-06-26T07:56:32.0339049Z Internal Node ID: 4 2024-06-26T07:56:32.0340053Z Compute Unit: 104 2024-06-26T07:56:32.0340868Z SIMDs per CU: 4 2024-06-26T07:56:32.0341384Z Shader Engines: 8 2024-06-26T07:56:32.0341859Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0342345Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0342848Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0343284Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0343727Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0344215Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0344697Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0345124Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0345519Z x 1024(0x400) 2024-06-26T07:56:32.0345944Z y 1024(0x400) 2024-06-26T07:56:32.0346358Z z 1024(0x400) 2024-06-26T07:56:32.0346810Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0347360Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0347850Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0348351Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0348708Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0349126Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0349561Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0350031Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0350545Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0351039Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0351512Z IOMMU Support:: None 2024-06-26T07:56:32.0351918Z Pool Info: 2024-06-26T07:56:32.0352218Z Pool 1 2024-06-26T07:56:32.0352611Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0353153Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0353731Z Allocatable: TRUE 2024-06-26T07:56:32.0354317Z Alloc Granule: 4KB 2024-06-26T07:56:32.0354909Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0355408Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0355827Z Accessible by all: FALSE 2024-06-26T07:56:32.0356181Z Pool 2 2024-06-26T07:56:32.0356513Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0356918Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0357309Z Allocatable: TRUE 2024-06-26T07:56:32.0357714Z Alloc Granule: 4KB 2024-06-26T07:56:32.0358141Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0358586Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0359011Z Accessible by all: FALSE 2024-06-26T07:56:32.0359359Z Pool 3 2024-06-26T07:56:32.0359683Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0360076Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0360524Z Allocatable: TRUE 2024-06-26T07:56:32.0360934Z Alloc Granule: 4KB 2024-06-26T07:56:32.0361364Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0361793Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0362208Z Accessible by all: FALSE 2024-06-26T07:56:32.0362556Z Pool 4 2024-06-26T07:56:32.0362864Z Segment: GROUP 2024-06-26T07:56:32.0363252Z Size: 64(0x40) KB 2024-06-26T07:56:32.0363636Z Allocatable: FALSE 2024-06-26T07:56:32.0364033Z Alloc Granule: 0KB 2024-06-26T07:56:32.0364460Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0364889Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0365299Z Accessible by all: FALSE 2024-06-26T07:56:32.0365640Z ISA Info: 2024-06-26T07:56:32.0365896Z ISA 1 2024-06-26T07:56:32.0366288Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0366734Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0367168Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0367652Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0368137Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0368544Z Fast f16: TRUE 2024-06-26T07:56:32.0368943Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0369337Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0369748Z x 1024(0x400) 2024-06-26T07:56:32.0370246Z y 1024(0x400) 2024-06-26T07:56:32.0370768Z z 1024(0x400) 2024-06-26T07:56:32.0371292Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0371718Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0372097Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0372538Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0372971Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0373445Z FBarrier Max Size: 32 2024-06-26T07:56:32.0373864Z ******* 2024-06-26T07:56:32.0374146Z Agent 6 2024-06-26T07:56:32.0374422Z ******* 2024-06-26T07:56:32.0374760Z Name: gfx90a 2024-06-26T07:56:32.0375285Z Uuid: GPU-77ea2eecb6e14f86 2024-06-26T07:56:32.0375776Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0376264Z Vendor Name: AMD 2024-06-26T07:56:32.0376744Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0377208Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0377686Z Float Round Mode: NEAR 2024-06-26T07:56:32.0378167Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0378639Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0379105Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0379564Z Queue Type: MULTI 2024-06-26T07:56:32.0380080Z Node: 5 2024-06-26T07:56:32.0380521Z Device Type: GPU 2024-06-26T07:56:32.0380914Z Cache Info: 2024-06-26T07:56:32.0381270Z L1: 16(0x10) KB 2024-06-26T07:56:32.0381707Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0382167Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0382637Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0383122Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0383597Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0384050Z BDFID: 5120 2024-06-26T07:56:32.0384505Z Internal Node ID: 5 2024-06-26T07:56:32.0384972Z Compute Unit: 104 2024-06-26T07:56:32.0385432Z SIMDs per CU: 4 2024-06-26T07:56:32.0385907Z Shader Engines: 8 2024-06-26T07:56:32.0386388Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0386878Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0387377Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0387813Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0388342Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0388822Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0389296Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0389726Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0390115Z x 1024(0x400) 2024-06-26T07:56:32.0390540Z y 1024(0x400) 2024-06-26T07:56:32.0390951Z z 1024(0x400) 2024-06-26T07:56:32.0391401Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0391947Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0392440Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0392871Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0393302Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0393807Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0394300Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0394833Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0395328Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0395753Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0396161Z IOMMU Support:: None 2024-06-26T07:56:32.0396499Z Pool Info: 2024-06-26T07:56:32.0396753Z Pool 1 2024-06-26T07:56:32.0397086Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0397499Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0397901Z Allocatable: TRUE 2024-06-26T07:56:32.0398325Z Alloc Granule: 4KB 2024-06-26T07:56:32.0398761Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0399203Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0399624Z Accessible by all: FALSE 2024-06-26T07:56:32.0399979Z Pool 2 2024-06-26T07:56:32.0400371Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0400781Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0401171Z Allocatable: TRUE 2024-06-26T07:56:32.0401574Z Alloc Granule: 4KB 2024-06-26T07:56:32.0401999Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0402424Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0402839Z Accessible by all: FALSE 2024-06-26T07:56:32.0403187Z Pool 3 2024-06-26T07:56:32.0403508Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0403899Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0404301Z Allocatable: TRUE 2024-06-26T07:56:32.0404703Z Alloc Granule: 4KB 2024-06-26T07:56:32.0405132Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0405562Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0405981Z Accessible by all: FALSE 2024-06-26T07:56:32.0406324Z Pool 4 2024-06-26T07:56:32.0406630Z Segment: GROUP 2024-06-26T07:56:32.0407015Z Size: 64(0x40) KB 2024-06-26T07:56:32.0407470Z Allocatable: FALSE 2024-06-26T07:56:32.0407927Z Alloc Granule: 0KB 2024-06-26T07:56:32.0408348Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0408772Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0409195Z Accessible by all: FALSE 2024-06-26T07:56:32.0409602Z ISA Info: 2024-06-26T07:56:32.0409918Z ISA 1 2024-06-26T07:56:32.0410468Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0411089Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0411590Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0412089Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0412605Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0413089Z Fast f16: TRUE 2024-06-26T07:56:32.0413567Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0414009Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0414413Z x 1024(0x400) 2024-06-26T07:56:32.0414845Z y 1024(0x400) 2024-06-26T07:56:32.0415280Z z 1024(0x400) 2024-06-26T07:56:32.0415749Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0416176Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0416547Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0416981Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0417422Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0417898Z FBarrier Max Size: 32 2024-06-26T07:56:32.0418309Z ******* 2024-06-26T07:56:32.0418597Z Agent 7 2024-06-26T07:56:32.0418871Z ******* 2024-06-26T07:56:32.0419210Z Name: gfx90a 2024-06-26T07:56:32.0419817Z Uuid: GPU-1f7ea6a0f5503a53 2024-06-26T07:56:32.0420306Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0420787Z Vendor Name: AMD 2024-06-26T07:56:32.0421252Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0421714Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0422183Z Float Round Mode: NEAR 2024-06-26T07:56:32.0422678Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0423184Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0423660Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0424130Z Queue Type: MULTI 2024-06-26T07:56:32.0424569Z Node: 6 2024-06-26T07:56:32.0425013Z Device Type: GPU 2024-06-26T07:56:32.0425412Z Cache Info: 2024-06-26T07:56:32.0425767Z L1: 16(0x10) KB 2024-06-26T07:56:32.0426203Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0426650Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0427112Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0427588Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0428142Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0428590Z BDFID: 44544 2024-06-26T07:56:32.0429047Z Internal Node ID: 6 2024-06-26T07:56:32.0429592Z Compute Unit: 104 2024-06-26T07:56:32.0430125Z SIMDs per CU: 4 2024-06-26T07:56:32.0430678Z Shader Engines: 8 2024-06-26T07:56:32.0431223Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0431688Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0432106Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0432480Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0432854Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0433263Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0433659Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0434016Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0434344Z x 1024(0x400) 2024-06-26T07:56:32.0434691Z y 1024(0x400) 2024-06-26T07:56:32.0435037Z z 1024(0x400) 2024-06-26T07:56:32.0435416Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0435883Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0436296Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0436647Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0436937Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0437286Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0437636Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0438017Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0438438Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0438845Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0439240Z IOMMU Support:: None 2024-06-26T07:56:32.0439638Z Pool Info: 2024-06-26T07:56:32.0439894Z Pool 1 2024-06-26T07:56:32.0440233Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0440647Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0441048Z Allocatable: TRUE 2024-06-26T07:56:32.0441462Z Alloc Granule: 4KB 2024-06-26T07:56:32.0441894Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0442336Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0442756Z Accessible by all: FALSE 2024-06-26T07:56:32.0443111Z Pool 2 2024-06-26T07:56:32.0443443Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0443846Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0444242Z Allocatable: TRUE 2024-06-26T07:56:32.0444646Z Alloc Granule: 4KB 2024-06-26T07:56:32.0445070Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0445823Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0446233Z Accessible by all: FALSE 2024-06-26T07:56:32.0446580Z Pool 3 2024-06-26T07:56:32.0446955Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0447358Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0447834Z Allocatable: TRUE 2024-06-26T07:56:32.0448237Z Alloc Granule: 4KB 2024-06-26T07:56:32.0448657Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0449097Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0449582Z Accessible by all: FALSE 2024-06-26T07:56:32.0450026Z Pool 4 2024-06-26T07:56:32.0450467Z Segment: GROUP 2024-06-26T07:56:32.0451006Z Size: 64(0x40) KB 2024-06-26T07:56:32.0451475Z Allocatable: FALSE 2024-06-26T07:56:32.0451956Z Alloc Granule: 0KB 2024-06-26T07:56:32.0452459Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0452961Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0453449Z Accessible by all: FALSE 2024-06-26T07:56:32.0453855Z ISA Info: 2024-06-26T07:56:32.0454149Z ISA 1 2024-06-26T07:56:32.0454615Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0455140Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0455645Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0456145Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0456653Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0457135Z Fast f16: TRUE 2024-06-26T07:56:32.0457617Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0458055Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0458457Z x 1024(0x400) 2024-06-26T07:56:32.0458897Z y 1024(0x400) 2024-06-26T07:56:32.0459411Z z 1024(0x400) 2024-06-26T07:56:32.0459880Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0460302Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0460677Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0461104Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0461535Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0462002Z FBarrier Max Size: 32 2024-06-26T07:56:32.0462415Z ******* 2024-06-26T07:56:32.0462693Z Agent 8 2024-06-26T07:56:32.0462967Z ******* 2024-06-26T07:56:32.0463302Z Name: gfx90a 2024-06-26T07:56:32.0463818Z Uuid: GPU-36bcab48b29cafe0 2024-06-26T07:56:32.0464302Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0464788Z Vendor Name: AMD 2024-06-26T07:56:32.0465250Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0465707Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0466173Z Float Round Mode: NEAR 2024-06-26T07:56:32.0466650Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0467121Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0467667Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0468146Z Queue Type: MULTI 2024-06-26T07:56:32.0468600Z Node: 7 2024-06-26T07:56:32.0469045Z Device Type: GPU 2024-06-26T07:56:32.0469441Z Cache Info: 2024-06-26T07:56:32.0469798Z L1: 16(0x10) KB 2024-06-26T07:56:32.0470232Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0470682Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0471142Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0471619Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0472090Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0472544Z BDFID: 45824 2024-06-26T07:56:32.0472999Z Internal Node ID: 7 2024-06-26T07:56:32.0473546Z Compute Unit: 104 2024-06-26T07:56:32.0474083Z SIMDs per CU: 4 2024-06-26T07:56:32.0474625Z Shader Engines: 8 2024-06-26T07:56:32.0475156Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0475623Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0476036Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0476402Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0476775Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0477182Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0477592Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0477960Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0478281Z x 1024(0x400) 2024-06-26T07:56:32.0478639Z y 1024(0x400) 2024-06-26T07:56:32.0478986Z z 1024(0x400) 2024-06-26T07:56:32.0479363Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0479885Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0480306Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0480654Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0480950Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0481309Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0481661Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0482051Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0482481Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0482891Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0483286Z IOMMU Support:: None 2024-06-26T07:56:32.0483621Z Pool Info: 2024-06-26T07:56:32.0483870Z Pool 1 2024-06-26T07:56:32.0484206Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0484614Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0485018Z Allocatable: TRUE 2024-06-26T07:56:32.0485430Z Alloc Granule: 4KB 2024-06-26T07:56:32.0485862Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0486298Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0486800Z Accessible by all: FALSE 2024-06-26T07:56:32.0487150Z Pool 2 2024-06-26T07:56:32.0487479Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0487926Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0488323Z Allocatable: TRUE 2024-06-26T07:56:32.0488740Z Alloc Granule: 4KB 2024-06-26T07:56:32.0489163Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0489656Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0490197Z Accessible by all: FALSE 2024-06-26T07:56:32.0490689Z Pool 3 2024-06-26T07:56:32.0491133Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0491610Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0492077Z Allocatable: TRUE 2024-06-26T07:56:32.0492555Z Alloc Granule: 4KB 2024-06-26T07:56:32.0493058Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0493564Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0494054Z Accessible by all: FALSE 2024-06-26T07:56:32.0494462Z Pool 4 2024-06-26T07:56:32.0494819Z Segment: GROUP 2024-06-26T07:56:32.0495274Z Size: 64(0x40) KB 2024-06-26T07:56:32.0495736Z Allocatable: FALSE 2024-06-26T07:56:32.0496214Z Alloc Granule: 0KB 2024-06-26T07:56:32.0496713Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0497228Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0497722Z Accessible by all: FALSE 2024-06-26T07:56:32.0498129Z ISA Info: 2024-06-26T07:56:32.0498425Z ISA 1 2024-06-26T07:56:32.0498880Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0499486Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0499991Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0500487Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0500995Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0501474Z Fast f16: TRUE 2024-06-26T07:56:32.0501950Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0502390Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0502798Z x 1024(0x400) 2024-06-26T07:56:32.0503275Z y 1024(0x400) 2024-06-26T07:56:32.0503782Z z 1024(0x400) 2024-06-26T07:56:32.0504339Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0504857Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0505240Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0505638Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0505999Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0506389Z FBarrier Max Size: 32 2024-06-26T07:56:32.0506732Z ******* 2024-06-26T07:56:32.0506965Z Agent 9 2024-06-26T07:56:32.0507265Z ******* 2024-06-26T07:56:32.0507545Z Name: gfx90a 2024-06-26T07:56:32.0507977Z Uuid: GPU-9ab8502ae477e3b0 2024-06-26T07:56:32.0508383Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0508787Z Vendor Name: AMD 2024-06-26T07:56:32.0509178Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0509565Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0509965Z Float Round Mode: NEAR 2024-06-26T07:56:32.0510369Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0510765Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0511160Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0511549Z Queue Type: MULTI 2024-06-26T07:56:32.0511922Z Node: 8 2024-06-26T07:56:32.0512288Z Device Type: GPU 2024-06-26T07:56:32.0512617Z Cache Info: 2024-06-26T07:56:32.0512918Z L1: 16(0x10) KB 2024-06-26T07:56:32.0513283Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0513669Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0514054Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0514448Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0514842Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0515219Z BDFID: 36352 2024-06-26T07:56:32.0515596Z Internal Node ID: 8 2024-06-26T07:56:32.0515988Z Compute Unit: 104 2024-06-26T07:56:32.0516367Z SIMDs per CU: 4 2024-06-26T07:56:32.0516753Z Shader Engines: 8 2024-06-26T07:56:32.0517154Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0517562Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0518033Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0518408Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0518779Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0519190Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0519598Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0519957Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0520286Z x 1024(0x400) 2024-06-26T07:56:32.0520638Z y 1024(0x400) 2024-06-26T07:56:32.0520985Z z 1024(0x400) 2024-06-26T07:56:32.0521369Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0521830Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0522248Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0522597Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0522898Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0523256Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0523617Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0524001Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0524427Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0524903Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0525300Z IOMMU Support:: None 2024-06-26T07:56:32.0525632Z Pool Info: 2024-06-26T07:56:32.0525881Z Pool 1 2024-06-26T07:56:32.0526211Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0526623Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0527023Z Allocatable: TRUE 2024-06-26T07:56:32.0527435Z Alloc Granule: 4KB 2024-06-26T07:56:32.0527616Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0527815Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0527980Z Accessible by all: FALSE 2024-06-26T07:56:32.0528066Z Pool 2 2024-06-26T07:56:32.0528238Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0528378Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0528537Z Allocatable: TRUE 2024-06-26T07:56:32.0528688Z Alloc Granule: 4KB 2024-06-26T07:56:32.0528857Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0529012Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0529172Z Accessible by all: FALSE 2024-06-26T07:56:32.0529279Z Pool 3 2024-06-26T07:56:32.0529455Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0529615Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0529785Z Allocatable: TRUE 2024-06-26T07:56:32.0529998Z Alloc Granule: 4KB 2024-06-26T07:56:32.0530232Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0530453Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0530672Z Accessible by all: FALSE 2024-06-26T07:56:32.0530789Z Pool 4 2024-06-26T07:56:32.0531079Z Segment: GROUP 2024-06-26T07:56:32.0531237Z Size: 64(0x40) KB 2024-06-26T07:56:32.0531414Z Allocatable: FALSE 2024-06-26T07:56:32.0531586Z Alloc Granule: 0KB 2024-06-26T07:56:32.0531784Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0531960Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0532152Z Accessible by all: FALSE 2024-06-26T07:56:32.0532254Z ISA Info: 2024-06-26T07:56:32.0532360Z ISA 1 2024-06-26T07:56:32.0532629Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0532825Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0533012Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0533202Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0533394Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0533557Z Fast f16: TRUE 2024-06-26T07:56:32.0533747Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0533875Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0534034Z x 1024(0x400) 2024-06-26T07:56:32.0534268Z y 1024(0x400) 2024-06-26T07:56:32.0534425Z z 1024(0x400) 2024-06-26T07:56:32.0534607Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0534729Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0534886Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0535036Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0535198Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0535378Z FBarrier Max Size: 32 2024-06-26T07:56:32.0535476Z ******* 2024-06-26T07:56:32.0535573Z Agent 10 2024-06-26T07:56:32.0535668Z ******* 2024-06-26T07:56:32.0535828Z Name: gfx90a 2024-06-26T07:56:32.0536056Z Uuid: GPU-d313938d3c8e3d86 2024-06-26T07:56:32.0536241Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.0536411Z Vendor Name: AMD 2024-06-26T07:56:32.0536577Z Feature: KERNEL_DISPATCH 2024-06-26T07:56:32.0536745Z Profile: BASE_PROFILE 2024-06-26T07:56:32.0536932Z Float Round Mode: NEAR 2024-06-26T07:56:32.0537106Z Max Queue Number: 128(0x80) 2024-06-26T07:56:32.0537281Z Queue Min Size: 64(0x40) 2024-06-26T07:56:32.0537448Z Queue Max Size: 131072(0x20000) 2024-06-26T07:56:32.0537615Z Queue Type: MULTI 2024-06-26T07:56:32.0537760Z Node: 9 2024-06-26T07:56:32.0537936Z Device Type: GPU 2024-06-26T07:56:32.0538039Z Cache Info: 2024-06-26T07:56:32.0538206Z L1: 16(0x10) KB 2024-06-26T07:56:32.0538352Z L2: 8192(0x2000) KB 2024-06-26T07:56:32.0538523Z Chip ID: 29708(0x740c) 2024-06-26T07:56:32.0538766Z ASIC Revision: 1(0x1) 2024-06-26T07:56:32.0538941Z Cacheline Size: 64(0x40) 2024-06-26T07:56:32.0539126Z Max Clock Freq. (MHz): 1700 2024-06-26T07:56:32.0539274Z BDFID: 37632 2024-06-26T07:56:32.0539454Z Internal Node ID: 9 2024-06-26T07:56:32.0539620Z Compute Unit: 104 2024-06-26T07:56:32.0539795Z SIMDs per CU: 4 2024-06-26T07:56:32.0539965Z Shader Engines: 8 2024-06-26T07:56:32.0540150Z Shader Arrs. per Eng.: 1 2024-06-26T07:56:32.0540334Z WatchPts on Addr. Ranges:4 2024-06-26T07:56:32.0540520Z Coherent Host Access: FALSE 2024-06-26T07:56:32.0540654Z Features: KERNEL_DISPATCH 2024-06-26T07:56:32.0540848Z Fast F16 Operation: TRUE 2024-06-26T07:56:32.0541024Z Wavefront Size: 64(0x40) 2024-06-26T07:56:32.0541203Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0541333Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0541479Z x 1024(0x400) 2024-06-26T07:56:32.0541623Z y 1024(0x400) 2024-06-26T07:56:32.0541839Z z 1024(0x400) 2024-06-26T07:56:32.0542021Z Max Waves Per CU: 32(0x20) 2024-06-26T07:56:32.0542258Z Max Work-item Per CU: 2048(0x800) 2024-06-26T07:56:32.0542440Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0542558Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0542719Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0542870Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0543013Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0543210Z Max fbarriers/Workgrp: 32 2024-06-26T07:56:32.0543432Z Packet Processor uCode:: 78 2024-06-26T07:56:32.0543644Z SDMA engine uCode:: 8 2024-06-26T07:56:32.0543855Z IOMMU Support:: None 2024-06-26T07:56:32.0543985Z Pool Info: 2024-06-26T07:56:32.0544108Z Pool 1 2024-06-26T07:56:32.0544337Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-06-26T07:56:32.0544533Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0544745Z Allocatable: TRUE 2024-06-26T07:56:32.0544975Z Alloc Granule: 4KB 2024-06-26T07:56:32.0545199Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0545385Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0545566Z Accessible by all: FALSE 2024-06-26T07:56:32.0545670Z Pool 2 2024-06-26T07:56:32.0545831Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-06-26T07:56:32.0545973Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0546122Z Allocatable: TRUE 2024-06-26T07:56:32.0546273Z Alloc Granule: 4KB 2024-06-26T07:56:32.0546435Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0546590Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0546801Z Accessible by all: FALSE 2024-06-26T07:56:32.0546894Z Pool 3 2024-06-26T07:56:32.0547048Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-06-26T07:56:32.0547177Z Size: 67092480(0x3ffc000) KB 2024-06-26T07:56:32.0547327Z Allocatable: TRUE 2024-06-26T07:56:32.0547470Z Alloc Granule: 4KB 2024-06-26T07:56:32.0547648Z Alloc Recommended Granule:2048KB 2024-06-26T07:56:32.0547795Z Alloc Alignment: 4KB 2024-06-26T07:56:32.0547951Z Accessible by all: FALSE 2024-06-26T07:56:32.0548033Z Pool 4 2024-06-26T07:56:32.0548175Z Segment: GROUP 2024-06-26T07:56:32.0548305Z Size: 64(0x40) KB 2024-06-26T07:56:32.0548457Z Allocatable: FALSE 2024-06-26T07:56:32.0548605Z Alloc Granule: 0KB 2024-06-26T07:56:32.0548768Z Alloc Recommended Granule:0KB 2024-06-26T07:56:32.0548920Z Alloc Alignment: 0KB 2024-06-26T07:56:32.0549072Z Accessible by all: FALSE 2024-06-26T07:56:32.0549222Z ISA Info: 2024-06-26T07:56:32.0549305Z ISA 1 2024-06-26T07:56:32.0549530Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-06-26T07:56:32.0549690Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-06-26T07:56:32.0549841Z Profiles: HSA_PROFILE_BASE 2024-06-26T07:56:32.0549999Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0550164Z Default Rounding Mode: NEAR 2024-06-26T07:56:32.0550301Z Fast f16: TRUE 2024-06-26T07:56:32.0550459Z Workgroup Max Size: 1024(0x400) 2024-06-26T07:56:32.0550571Z Workgroup Max Size per Dimension: 2024-06-26T07:56:32.0550700Z x 1024(0x400) 2024-06-26T07:56:32.0550830Z y 1024(0x400) 2024-06-26T07:56:32.0550960Z z 1024(0x400) 2024-06-26T07:56:32.0551116Z Grid Max Size: 4294967295(0xffffffff) 2024-06-26T07:56:32.0551213Z Grid Max Size per Dimension: 2024-06-26T07:56:32.0551343Z x 4294967295(0xffffffff) 2024-06-26T07:56:32.0551467Z y 4294967295(0xffffffff) 2024-06-26T07:56:32.0551605Z z 4294967295(0xffffffff) 2024-06-26T07:56:32.0551758Z FBarrier Max Size: 32 2024-06-26T07:56:32.0551846Z *** Done *** 2024-06-26T07:56:32.0551923Z + rocminfo 2024-06-26T07:56:32.0552056Z + grep -E 'Name:.*\sgfx|Marketing' 2024-06-26T07:56:32.1126312Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.1126780Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-06-26T07:56:32.1127096Z Name: gfx90a 2024-06-26T07:56:32.1127433Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1127879Z Name: gfx90a 2024-06-26T07:56:32.1128248Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1128552Z Name: gfx90a 2024-06-26T07:56:32.1129734Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1130122Z Name: gfx90a 2024-06-26T07:56:32.1130517Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1130838Z Name: gfx90a 2024-06-26T07:56:32.1131231Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1131546Z Name: gfx90a 2024-06-26T07:56:32.1137358Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1137783Z Name: gfx90a 2024-06-26T07:56:32.1138233Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1138617Z Name: gfx90a 2024-06-26T07:56:32.1138993Z Marketing Name: AMD Instinct MI250X/MI250 2024-06-26T07:56:32.1305171Z + [[ linux-focal-rocm6.1-py3.8 == *xpu* ]] 2024-06-26T07:56:32.1305814Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-06-26T07:56:32.1306142Z + pip_install --user ninja==1.10.2 2024-06-26T07:56:32.1306651Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:56:32.8972442Z Collecting ninja==1.10.2 2024-06-26T07:56:33.1859792Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:56:33.2733091Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:56:34.1915843Z Installing collected packages: ninja 2024-06-26T07:56:34.1987559Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:56:34.1989956Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:56:34.2007524Z Successfully installed ninja-1.10.2 2024-06-26T07:56:34.2602930Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:56:34.2608183Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:56:34.2611066Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-06-26T07:56:34.2611884Z + install_tlparse 2024-06-26T07:56:34.2612550Z + pip_install --user tlparse==0.3.7 2024-06-26T07:56:34.2613532Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:56:34.9776405Z Collecting tlparse==0.3.7 2024-06-26T07:56:35.2120579Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:56:35.2925480Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:56:36.5498223Z Installing collected packages: tlparse 2024-06-26T07:56:36.5806094Z Successfully installed tlparse-0.3.7 2024-06-26T07:56:36.6419419Z ++ python -m site --user-base 2024-06-26T07:56:36.6786537Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:56:36.6789996Z + [[ linux-focal-rocm6.1-py3.8 == *asan* ]] 2024-06-26T07:56:36.6790975Z + [[ linux-focal-rocm6.1-py3.8 == *-debug* ]] 2024-06-26T07:56:36.6791939Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-06-26T07:56:36.6793400Z + echo 'We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass' 2024-06-26T07:56:36.6795149Z We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass 2024-06-26T07:56:36.6796252Z + cd test 2024-06-26T07:56:36.6797629Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:56:38.2528557Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:56:38.2529740Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:56:38.2531077Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-06-26T07:56:38.2531945Z + pushd test 2024-06-26T07:56:38.2532490Z ~/pytorch/test ~/pytorch 2024-06-26T07:56:38.2540215Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:56:39.7589598Z + CUDA_VERSION=None 2024-06-26T07:56:39.7590715Z + '[' None == 12.4 ']' 2024-06-26T07:56:39.7591344Z + ISCUDA124= 2024-06-26T07:56:39.7591842Z + popd 2024-06-26T07:56:39.7592292Z ~/pytorch 2024-06-26T07:56:39.7595791Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:56:39.7596618Z + [[ distributed == *dynamo_eager* ]] 2024-06-26T07:56:39.7597412Z + [[ distributed == *aot_eager* ]] 2024-06-26T07:56:39.7598194Z + [[ distributed == *aot_inductor* ]] 2024-06-26T07:56:39.7599095Z + [[ distributed == *inductor* ]] 2024-06-26T07:56:39.7599917Z + [[ distributed == *dynamic* ]] 2024-06-26T07:56:39.7600697Z + [[ distributed == *cpu_inductor* ]] 2024-06-26T07:56:39.7601652Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:56:39.7638150Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-06-26T07:56:39.7639262Z + [[ linux-focal-rocm6.1-py3.8 == *-bazel-* ]] 2024-06-26T07:56:39.7643331Z + cd test 2024-06-26T07:56:39.7644368Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:56:41.6095464Z PyTorch built with: 2024-06-26T07:56:41.6096375Z - GCC 9.4 2024-06-26T07:56:41.6096982Z - C++ Version: 201703 2024-06-26T07:56:41.6098730Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:56:41.6100855Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:56:41.6102036Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:56:41.6102965Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:56:41.6103810Z - NNPACK is enabled 2024-06-26T07:56:41.6104476Z - CPU capability usage: AVX2 2024-06-26T07:56:41.6105294Z - HIP Runtime 6.1.40091 2024-06-26T07:56:41.6106046Z - MIOpen 3.1.0 2024-06-26T07:56:41.6106703Z - Magma 2.7.2 2024-06-26T07:56:41.6119992Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:56:41.6131627Z 2024-06-26T07:56:42.0249160Z + cd test 2024-06-26T07:56:42.0250283Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:56:43.1775317Z ATen/Parallel: 2024-06-26T07:56:43.1776027Z at::get_num_threads() : 128 2024-06-26T07:56:43.1776750Z at::get_num_interop_threads() : 128 2024-06-26T07:56:43.1777574Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:56:43.1778399Z omp_get_max_threads() : 128 2024-06-26T07:56:43.1780923Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:56:43.1782425Z mkl_get_max_threads() : 128 2024-06-26T07:56:43.1783562Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:56:43.1784686Z std::thread::hardware_concurrency() : 128 2024-06-26T07:56:43.1785424Z Environment variables: 2024-06-26T07:56:43.1786021Z OMP_NUM_THREADS : [not set] 2024-06-26T07:56:43.1786656Z MKL_NUM_THREADS : [not set] 2024-06-26T07:56:43.1787302Z ATen parallel backend: OpenMP 2024-06-26T07:56:43.1787752Z 2024-06-26T07:56:43.5831614Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-06-26T07:56:43.5832626Z + [[ distributed == *backward* ]] 2024-06-26T07:56:43.5833343Z + [[ distributed == *xla* ]] 2024-06-26T07:56:43.5834011Z + [[ distributed == *executorch* ]] 2024-06-26T07:56:43.5834872Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:56:43.5836028Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-06-26T07:56:43.5836974Z + [[ distributed == distributed ]] 2024-06-26T07:56:43.5837650Z + test_distributed 2024-06-26T07:56:43.5838368Z + echo 'Testing distributed python tests' 2024-06-26T07:56:43.5839159Z Testing distributed python tests 2024-06-26T07:56:43.5841999Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-06-26T07:56:43.6880361Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T07:56:43.6882811Z import pkg_resources 2024-06-26T07:56:47.1305444Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-06-26T07:56:47.4436689Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-06-26T07:56:47.8693137Z Ignoring disabled issues: [''] 2024-06-26T07:56:47.8857892Z Found test times from artifacts 2024-06-26T07:56:47.9275782Z Found test times from artifacts 2024-06-26T07:56:47.9284574Z Running all tests 2024-06-26T07:56:47.9288548Z Running parallel tests on 8 processes 2024-06-26T07:56:47.9289463Z Name: tests to run (est. time: 34.46min) 2024-06-26T07:56:47.9290215Z Serial tests (5): 2024-06-26T07:56:47.9290830Z distributed/test_c10d_common 1/1 2024-06-26T07:56:47.9291569Z distributed/test_c10d_nccl 1/4 2024-06-26T07:56:47.9292286Z distributed/test_c10d_nccl 2/4 2024-06-26T07:56:47.9292964Z distributed/test_c10d_nccl 3/4 2024-06-26T07:56:47.9293655Z distributed/test_c10d_nccl 4/4 2024-06-26T07:56:47.9294318Z Parallel tests (0): 2024-06-26T07:56:47.9294919Z Name: excluded (est. time: 0.0min) 2024-06-26T07:56:47.9295558Z Serial tests (0): 2024-06-26T07:56:47.9296115Z Parallel tests (0): 2024-06-26T07:56:47.9297260Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T07:56:48.0247315Z Running distributed/test_c10d_common 1/1 ... [2024-06-26 07:56:48.023889] 2024-06-26T07:56:48.0257829Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 07:56:48.025024] 2024-06-26T07:59:30.4998620Z 2024-06-26T07:59:30.5004316Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_55a0b668d6f81b99_.log 2024-06-26T07:59:30.5024972Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-06-26T07:59:30.5044971Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-06-26T07:59:30.5047126Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-06-26T07:59:30.5049546Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-06-26T07:59:30.5051909Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-06-26T07:59:30.5054239Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-06-26T07:59:30.5056282Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-06-26T07:59:30.5058302Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-06-26T07:59:30.5060558Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-06-26T07:59:30.5062788Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-06-26T07:59:30.5065025Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-06-26T07:59:30.5067505Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-06-26T07:59:30.5070009Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-06-26T07:59:30.5072254Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-06-26T07:59:30.5074743Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-06-26T07:59:30.5077775Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-06-26T07:59:30.5080222Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-06-26T07:59:30.5082176Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-06-26T07:59:30.5084112Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-06-26T07:59:30.5085972Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-06-26T07:59:30.5088065Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-06-26T07:59:30.5090080Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-06-26T07:59:30.5092059Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-06-26T07:59:30.5093958Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-06-26T07:59:30.5095108Z 2024-06-26T07:59:30.5095781Z Running distributed/test_c10d_nccl 1/4 ... [2024-06-26 07:59:30.500807] 2024-06-26T07:59:30.5099070Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 07:59:30.501424] 2024-06-26T08:05:06.8099387Z 2024-06-26T08:05:06.8101703Z distributed/test_c10d_nccl 1/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.4_c165474cf5f32955_.log 2024-06-26T08:05:06.8140298Z Running 37 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_into_tensor_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CompilerTest::test_scatter_work_wait_gpu, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True, test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-06-26T08:05:06.8172520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-06-26T08:05:06.8174507Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg 2024-06-26T08:05:06.8177392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-06-26T08:05:06.8179876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-06-26T08:05:06.8182046Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-06-26T08:05:06.8184205Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-06-26T08:05:06.8186338Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-06-26T08:05:06.8188943Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-06-26T08:05:06.8191512Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-06-26T08:05:06.8193890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-06-26T08:05:06.8196270Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-06-26T08:05:06.8198924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-06-26T08:05:06.8201585Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-06-26T08:05:06.8204099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-06-26T08:05:06.8206420Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-06-26T08:05:06.8208759Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-06-26T08:05:06.8211062Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-06-26T08:05:06.8213914Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-06-26T08:05:06.8216767Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-06-26T08:05:06.8219457Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-06-26T08:05:06.8221699Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-06-26T08:05:06.8223865Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-06-26T08:05:06.8226195Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-06-26T08:05:06.8228406Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-06-26T08:05:06.8230392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-06-26T08:05:06.8232472Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_into_tensor_work_wait_gpu 2024-06-26T08:05:06.8234494Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k 2024-06-26T08:05:06.8236412Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_scatter_work_wait_gpu 2024-06-26T08:05:06.8238652Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-06-26T08:05:06.8241497Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-06-26T08:05:06.8243808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-06-26T08:05:06.8245797Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-06-26T08:05:06.8248248Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-06-26T08:05:06.8250807Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-06-26T08:05:06.8253153Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-06-26T08:05:06.8255506Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-06-26T08:05:06.8258020Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-06-26T08:05:06.8259330Z 2024-06-26T08:05:06.8260307Z Running distributed/test_c10d_nccl 2/4 ... [2024-06-26 08:05:06.810927] 2024-06-26T08:05:06.8263604Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 08:05:06.811523] 2024-06-26T08:13:56.6156251Z 2024-06-26T08:13:56.6158495Z distributed/test_c10d_nccl 2/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.4_f2044ca54a0b4b2b_.log 2024-06-26T08:13:56.6225410Z Running 51 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::CompilerTest::test_broadcast_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_nested_comm_tensor_wrapping, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_work_wait_gpu, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks 2024-06-26T08:13:56.6271919Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-06-26T08:13:56.6274255Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-06-26T08:13:56.6277040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-06-26T08:13:56.6279620Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-06-26T08:13:56.6282026Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-06-26T08:13:56.6284546Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-06-26T08:13:56.6287193Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-06-26T08:13:56.6290196Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-06-26T08:13:56.6292869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-06-26T08:13:56.6295703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-06-26T08:13:56.6298279Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-06-26T08:13:56.6300686Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-06-26T08:13:56.6303239Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-06-26T08:13:56.6306223Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-06-26T08:13:56.6308791Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-06-26T08:13:56.6311268Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-06-26T08:13:56.6313533Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-06-26T08:13:56.6315950Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-06-26T08:13:56.6318592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-06-26T08:13:56.6320947Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-06-26T08:13:56.6323126Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-06-26T08:13:56.6325846Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-06-26T08:13:56.6328546Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-06-26T08:13:56.6331393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-06-26T08:13:56.6333404Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-06-26T08:13:56.6335369Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-06-26T08:13:56.6337470Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-06-26T08:13:56.6339695Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-06-26T08:13:56.6341821Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-06-26T08:13:56.6343658Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-06-26T08:13:56.6345371Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-06-26T08:13:56.6347144Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_broadcast_work_wait_gpu 2024-06-26T08:13:56.6349232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_nested_comm_tensor_wrapping 2024-06-26T08:13:56.6351280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_work_wait_gpu 2024-06-26T08:13:56.6353134Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_work_wait_gpu 2024-06-26T08:13:56.6355334Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-06-26T08:13:56.6357425Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-06-26T08:13:56.6359195Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-06-26T08:13:56.6361059Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-06-26T08:13:56.6362974Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0 2024-06-26T08:13:56.6365155Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-06-26T08:13:56.6367446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-06-26T08:13:56.6369866Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-06-26T08:13:56.6371956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-06-26T08:13:56.6373818Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-06-26T08:13:56.6375960Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-06-26T08:13:56.6378015Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-06-26T08:13:56.6380185Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False_include_collectives_False 2024-06-26T08:13:56.6382712Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True_include_collectives_False 2024-06-26T08:13:56.6384942Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-06-26T08:13:56.6387258Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks 2024-06-26T08:13:56.6388725Z 2024-06-26T08:13:56.6389726Z Running distributed/test_c10d_nccl 3/4 ... [2024-06-26 08:13:56.618941] 2024-06-26T08:13:56.6392681Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 08:13:56.619536] 2024-06-26T08:21:06.4218353Z 2024-06-26T08:21:06.4221476Z distributed/test_c10d_nccl 3/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.4_94de54b3e138ddd8_.log 2024-06-26T08:21:06.4266507Z Running 37 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CompilerTest::test_alltoall_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True 2024-06-26T08:21:06.4300540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-06-26T08:21:06.4302862Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-06-26T08:21:06.4305123Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-06-26T08:21:06.4307672Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-06-26T08:21:06.4310249Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-06-26T08:21:06.4312507Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-06-26T08:21:06.4315282Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-06-26T08:21:06.4318093Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-06-26T08:21:06.4320693Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-06-26T08:21:06.4323380Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-06-26T08:21:06.4325864Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-06-26T08:21:06.4328524Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-06-26T08:21:06.4330842Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-06-26T08:21:06.4333185Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-06-26T08:21:06.4335718Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-06-26T08:21:06.4338350Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-06-26T08:21:06.4341569Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-06-26T08:21:06.4344582Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-06-26T08:21:06.4347055Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-06-26T08:21:06.4349329Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-06-26T08:21:06.4351957Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-06-26T08:21:06.4353943Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-06-26T08:21:06.4355740Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-06-26T08:21:06.4357765Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-06-26T08:21:06.4359924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-06-26T08:21:06.4362008Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-06-26T08:21:06.4363987Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_alltoall_work_wait_gpu 2024-06-26T08:21:06.4366003Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced 2024-06-26T08:21:06.4368515Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-06-26T08:21:06.4371159Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-06-26T08:21:06.4373571Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-06-26T08:21:06.4375839Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-06-26T08:21:06.4378331Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-06-26T08:21:06.4381182Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-06-26T08:21:06.4384059Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-06-26T08:21:06.4386528Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False 2024-06-26T08:21:06.4388997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True 2024-06-26T08:21:06.4390588Z 2024-06-26T08:21:06.4391654Z Running distributed/test_c10d_nccl 4/4 ... [2024-06-26 08:21:06.424254] 2024-06-26T08:21:06.4394955Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=4', '--num-shards=4', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-06-26 08:21:06.424867] 2024-06-26T08:32:43.1783819Z 2024-06-26T08:32:43.1786538Z distributed/test_c10d_nccl 4/4 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_4.4_f9b56d7442405cdc_.log 2024-06-26T08:32:43.1858062Z Running 65 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_allreduce_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_consecutive_comm_work_wait_gpu, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-06-26T08:32:43.1913659Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-06-26T08:32:43.1915850Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-06-26T08:32:43.1918262Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-06-26T08:32:43.1920572Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-06-26T08:32:43.1922829Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization 2024-06-26T08:32:43.1924923Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-06-26T08:32:43.1927092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-06-26T08:32:43.1929544Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-06-26T08:32:43.1931793Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-06-26T08:32:43.1934140Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-06-26T08:32:43.1936473Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-06-26T08:32:43.1938713Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-06-26T08:32:43.1940994Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-06-26T08:32:43.1943495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-06-26T08:32:43.1946086Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-06-26T08:32:43.1948783Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-06-26T08:32:43.1951492Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-06-26T08:32:43.1954313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-06-26T08:32:43.1957067Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-06-26T08:32:43.1959508Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-06-26T08:32:43.1961736Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-06-26T08:32:43.1964045Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-06-26T08:32:43.1966495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-06-26T08:32:43.1969095Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-06-26T08:32:43.1971747Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-06-26T08:32:43.1973992Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-06-26T08:32:43.1976500Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-06-26T08:32:43.1979404Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-06-26T08:32:43.1981959Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-06-26T08:32:43.1984203Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-06-26T08:32:43.1986430Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-06-26T08:32:43.1988811Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-06-26T08:32:43.1991119Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-06-26T08:32:43.1993236Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-06-26T08:32:43.1995412Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-06-26T08:32:43.1997645Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-06-26T08:32:43.1999873Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-06-26T08:32:43.2001953Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-06-26T08:32:43.2003994Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-06-26T08:32:43.2006111Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-06-26T08:32:43.2008392Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-06-26T08:32:43.2010371Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-06-26T08:32:43.2012400Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-06-26T08:32:43.2014558Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-06-26T08:32:43.2016483Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allgather_work_wait_gpu 2024-06-26T08:32:43.2018446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_allreduce_work_wait_gpu 2024-06-26T08:32:43.2020373Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_consecutive_comm_work_wait_gpu 2024-06-26T08:32:43.2022241Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_base_k_float8_errors 2024-06-26T08:32:43.2024247Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CompilerTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-06-26T08:32:43.2026430Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-06-26T08:32:43.2028741Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-06-26T08:32:43.2031183Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-06-26T08:32:43.2033289Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-06-26T08:32:43.2034988Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-06-26T08:32:43.2036867Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-06-26T08:32:43.2038671Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-06-26T08:32:43.2040431Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-06-26T08:32:43.2042271Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1 2024-06-26T08:32:43.2044225Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-06-26T08:32:43.2046061Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-06-26T08:32:43.2048185Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-06-26T08:32:43.2050437Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-06-26T08:32:43.2052592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_False_include_collectives_True 2024-06-26T08:32:43.2054755Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_timing_enabled_True_include_collectives_True 2024-06-26T08:32:43.2056943Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-06-26T08:32:43.2058173Z 2024-06-26T08:32:44.5331105Z 2024-06-26T08:32:44.5331859Z real 36m0.950s 2024-06-26T08:32:44.5332535Z user 65m46.942s 2024-06-26T08:32:44.5333087Z sys 76m51.555s 2024-06-26T08:32:44.5333685Z + assert_git_not_dirty 2024-06-26T08:32:44.5334957Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-06-26T08:32:44.5335914Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-06-26T08:32:44.5336804Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-06-26T08:32:44.5337527Z + [[ 1 == 1 ]] 2024-06-26T08:32:44.5338187Z + echo 'Testing distributed C++ tests' 2024-06-26T08:32:44.5338950Z Testing distributed C++ tests 2024-06-26T08:32:44.5345834Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:32:44.5358772Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:32:44.5378863Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:32:44.5380609Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:32:44.5382002Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-06-26T08:32:44.6453396Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T08:32:44.6455195Z import pkg_resources 2024-06-26T08:32:48.0613701Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-06-26T08:32:48.0616723Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-06-26T08:32:48.0685325Z Found test times from artifacts 2024-06-26T08:32:48.0971171Z Found test times from artifacts 2024-06-26T08:32:48.0979752Z Running all tests 2024-06-26T08:32:48.0984508Z Running parallel tests on 8 processes 2024-06-26T08:32:48.0985451Z Name: tests to run (est. time: 0.0min) 2024-06-26T08:32:48.0986209Z Serial tests (1): 2024-06-26T08:32:48.0986827Z cpp/FileStoreTest 1/1 2024-06-26T08:32:48.0987464Z Parallel tests (0): 2024-06-26T08:32:48.0988083Z Name: excluded (est. time: 0.0min) 2024-06-26T08:32:48.0988797Z Serial tests (0): 2024-06-26T08:32:48.0989364Z Parallel tests (0): 2024-06-26T08:32:48.0990511Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T08:32:48.1694629Z Running cpp/FileStoreTest 1/1 ... [2024-06-26 08:32:48.168483] 2024-06-26T08:32:48.1701822Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-f9ba3626904eb75f.xml', '-x', '--reruns=2'] ... [2024-06-26 08:32:48.169400] 2024-06-26T08:32:50.9681714Z 2024-06-26T08:32:50.9684303Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_d1330ef05608fd61_.log 2024-06-26T08:32:50.9685968Z 2024-06-26T08:32:52.8348566Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-06-26T08:32:52.9344668Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T08:32:52.9346442Z import pkg_resources 2024-06-26T08:32:56.3989965Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-06-26T08:32:56.3992978Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-06-26T08:32:56.4062422Z Found test times from artifacts 2024-06-26T08:32:56.4349759Z Found test times from artifacts 2024-06-26T08:32:56.4357606Z Running all tests 2024-06-26T08:32:56.4362791Z Running parallel tests on 8 processes 2024-06-26T08:32:56.4363719Z Name: tests to run (est. time: 0.0min) 2024-06-26T08:32:56.4364474Z Serial tests (1): 2024-06-26T08:32:56.4365098Z cpp/HashStoreTest 1/1 2024-06-26T08:32:56.4365732Z Parallel tests (0): 2024-06-26T08:32:56.4366339Z Name: excluded (est. time: 0.0min) 2024-06-26T08:32:56.4367021Z Serial tests (0): 2024-06-26T08:32:56.4368340Z Parallel tests (0): 2024-06-26T08:32:56.4369517Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T08:32:56.5067304Z Running cpp/HashStoreTest 1/1 ... [2024-06-26 08:32:56.505888] 2024-06-26T08:32:56.5073874Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e9d3a38e2c222504.xml', '-x', '--reruns=2'] ... [2024-06-26 08:32:56.506772] 2024-06-26T08:32:59.1556334Z 2024-06-26T08:32:59.1558163Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_806450c6ca425845_.log 2024-06-26T08:32:59.1559252Z 2024-06-26T08:33:01.0601928Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-06-26T08:33:01.1636082Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T08:33:01.1637859Z import pkg_resources 2024-06-26T08:33:04.6135603Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-06-26T08:33:04.6138580Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-06-26T08:33:04.6206493Z Found test times from artifacts 2024-06-26T08:33:04.6497708Z Found test times from artifacts 2024-06-26T08:33:04.6506997Z Running all tests 2024-06-26T08:33:04.6508132Z Running parallel tests on 8 processes 2024-06-26T08:33:04.6508984Z Name: tests to run (est. time: 0.0min) 2024-06-26T08:33:04.6509714Z Serial tests (1): 2024-06-26T08:33:04.6510326Z cpp/TCPStoreTest 1/1 2024-06-26T08:33:04.6510966Z Parallel tests (0): 2024-06-26T08:33:04.6511569Z Name: excluded (est. time: 0.0min) 2024-06-26T08:33:04.6512242Z Serial tests (0): 2024-06-26T08:33:04.6512794Z Parallel tests (0): 2024-06-26T08:33:04.6513962Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T08:33:04.7264053Z Running cpp/TCPStoreTest 1/1 ... [2024-06-26 08:33:04.725814] 2024-06-26T08:33:04.7272992Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-54a2904fbb4d73fc.xml', '-x', '--reruns=2'] ... [2024-06-26 08:33:04.726615] 2024-06-26T08:33:17.1947111Z 2024-06-26T08:33:17.1949618Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_fee41b6b4eeaf06c_.log 2024-06-26T08:33:17.1951277Z 2024-06-26T08:33:18.6016246Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-06-26T08:33:18.6017167Z + [[ 1 == 1 ]] 2024-06-26T08:33:18.6017719Z + test_rpc 2024-06-26T08:33:18.6018406Z + echo 'Testing RPC C++ tests' 2024-06-26T08:33:18.6019110Z Testing RPC C++ tests 2024-06-26T08:33:18.6025659Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:33:18.6048702Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:33:18.6069862Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-06-26T08:33:18.6071479Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-06-26T08:33:18.7095252Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-06-26T08:33:18.7097271Z import pkg_resources 2024-06-26T08:33:22.1435565Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-06-26T08:33:22.1438552Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-06-26T08:33:22.1506871Z Found test times from artifacts 2024-06-26T08:33:22.1793589Z Found test times from artifacts 2024-06-26T08:33:22.1800619Z Running all tests 2024-06-26T08:33:22.1802612Z Running parallel tests on 8 processes 2024-06-26T08:33:22.1803442Z Name: tests to run (est. time: 0.0min) 2024-06-26T08:33:22.1804178Z Serial tests (1): 2024-06-26T08:33:22.1804749Z cpp/test_cpp_rpc 1/1 2024-06-26T08:33:22.1805370Z Parallel tests (0): 2024-06-26T08:33:22.1805991Z Name: excluded (est. time: 0.0min) 2024-06-26T08:33:22.1806660Z Serial tests (0): 2024-06-26T08:33:22.1807196Z Parallel tests (0): 2024-06-26T08:33:22.1808480Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-06-26T08:33:22.2507488Z Running cpp/test_cpp_rpc 1/1 ... [2024-06-26 08:33:22.249918] 2024-06-26T08:33:22.2514553Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-a00483cfb525f067.xml', '-x', '--reruns=2'] ... [2024-06-26 08:33:22.250734] 2024-06-26T08:33:28.1086714Z 2024-06-26T08:33:28.1089534Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_9deaa006b94349ac_.log 2024-06-26T08:33:28.1091069Z 2024-06-26T08:33:29.4427931Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-06-26T08:33:29.4430084Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-06-26T08:33:29.4432594Z docker exec -t "bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-06-26T08:33:29.4486094Z shell: /usr/bin/bash -e {0} 2024-06-26T08:33:29.4486766Z env: 2024-06-26T08:33:29.4487285Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:29.4488186Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:29.4489616Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:29.4490931Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:29.4491630Z AWS_REGION: us-east-1 2024-06-26T08:33:29.4492401Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:29.4493407Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:29.4507862Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:29.4508934Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:29.4510078Z ##[endgroup] 2024-06-26T08:33:29.5142764Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:33:29.5396122Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:33:29.5397050Z cat test/**/*_toprint.log || true 2024-06-26T08:33:29.5453897Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:29.5454790Z env: 2024-06-26T08:33:29.5455290Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:29.5456031Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:29.5457395Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:29.5458641Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:29.5459306Z AWS_REGION: us-east-1 2024-06-26T08:33:29.5460055Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:29.5460986Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:29.5474890Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:29.5475912Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:29.5476996Z ##[endgroup] 2024-06-26T08:33:29.5674444Z cat: 'test/**/*_toprint.log': No such file or directory 2024-06-26T08:33:29.5736607Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:33:29.5737464Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:33:29.5789464Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:33:29.5790381Z env: 2024-06-26T08:33:29.5790887Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:29.5791650Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:29.5792973Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:29.5794225Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:29.5794893Z AWS_REGION: us-east-1 2024-06-26T08:33:29.5795644Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:29.5796564Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:29.5810308Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:29.5811323Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:29.5812450Z MONITOR_SCRIPT_PID: 2112692 2024-06-26T08:33:29.5813082Z ##[endgroup] 2024-06-26T08:33:29.6180258Z Prepare all required actions 2024-06-26T08:33:29.6181009Z Getting action download info 2024-06-26T08:33:30.1908206Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T08:33:31.2915231Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:33:31.2915773Z with: 2024-06-26T08:33:31.2915987Z use-gha: true 2024-06-26T08:33:31.2916334Z file-suffix: test-distributed-1-1-linux.rocm.gpu_26688598647 2024-06-26T08:33:31.2916740Z s3-bucket: gha-artifacts 2024-06-26T08:33:31.2917001Z env: 2024-06-26T08:33:31.2917217Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:31.2917531Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:31.2918085Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:31.2918619Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:31.2918905Z AWS_REGION: us-east-1 2024-06-26T08:33:31.2919261Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:31.2919667Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:31.2925237Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:31.2925671Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:31.2926134Z ##[endgroup] 2024-06-26T08:33:31.2985015Z ##[group]Run actions/upload-artifact@v3 2024-06-26T08:33:31.2985334Z with: 2024-06-26T08:33:31.2985735Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip 2024-06-26T08:33:31.2986230Z retention-days: 14 2024-06-26T08:33:31.2986475Z if-no-files-found: warn 2024-06-26T08:33:31.2986730Z path: test/**/*.json 2024-06-26T08:33:31.2986950Z env: 2024-06-26T08:33:31.2987142Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:31.2987438Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:31.2987974Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:31.2988478Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:31.2988760Z AWS_REGION: us-east-1 2024-06-26T08:33:31.2989048Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:31.2989420Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:31.2994989Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:31.2995410Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:31.2995861Z ##[endgroup] 2024-06-26T08:33:31.7689867Z With the provided path, there will be 9 files uploaded 2024-06-26T08:33:31.7693622Z Starting artifact upload 2024-06-26T08:33:31.7696461Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-06-26T08:33:31.7698827Z Artifact name is valid! 2024-06-26T08:33:32.1566968Z Container for artifact "test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip" successfully created. Starting upload of file(s) 2024-06-26T08:33:34.1248709Z Total size of all the files uploaded is 35842 bytes 2024-06-26T08:33:34.1249942Z File upload process has finished. Finalizing the artifact upload 2024-06-26T08:33:34.4177143Z Artifact has been finalized. All files have been successfully uploaded! 2024-06-26T08:33:34.4178045Z 2024-06-26T08:33:34.4178659Z The raw size of all the files that were specified for upload is 331433 bytes 2024-06-26T08:33:34.4180804Z The size of all the files that were uploaded is 35842 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-06-26T08:33:34.4182319Z 2024-06-26T08:33:34.4184111Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-06-26T08:33:34.4185874Z 2024-06-26T08:33:34.4187102Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip has been successfully uploaded! 2024-06-26T08:33:34.4333678Z ##[group]Run actions/upload-artifact@v3 2024-06-26T08:33:34.4334774Z with: 2024-06-26T08:33:34.4335798Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip 2024-06-26T08:33:34.4337039Z retention-days: 14 2024-06-26T08:33:34.4337641Z if-no-files-found: ignore 2024-06-26T08:33:34.4338307Z path: test/**/*.xml test/**/*.csv 2024-06-26T08:33:34.4339261Z env: 2024-06-26T08:33:34.4339745Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:33:34.4340481Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:33:34.4341841Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:33:34.4343103Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:33:34.4343776Z AWS_REGION: us-east-1 2024-06-26T08:33:34.4344523Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:33:34.4345443Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:33:34.4359157Z AWS_SESSION_TOKEN: *** 2024-06-26T08:33:34.4360180Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:33:34.4361282Z ##[endgroup] 2024-06-26T08:33:34.9274429Z With the provided path, there will be 217 files uploaded 2024-06-26T08:33:34.9275874Z Starting artifact upload 2024-06-26T08:33:34.9278451Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-06-26T08:33:34.9280821Z Artifact name is valid! 2024-06-26T08:33:35.3067348Z Container for artifact "test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip" successfully created. Starting upload of file(s) 2024-06-26T08:33:45.3120249Z Total file count: 217 ---- Processed file #49 (22.5%) 2024-06-26T08:33:55.3124650Z Total file count: 217 ---- Processed file #99 (45.6%) 2024-06-26T08:34:05.3124419Z Total file count: 217 ---- Processed file #149 (68.6%) 2024-06-26T08:34:15.3120527Z Total file count: 217 ---- Processed file #199 (91.7%) 2024-06-26T08:34:18.7479645Z Total size of all the files uploaded is 53876 bytes 2024-06-26T08:34:18.7480943Z File upload process has finished. Finalizing the artifact upload 2024-06-26T08:34:19.1098051Z Artifact has been finalized. All files have been successfully uploaded! 2024-06-26T08:34:19.1098946Z 2024-06-26T08:34:19.1099545Z The raw size of all the files that were specified for upload is 82511 bytes 2024-06-26T08:34:19.1101715Z The size of all the files that were uploaded is 53876 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-06-26T08:34:19.1103641Z 2024-06-26T08:34:19.1105481Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-06-26T08:34:19.1107267Z 2024-06-26T08:34:19.1108542Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip has been successfully uploaded! 2024-06-26T08:34:19.1251884Z ##[group]Run actions/upload-artifact@v3 2024-06-26T08:34:19.1252652Z with: 2024-06-26T08:34:19.1253548Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip 2024-06-26T08:34:19.1254674Z retention-days: 14 2024-06-26T08:34:19.1255255Z if-no-files-found: ignore 2024-06-26T08:34:19.1255905Z path: usage_log.txt test/**/*.log 2024-06-26T08:34:19.1256604Z env: 2024-06-26T08:34:19.1257074Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:19.1257789Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:19.1259109Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:19.1260352Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:19.1261014Z AWS_REGION: us-east-1 2024-06-26T08:34:19.1261733Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:19.1262619Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:19.1276538Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:19.1277549Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:19.1278912Z ##[endgroup] 2024-06-26T08:34:19.6507819Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-06-26T08:34:19.6511130Z The least common ancestor is /home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-06-26T08:34:19.6512908Z With the provided path, there will be 10 files uploaded 2024-06-26T08:34:19.6515044Z Starting artifact upload 2024-06-26T08:34:19.6517615Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-06-26T08:34:19.6519928Z Artifact name is valid! 2024-06-26T08:34:20.0337552Z Container for artifact "logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip" successfully created. Starting upload of file(s) 2024-06-26T08:34:22.2432980Z Total size of all the files uploaded is 56885 bytes 2024-06-26T08:34:22.2434294Z File upload process has finished. Finalizing the artifact upload 2024-06-26T08:34:22.5264261Z Artifact has been finalized. All files have been successfully uploaded! 2024-06-26T08:34:22.5265121Z 2024-06-26T08:34:22.5265759Z The raw size of all the files that were specified for upload is 876117 bytes 2024-06-26T08:34:22.5267968Z The size of all the files that were uploaded is 56885 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-06-26T08:34:22.5269514Z 2024-06-26T08:34:22.5271335Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-06-26T08:34:22.5273091Z 2024-06-26T08:34:22.5274439Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_26688598647.zip has been successfully uploaded! 2024-06-26T08:34:22.5411519Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:34:22.5412394Z # shellcheck disable=SC2156 2024-06-26T08:34:22.5413776Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:34:22.5466804Z shell: /usr/bin/bash -e {0} 2024-06-26T08:34:22.5467433Z env: 2024-06-26T08:34:22.5467923Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:22.5468666Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:22.5470006Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:22.5471251Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:22.5471918Z AWS_REGION: us-east-1 2024-06-26T08:34:22.5472668Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:22.5473586Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:22.5487270Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:22.5488418Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:22.5489511Z ##[endgroup] 2024-06-26T08:34:22.7412836Z Prepare all required actions 2024-06-26T08:34:22.7413634Z Getting action download info 2024-06-26T08:34:22.7471202Z ##[group]Run ./.github/actions/teardown-rocm 2024-06-26T08:34:22.7471952Z env: 2024-06-26T08:34:22.7472430Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:22.7473155Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:22.7474475Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:22.7475751Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:22.7476416Z AWS_REGION: us-east-1 2024-06-26T08:34:22.7477147Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:22.7478054Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:22.7491817Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:22.7492824Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:22.7493903Z ##[endgroup] 2024-06-26T08:34:22.7522365Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:34:22.7523693Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:34:22.7524725Z # shellcheck disable=SC2046 2024-06-26T08:34:22.7525531Z docker stop $(docker ps -q) || true 2024-06-26T08:34:22.7526375Z # Prune all stopped containers. 2024-06-26T08:34:22.7527147Z docker container prune -f 2024-06-26T08:34:22.7582971Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:22.7584109Z env: 2024-06-26T08:34:22.7584594Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:22.7585318Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:22.7586632Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:22.7587875Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:22.7588541Z AWS_REGION: us-east-1 2024-06-26T08:34:22.7589254Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:22.7590165Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:22.7603763Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:22.7604781Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:22.7605860Z ##[endgroup] 2024-06-26T08:34:23.2796088Z bd5a72183d56 2024-06-26T08:34:33.7181809Z Deleted Containers: 2024-06-26T08:34:33.7182878Z bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:33.7183712Z 2024-06-26T08:34:33.7184017Z Total reclaimed space: 8.261GB 2024-06-26T08:34:33.7249278Z Prepare all required actions 2024-06-26T08:34:33.7311928Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-06-26T08:34:33.7312724Z with: 2024-06-26T08:34:33.7313221Z diskspace-cutoff: 70 2024-06-26T08:34:33.7313777Z env: 2024-06-26T08:34:33.7314254Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:33.7314978Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:33.7316287Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:33.7317549Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:33.7318225Z AWS_REGION: us-east-1 2024-06-26T08:34:33.7318980Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:33.7319892Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:33.7333981Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:33.7335008Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:33.7336103Z ##[endgroup] 2024-06-26T08:34:33.7364715Z ##[group]Run set -ex 2024-06-26T08:34:33.7365348Z set -ex 2024-06-26T08:34:33.7365923Z diskspace_cutoff=70 2024-06-26T08:34:33.7366817Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-06-26T08:34:33.7368637Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-06-26T08:34:33.7370881Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-06-26T08:34:33.7373090Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-06-26T08:34:33.7374041Z  docker system prune -af 2024-06-26T08:34:33.7375298Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-06-26T08:34:33.7376759Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-06-26T08:34:33.7378296Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-06-26T08:34:33.7379619Z  echo "$msg" 2024-06-26T08:34:33.7380247Z  exit 1 2024-06-26T08:34:33.7380832Z  else 2024-06-26T08:34:33.7381537Z  difference=$((diskspace - diskspace_new)) 2024-06-26T08:34:33.7382563Z  echo "Diskspace saved: $difference percent" 2024-06-26T08:34:33.7383405Z  fi 2024-06-26T08:34:33.7383909Z fi 2024-06-26T08:34:33.7435827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:34:33.7436729Z env: 2024-06-26T08:34:33.7437225Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:34:33.7437957Z DOCKER_HOST: unix:///run/user/4001/docker.sock 2024-06-26T08:34:33.7439272Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-06-26T08:34:33.7440517Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T08:34:33.7441185Z AWS_REGION: us-east-1 2024-06-26T08:34:33.7441896Z AWS_ACCESS_KEY_ID: *** 2024-06-26T08:34:33.7443112Z AWS_SECRET_ACCESS_KEY: *** 2024-06-26T08:34:33.7457013Z AWS_SESSION_TOKEN: *** 2024-06-26T08:34:33.7458023Z CONTAINER_NAME: bd5a72183d56396f677ca0dd1752f5057101373124a19aae0f208331cb5d88cd 2024-06-26T08:34:33.7459092Z ##[endgroup] 2024-06-26T08:34:33.7534722Z + diskspace_cutoff=70 2024-06-26T08:34:33.7541301Z ++ docker info -f '{{.DockerRootDir}}' 2024-06-26T08:34:33.8252512Z + docker_root_dir=/media/4TB/docker-rootless 2024-06-26T08:34:33.8260996Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-06-26T08:34:33.8261901Z ++ sed -n 2p 2024-06-26T08:34:33.8263530Z ++ sed s/%// 2024-06-26T08:34:33.8265008Z ++ sed 's/ //' 2024-06-26T08:34:33.8291204Z + diskspace=24 2024-06-26T08:34:33.8294654Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-06-26T08:34:33.8296627Z + [[ 24 -ge 70 ]] 2024-06-26T08:34:33.8396753Z Post job cleanup. 2024-06-26T08:34:33.8472495Z Post job cleanup. 2024-06-26T08:34:33.9065436Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T08:34:33.9510172Z Post job cleanup. 2024-06-26T08:34:34.0893901Z Post job cleanup. 2024-06-26T08:34:34.0993079Z Post job cleanup. 2024-06-26T08:34:34.1086783Z Post job cleanup. 2024-06-26T08:34:34.2062632Z [command]/usr/bin/git version 2024-06-26T08:34:34.2107192Z git version 2.34.1 2024-06-26T08:34:34.2140189Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/794ec48e-99b9-49fc-9736-06d151b9f8cc' before making global git config changes 2024-06-26T08:34:34.2142465Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:34:34.2144502Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-06-26T08:34:34.2192829Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:34:34.2235467Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:34:34.2625807Z Entering 'android/libs/fbjni' 2024-06-26T08:34:34.2689694Z Entering 'third_party/FP16' 2024-06-26T08:34:34.2760567Z Entering 'third_party/FXdiv' 2024-06-26T08:34:34.2828094Z Entering 'third_party/NNPACK' 2024-06-26T08:34:34.2892839Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:34:34.2956938Z Entering 'third_party/XNNPACK' 2024-06-26T08:34:34.3042103Z Entering 'third_party/benchmark' 2024-06-26T08:34:34.3106565Z Entering 'third_party/cpp-httplib' 2024-06-26T08:34:34.3168837Z Entering 'third_party/cpuinfo' 2024-06-26T08:34:34.3235690Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:34:34.3300407Z Entering 'third_party/cutlass' 2024-06-26T08:34:34.3374999Z Entering 'third_party/eigen' 2024-06-26T08:34:34.3442484Z Entering 'third_party/fbgemm' 2024-06-26T08:34:34.3507343Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:34:34.3568918Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:34:34.3632951Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:34:34.3704797Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:34:34.3767040Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:34:34.3832622Z Entering 'third_party/flatbuffers' 2024-06-26T08:34:34.3901074Z Entering 'third_party/fmt' 2024-06-26T08:34:34.3963971Z Entering 'third_party/foxi' 2024-06-26T08:34:34.4026818Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:34:34.4091476Z Entering 'third_party/gloo' 2024-06-26T08:34:34.4153727Z Entering 'third_party/googletest' 2024-06-26T08:34:34.4215760Z Entering 'third_party/ideep' 2024-06-26T08:34:34.4281594Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:34:34.4356062Z Entering 'third_party/ittapi' 2024-06-26T08:34:34.4426395Z Entering 'third_party/kineto' 2024-06-26T08:34:34.4495175Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:34:34.4552328Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:34:34.4614953Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:34:34.4678040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:34:34.4738645Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:34:34.4799800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:34:34.4862343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:34:34.4923476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:34:34.4987045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:34:34.5047311Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:34:34.5112094Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:34:34.5173906Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:34:34.5241940Z Entering 'third_party/mimalloc' 2024-06-26T08:34:34.5311842Z Entering 'third_party/nccl/nccl' 2024-06-26T08:34:34.5381608Z Entering 'third_party/nlohmann' 2024-06-26T08:34:34.5449881Z Entering 'third_party/onnx' 2024-06-26T08:34:34.5543742Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:34:34.5608814Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:34:34.5680966Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:34:34.5748989Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:34:34.5815470Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:34:34.5873913Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:34:34.5935014Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:34:34.5996943Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:34:34.6056679Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:34:34.6118930Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:34:34.6176441Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:34:34.6239697Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:34:34.6306248Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:34:34.6396020Z Entering 'third_party/pocketfft' 2024-06-26T08:34:34.6459837Z Entering 'third_party/protobuf' 2024-06-26T08:34:34.6527033Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:34:34.6589779Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:34:34.6656033Z Entering 'third_party/psimd' 2024-06-26T08:34:34.6719390Z Entering 'third_party/pthreadpool' 2024-06-26T08:34:34.6783837Z Entering 'third_party/pybind11' 2024-06-26T08:34:34.6847014Z Entering 'third_party/python-peachpy' 2024-06-26T08:34:34.6911004Z Entering 'third_party/sleef' 2024-06-26T08:34:34.6980152Z Entering 'third_party/tensorpipe' 2024-06-26T08:34:34.7043117Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:34:34.7103972Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:34:34.7161723Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:34:34.7221432Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:34:34.7282072Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:34:34.7376366Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:34:34.7421393Z http.https://github.com/.extraheader 2024-06-26T08:34:34.7437303Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:34:34.7496892Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:34:34.7875490Z Entering 'android/libs/fbjni' 2024-06-26T08:34:34.7912452Z http.https://github.com/.extraheader 2024-06-26T08:34:34.7964879Z Entering 'third_party/FP16' 2024-06-26T08:34:34.8001845Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8060584Z Entering 'third_party/FXdiv' 2024-06-26T08:34:34.8096953Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8148861Z Entering 'third_party/NNPACK' 2024-06-26T08:34:34.8185147Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8233538Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:34:34.8267529Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8323646Z Entering 'third_party/XNNPACK' 2024-06-26T08:34:34.8359641Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8431859Z Entering 'third_party/benchmark' 2024-06-26T08:34:34.8469813Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8521559Z Entering 'third_party/cpp-httplib' 2024-06-26T08:34:34.8557151Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8606082Z Entering 'third_party/cpuinfo' 2024-06-26T08:34:34.8642461Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8694637Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:34:34.8735187Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8783890Z Entering 'third_party/cutlass' 2024-06-26T08:34:34.8819920Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8881998Z Entering 'third_party/eigen' 2024-06-26T08:34:34.8917227Z http.https://github.com/.extraheader 2024-06-26T08:34:34.8969567Z Entering 'third_party/fbgemm' 2024-06-26T08:34:34.9005812Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9056846Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:34:34.9092372Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9139561Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:34:34.9176054Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9223282Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:34:34.9257047Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9311206Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:34:34.9345321Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9396755Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:34:34.9429597Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9481863Z Entering 'third_party/flatbuffers' 2024-06-26T08:34:34.9517238Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9568194Z Entering 'third_party/fmt' 2024-06-26T08:34:34.9604480Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9654881Z Entering 'third_party/foxi' 2024-06-26T08:34:34.9688298Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9737494Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:34:34.9772320Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9821080Z Entering 'third_party/gloo' 2024-06-26T08:34:34.9856266Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9905749Z Entering 'third_party/googletest' 2024-06-26T08:34:34.9941138Z http.https://github.com/.extraheader 2024-06-26T08:34:34.9990162Z Entering 'third_party/ideep' 2024-06-26T08:34:35.0030481Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0079409Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:34:35.0113498Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0175950Z Entering 'third_party/ittapi' 2024-06-26T08:34:35.0207386Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0257191Z Entering 'third_party/kineto' 2024-06-26T08:34:35.0295769Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0340994Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:34:35.0375844Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0419930Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:34:35.0455958Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0501736Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:34:35.0533926Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0578737Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:34:35.0611778Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0659131Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:34:35.0691993Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0736003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:34:35.0768179Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0818590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:34:35.0852464Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0899473Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:34:35.0932908Z http.https://github.com/.extraheader 2024-06-26T08:34:35.0978996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:34:35.1014966Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1059357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:34:35.1094523Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1141556Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:34:35.1175846Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1222350Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:34:35.1255007Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1305207Z Entering 'third_party/mimalloc' 2024-06-26T08:34:35.1339566Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1395484Z Entering 'third_party/nccl/nccl' 2024-06-26T08:34:35.1423926Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1475645Z Entering 'third_party/nlohmann' 2024-06-26T08:34:35.1506148Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1560328Z Entering 'third_party/onnx' 2024-06-26T08:34:35.1595548Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1655891Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:34:35.1694680Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1742842Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:34:35.1778208Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1836713Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:34:35.1876062Z http.https://github.com/.extraheader 2024-06-26T08:34:35.1922106Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:34:35.1957349Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2002832Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:34:35.2033965Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2078924Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:34:35.2113203Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2160285Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:34:35.2191801Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2240049Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:34:35.2272299Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2318367Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:34:35.2350086Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2401604Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:34:35.2436097Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2482108Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:34:35.2519268Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2569343Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:34:35.2602704Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2657014Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:34:35.2687071Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2764233Z Entering 'third_party/pocketfft' 2024-06-26T08:34:35.2800623Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2849322Z Entering 'third_party/protobuf' 2024-06-26T08:34:35.2884067Z http.https://github.com/.extraheader 2024-06-26T08:34:35.2937697Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:34:35.2969531Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3023753Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:34:35.3056874Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3108069Z Entering 'third_party/psimd' 2024-06-26T08:34:35.3144602Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3192881Z Entering 'third_party/pthreadpool' 2024-06-26T08:34:35.3227808Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3274569Z Entering 'third_party/pybind11' 2024-06-26T08:34:35.3309316Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3361182Z Entering 'third_party/python-peachpy' 2024-06-26T08:34:35.3403369Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3444208Z Entering 'third_party/sleef' 2024-06-26T08:34:35.3483130Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3528758Z Entering 'third_party/tensorpipe' 2024-06-26T08:34:35.3566676Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3615203Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:34:35.3648483Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3700872Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:34:35.3735285Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3782022Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:34:35.3815645Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3865819Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:34:35.3899043Z http.https://github.com/.extraheader 2024-06-26T08:34:35.3945402Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:34:35.3979263Z http.https://github.com/.extraheader 2024-06-26T08:34:35.4468480Z Cleaning up orphan processes