2024-08-08T20:36:53.8601993Z Current runner version: '2.318.0' 2024-08-08T20:36:53.8608001Z Runner name: 'pytorch-rocm-hw-22' 2024-08-08T20:36:53.8608745Z Runner group name: 'linux.rocm.gpu.group' 2024-08-08T20:36:53.8609631Z Machine name: 'pytorch-rocm-hw-22' 2024-08-08T20:36:53.8612277Z ##[group]GITHUB_TOKEN Permissions 2024-08-08T20:36:53.8614106Z Contents: read 2024-08-08T20:36:53.8614697Z Metadata: read 2024-08-08T20:36:53.8615239Z ##[endgroup] 2024-08-08T20:36:53.8617796Z Secret source: Actions 2024-08-08T20:36:53.8618661Z Prepare workflow directory 2024-08-08T20:36:54.1552651Z Prepare all required actions 2024-08-08T20:36:54.1589439Z Getting action download info 2024-08-08T20:36:54.4449870Z Download action repository 'pytorch/pytorch@main' (SHA:7bd0732cbdca4ae572a8b4de2017cdfdff158d55) 2024-08-08T20:37:02.2069612Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-08-08T20:37:02.8092204Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-08-08T20:37:03.4981392Z Download action repository 'pytorch/test-infra@main' (SHA:0b5c20199b94c3b12545de0d354b54a297b0e061) 2024-08-08T20:37:04.6263052Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-08T20:37:05.2875153Z Getting action download info 2024-08-08T20:37:05.4639529Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-08T20:37:06.1832990Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/132710 (b9d86fa89636e301796d4201f36d86c73f6e49bc) 2024-08-08T20:37:06.1834670Z ##[group] Inputs 2024-08-08T20:37:06.1834944Z build-environment: linux-focal-rocm6.1-py3.8 2024-08-08T20:37:06.1835750Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-08-08T20:37:06.1836769Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:37:06.1837279Z sync-tag: 2024-08-08T20:37:06.1837924Z timeout-minutes: 300 2024-08-08T20:37:06.1838537Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-08T20:37:06.1839204Z ##[endgroup] 2024-08-08T20:37:06.1839523Z Complete job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:37:06.2885971Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-08T20:37:06.2886576Z with: 2024-08-08T20:37:06.2886746Z no-sudo: true 2024-08-08T20:37:06.2886929Z submodules: recursive 2024-08-08T20:37:06.2887120Z fetch-depth: 0 2024-08-08T20:37:06.2887427Z env: 2024-08-08T20:37:06.2887614Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:37:06.2887797Z ##[endgroup] 2024-08-08T20:37:06.2957061Z ##[group]Run retry () { 2024-08-08T20:37:06.2957323Z retry () { 2024-08-08T20:37:06.2957600Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-08T20:37:06.2957888Z } 2024-08-08T20:37:06.2958063Z echo "${GITHUB_WORKSPACE}" 2024-08-08T20:37:06.2958313Z if [ -z "${NO_SUDO}" ]; then 2024-08-08T20:37:06.2958591Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-08T20:37:06.2958835Z else 2024-08-08T20:37:06.2959021Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-08T20:37:06.2959282Z fi 2024-08-08T20:37:06.2959450Z mkdir "${GITHUB_WORKSPACE}" 2024-08-08T20:37:06.2978591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:37:06.2978900Z env: 2024-08-08T20:37:06.2979069Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:37:06.2979265Z NO_SUDO: true 2024-08-08T20:37:06.2979748Z ##[endgroup] 2024-08-08T20:37:06.3074232Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T20:37:09.9214106Z ##[group]Run malfet/checkout@silent-checkout 2024-08-08T20:37:09.9214407Z with: 2024-08-08T20:37:09.9214604Z ref: b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:37:09.9214851Z fetch-depth: 0 2024-08-08T20:37:09.9215055Z submodules: recursive 2024-08-08T20:37:09.9215260Z quiet-checkout: true 2024-08-08T20:37:09.9215455Z repository: pytorch/pytorch 2024-08-08T20:37:09.9215759Z token: *** 2024-08-08T20:37:09.9215940Z ssh-strict: true 2024-08-08T20:37:09.9216129Z persist-credentials: true 2024-08-08T20:37:09.9216355Z clean: true 2024-08-08T20:37:09.9216550Z sparse-checkout-cone-mode: true 2024-08-08T20:37:09.9216762Z lfs: false 2024-08-08T20:37:09.9216938Z set-safe-directory: true 2024-08-08T20:37:09.9217146Z env: 2024-08-08T20:37:09.9217314Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:37:09.9217508Z ##[endgroup] 2024-08-08T20:37:10.0059720Z Syncing repository: pytorch/pytorch 2024-08-08T20:37:10.0062280Z ##[group]Getting Git version info 2024-08-08T20:37:10.0063197Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-08-08T20:37:10.0064384Z [command]/usr/bin/git version 2024-08-08T20:37:10.0064891Z git version 2.34.1 2024-08-08T20:37:10.0066513Z ##[endgroup] 2024-08-08T20:37:10.0073640Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/f047ea1e-49f2-4ea1-a2da-3f2be8cbf3e0' before making global git config changes 2024-08-08T20:37:10.0075357Z Adding repository directory to the temporary git global config as a safe directory 2024-08-08T20:37:10.0076749Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T20:37:10.0078808Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-08-08T20:37:10.0080181Z ##[group]Initializing the repository 2024-08-08T20:37:10.0081169Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T20:37:10.0082545Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-08T20:37:10.0083635Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-08T20:37:10.0084616Z hint: of your new repositories, which will suppress this warning, call: 2024-08-08T20:37:10.0085470Z hint: 2024-08-08T20:37:10.0086000Z hint: git config --global init.defaultBranch 2024-08-08T20:37:10.0086627Z hint: 2024-08-08T20:37:10.0087195Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-08T20:37:10.0088663Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-08T20:37:10.0089480Z hint: 2024-08-08T20:37:10.0089874Z hint: git branch -m 2024-08-08T20:37:10.0090799Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-08T20:37:10.0094454Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-08T20:37:10.0138082Z ##[endgroup] 2024-08-08T20:37:10.0138427Z ##[group]Disabling automatic garbage collection 2024-08-08T20:37:10.0140038Z [command]/usr/bin/git config --local gc.auto 0 2024-08-08T20:37:10.0161119Z ##[endgroup] 2024-08-08T20:37:10.0161439Z ##[group]Setting up auth 2024-08-08T20:37:10.0166720Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-08T20:37:10.0193812Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-08T20:37:10.0419380Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-08T20:37:10.0439692Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-08T20:37:10.0673700Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-08T20:37:10.0703289Z ##[endgroup] 2024-08-08T20:37:10.0703729Z ##[group]Fetching the repository 2024-08-08T20:37:10.0708191Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-08T20:37:12.7272213Z remote: Enumerating objects: 1008995 2024-08-08T20:37:12.7273283Z remote: Enumerating objects: 1011786, done. 2024-08-08T20:37:12.7274112Z remote: Counting objects: 0% (1/2791) 2024-08-08T20:37:12.7274858Z remote: Counting objects: 1% (28/2791) 2024-08-08T20:37:12.7275481Z remote: Counting objects: 2% (56/2791) 2024-08-08T20:37:12.7276087Z remote: Counting objects: 3% (84/2791) 2024-08-08T20:37:12.7276692Z remote: Counting objects: 4% (112/2791) 2024-08-08T20:37:12.7277317Z remote: Counting objects: 5% (140/2791) 2024-08-08T20:37:12.7277959Z remote: Counting objects: 6% (168/2791) 2024-08-08T20:37:12.7278550Z remote: Counting objects: 7% (196/2791) 2024-08-08T20:37:12.7279143Z remote: Counting objects: 8% (224/2791) 2024-08-08T20:37:12.7279740Z remote: Counting objects: 9% (252/2791) 2024-08-08T20:37:12.7280324Z remote: Counting objects: 10% (280/2791) 2024-08-08T20:37:12.7280914Z remote: Counting objects: 11% (308/2791) 2024-08-08T20:37:12.7281300Z remote: Counting objects: 12% (335/2791) 2024-08-08T20:37:12.7281602Z remote: Counting objects: 13% (363/2791) 2024-08-08T20:37:12.7281922Z remote: Counting objects: 14% (391/2791) 2024-08-08T20:37:12.7282262Z remote: Counting objects: 15% (419/2791) 2024-08-08T20:37:12.7282565Z remote: Counting objects: 16% (447/2791) 2024-08-08T20:37:12.7282881Z remote: Counting objects: 17% (475/2791) 2024-08-08T20:37:12.7283194Z remote: Counting objects: 18% (503/2791) 2024-08-08T20:37:12.7283507Z remote: Counting objects: 19% (531/2791) 2024-08-08T20:37:12.7283831Z remote: Counting objects: 20% (559/2791) 2024-08-08T20:37:12.7284142Z remote: Counting objects: 21% (587/2791) 2024-08-08T20:37:12.7284449Z remote: Counting objects: 22% (615/2791) 2024-08-08T20:37:12.7284825Z remote: Counting objects: 23% (642/2791) 2024-08-08T20:37:12.7285133Z remote: Counting objects: 24% (670/2791) 2024-08-08T20:37:12.7285438Z remote: Counting objects: 25% (698/2791) 2024-08-08T20:37:12.7285742Z remote: Counting objects: 26% (726/2791) 2024-08-08T20:37:12.7286440Z remote: Counting objects: 27% (754/2791) 2024-08-08T20:37:12.7286775Z remote: Counting objects: 28% (782/2791) 2024-08-08T20:37:12.7287082Z remote: Counting objects: 29% (810/2791) 2024-08-08T20:37:12.7287392Z remote: Counting objects: 30% (838/2791) 2024-08-08T20:37:12.7287727Z remote: Counting objects: 31% (866/2791) 2024-08-08T20:37:12.7288037Z remote: Counting objects: 32% (894/2791) 2024-08-08T20:37:12.7288349Z remote: Counting objects: 33% (922/2791) 2024-08-08T20:37:12.7288653Z remote: Counting objects: 34% (949/2791) 2024-08-08T20:37:12.7288962Z remote: Counting objects: 35% (977/2791) 2024-08-08T20:37:12.7289297Z remote: Counting objects: 36% (1005/2791) 2024-08-08T20:37:12.7289619Z remote: Counting objects: 37% (1033/2791) 2024-08-08T20:37:12.7289949Z remote: Counting objects: 38% (1061/2791) 2024-08-08T20:37:12.7290307Z remote: Counting objects: 39% (1089/2791) 2024-08-08T20:37:12.7290649Z remote: Counting objects: 40% (1117/2791) 2024-08-08T20:37:12.7290961Z remote: Counting objects: 41% (1145/2791) 2024-08-08T20:37:12.7291270Z remote: Counting objects: 42% (1173/2791) 2024-08-08T20:37:12.7291586Z remote: Counting objects: 43% (1201/2791) 2024-08-08T20:37:12.7291911Z remote: Counting objects: 44% (1229/2791) 2024-08-08T20:37:12.7292399Z remote: Counting objects: 45% (1256/2791) 2024-08-08T20:37:12.7292710Z remote: Counting objects: 46% (1284/2791) 2024-08-08T20:37:12.7293012Z remote: Counting objects: 47% (1312/2791) 2024-08-08T20:37:12.7293327Z remote: Counting objects: 48% (1340/2791) 2024-08-08T20:37:12.7293641Z remote: Counting objects: 49% (1368/2791) 2024-08-08T20:37:12.7293945Z remote: Counting objects: 50% (1396/2791) 2024-08-08T20:37:12.7294258Z remote: Counting objects: 51% (1424/2791) 2024-08-08T20:37:12.7294566Z remote: Counting objects: 52% (1452/2791) 2024-08-08T20:37:12.7294872Z remote: Counting objects: 53% (1480/2791) 2024-08-08T20:37:12.7295185Z remote: Counting objects: 54% (1508/2791) 2024-08-08T20:37:12.7295494Z remote: Counting objects: 55% (1536/2791) 2024-08-08T20:37:12.7295800Z remote: Counting objects: 56% (1563/2791) 2024-08-08T20:37:12.7296115Z remote: Counting objects: 57% (1591/2791) 2024-08-08T20:37:12.7296425Z remote: Counting objects: 58% (1619/2791) 2024-08-08T20:37:12.7296736Z remote: Counting objects: 59% (1647/2791) 2024-08-08T20:37:12.7297042Z remote: Counting objects: 60% (1675/2791) 2024-08-08T20:37:12.7297353Z remote: Counting objects: 61% (1703/2791) 2024-08-08T20:37:12.7297669Z remote: Counting objects: 62% (1731/2791) 2024-08-08T20:37:12.7298035Z remote: Counting objects: 63% (1759/2791) 2024-08-08T20:37:12.7298364Z remote: Counting objects: 64% (1787/2791) 2024-08-08T20:37:12.7298684Z remote: Counting objects: 65% (1815/2791) 2024-08-08T20:37:12.7299007Z remote: Counting objects: 66% (1843/2791) 2024-08-08T20:37:12.7299311Z remote: Counting objects: 67% (1870/2791) 2024-08-08T20:37:12.7299783Z remote: Counting objects: 68% (1898/2791) 2024-08-08T20:37:12.7300111Z remote: Counting objects: 69% (1926/2791) 2024-08-08T20:37:12.7300429Z remote: Counting objects: 70% (1954/2791) 2024-08-08T20:37:12.7300757Z remote: Counting objects: 71% (1982/2791) 2024-08-08T20:37:12.7301071Z remote: Counting objects: 72% (2010/2791) 2024-08-08T20:37:12.7301375Z remote: Counting objects: 73% (2038/2791) 2024-08-08T20:37:12.7301714Z remote: Counting objects: 74% (2066/2791) 2024-08-08T20:37:12.7302044Z remote: Counting objects: 75% (2094/2791) 2024-08-08T20:37:12.7302378Z remote: Counting objects: 76% (2122/2791) 2024-08-08T20:37:12.7302696Z remote: Counting objects: 77% (2150/2791) 2024-08-08T20:37:12.7303145Z remote: Counting objects: 78% (2177/2791) 2024-08-08T20:37:12.7303466Z remote: Counting objects: 79% (2205/2791) 2024-08-08T20:37:12.7303786Z remote: Counting objects: 80% (2233/2791) 2024-08-08T20:37:12.7304091Z remote: Counting objects: 81% (2261/2791) 2024-08-08T20:37:12.7304407Z remote: Counting objects: 82% (2289/2791) 2024-08-08T20:37:12.7304725Z remote: Counting objects: 83% (2317/2791) 2024-08-08T20:37:12.7305035Z remote: Counting objects: 84% (2345/2791) 2024-08-08T20:37:12.7305346Z remote: Counting objects: 85% (2373/2791) 2024-08-08T20:37:12.7305666Z remote: Counting objects: 86% (2401/2791) 2024-08-08T20:37:12.7305974Z remote: Counting objects: 87% (2429/2791) 2024-08-08T20:37:12.7306283Z remote: Counting objects: 88% (2457/2791) 2024-08-08T20:37:12.7306608Z remote: Counting objects: 89% (2484/2791) 2024-08-08T20:37:12.7306919Z remote: Counting objects: 90% (2512/2791) 2024-08-08T20:37:12.7307226Z remote: Counting objects: 91% (2540/2791) 2024-08-08T20:37:12.7307539Z remote: Counting objects: 92% (2568/2791) 2024-08-08T20:37:12.7307850Z remote: Counting objects: 93% (2596/2791) 2024-08-08T20:37:12.7308157Z remote: Counting objects: 94% (2624/2791) 2024-08-08T20:37:12.7308469Z remote: Counting objects: 95% (2652/2791) 2024-08-08T20:37:12.7308914Z remote: Counting objects: 96% (2680/2791) 2024-08-08T20:37:12.7309218Z remote: Counting objects: 97% (2708/2791) 2024-08-08T20:37:12.7309529Z remote: Counting objects: 98% (2736/2791) 2024-08-08T20:37:12.7309839Z remote: Counting objects: 99% (2764/2791) 2024-08-08T20:37:12.7310142Z remote: Counting objects: 100% (2791/2791) 2024-08-08T20:37:12.7310488Z remote: Counting objects: 100% (2791/2791), done. 2024-08-08T20:37:12.7895857Z remote: Compressing objects: 0% (1/1478) 2024-08-08T20:37:12.8525821Z remote: Compressing objects: 1% (15/1478) 2024-08-08T20:37:12.8526749Z remote: Compressing objects: 2% (30/1478) 2024-08-08T20:37:12.8527415Z remote: Compressing objects: 3% (45/1478) 2024-08-08T20:37:12.9153612Z remote: Compressing objects: 4% (60/1478) 2024-08-08T20:37:13.3185646Z remote: Compressing objects: 5% (74/1478) 2024-08-08T20:37:13.3186623Z remote: Compressing objects: 6% (89/1478) 2024-08-08T20:37:13.3187300Z remote: Compressing objects: 7% (104/1478) 2024-08-08T20:37:13.3815454Z remote: Compressing objects: 8% (119/1478) 2024-08-08T20:37:13.3816306Z remote: Compressing objects: 9% (134/1478) 2024-08-08T20:37:13.4443843Z remote: Compressing objects: 10% (148/1478) 2024-08-08T20:37:13.4444970Z remote: Compressing objects: 11% (163/1478) 2024-08-08T20:37:13.5073876Z remote: Compressing objects: 12% (178/1478) 2024-08-08T20:37:13.5074702Z remote: Compressing objects: 13% (193/1478) 2024-08-08T20:37:13.5075398Z remote: Compressing objects: 14% (207/1478) 2024-08-08T20:37:13.5704691Z remote: Compressing objects: 15% (222/1478) 2024-08-08T20:37:13.5705666Z remote: Compressing objects: 16% (237/1478) 2024-08-08T20:37:13.5706326Z remote: Compressing objects: 17% (252/1478) 2024-08-08T20:37:13.5706980Z remote: Compressing objects: 18% (267/1478) 2024-08-08T20:37:13.5707664Z remote: Compressing objects: 19% (281/1478) 2024-08-08T20:37:13.5708298Z remote: Compressing objects: 20% (296/1478) 2024-08-08T20:37:13.5708915Z remote: Compressing objects: 21% (311/1478) 2024-08-08T20:37:13.5709533Z remote: Compressing objects: 22% (326/1478) 2024-08-08T20:37:13.5710166Z remote: Compressing objects: 23% (340/1478) 2024-08-08T20:37:13.6088313Z remote: Compressing objects: 24% (355/1478) 2024-08-08T20:37:13.6089179Z remote: Compressing objects: 25% (370/1478) 2024-08-08T20:37:13.6089876Z remote: Compressing objects: 26% (385/1478) 2024-08-08T20:37:13.6091068Z remote: Compressing objects: 27% (400/1478) 2024-08-08T20:37:13.6091741Z remote: Compressing objects: 28% (414/1478) 2024-08-08T20:37:13.6092403Z remote: Compressing objects: 29% (429/1478) 2024-08-08T20:37:13.6093025Z remote: Compressing objects: 30% (444/1478) 2024-08-08T20:37:13.6093663Z remote: Compressing objects: 31% (459/1478) 2024-08-08T20:37:13.6094325Z remote: Compressing objects: 32% (473/1478) 2024-08-08T20:37:13.6094966Z remote: Compressing objects: 33% (488/1478) 2024-08-08T20:37:13.6095601Z remote: Compressing objects: 34% (503/1478) 2024-08-08T20:37:13.6096240Z remote: Compressing objects: 35% (518/1478) 2024-08-08T20:37:13.6096862Z remote: Compressing objects: 36% (533/1478) 2024-08-08T20:37:13.6097484Z remote: Compressing objects: 37% (547/1478) 2024-08-08T20:37:13.6098108Z remote: Compressing objects: 38% (562/1478) 2024-08-08T20:37:13.6098732Z remote: Compressing objects: 39% (577/1478) 2024-08-08T20:37:13.6099364Z remote: Compressing objects: 40% (592/1478) 2024-08-08T20:37:13.6099986Z remote: Compressing objects: 41% (606/1478) 2024-08-08T20:37:13.6100607Z remote: Compressing objects: 42% (621/1478) 2024-08-08T20:37:13.6101238Z remote: Compressing objects: 43% (636/1478) 2024-08-08T20:37:13.6220169Z remote: Compressing objects: 44% (651/1478) 2024-08-08T20:37:13.6220997Z remote: Compressing objects: 45% (666/1478) 2024-08-08T20:37:13.6221699Z remote: Compressing objects: 46% (680/1478) 2024-08-08T20:37:13.6222362Z remote: Compressing objects: 47% (695/1478) 2024-08-08T20:37:13.6223001Z remote: Compressing objects: 48% (710/1478) 2024-08-08T20:37:13.6223623Z remote: Compressing objects: 49% (725/1478) 2024-08-08T20:37:13.6224256Z remote: Compressing objects: 50% (739/1478) 2024-08-08T20:37:13.6224885Z remote: Compressing objects: 51% (754/1478) 2024-08-08T20:37:13.6225533Z remote: Compressing objects: 52% (769/1478) 2024-08-08T20:37:13.6226159Z remote: Compressing objects: 53% (784/1478) 2024-08-08T20:37:13.6226786Z remote: Compressing objects: 54% (799/1478) 2024-08-08T20:37:13.6227398Z remote: Compressing objects: 55% (813/1478) 2024-08-08T20:37:13.6228038Z remote: Compressing objects: 56% (828/1478) 2024-08-08T20:37:13.6228695Z remote: Compressing objects: 57% (843/1478) 2024-08-08T20:37:13.6229311Z remote: Compressing objects: 58% (858/1478) 2024-08-08T20:37:13.6229933Z remote: Compressing objects: 59% (873/1478) 2024-08-08T20:37:13.6230557Z remote: Compressing objects: 60% (887/1478) 2024-08-08T20:37:13.6231171Z remote: Compressing objects: 61% (902/1478) 2024-08-08T20:37:13.6231798Z remote: Compressing objects: 62% (917/1478) 2024-08-08T20:37:13.6232423Z remote: Compressing objects: 63% (932/1478) 2024-08-08T20:37:13.6300116Z remote: Compressing objects: 64% (946/1478) 2024-08-08T20:37:13.6301163Z remote: Compressing objects: 65% (961/1478) 2024-08-08T20:37:13.6301923Z remote: Compressing objects: 66% (976/1478) 2024-08-08T20:37:13.6302601Z remote: Compressing objects: 67% (991/1478) 2024-08-08T20:37:13.6303246Z remote: Compressing objects: 68% (1006/1478) 2024-08-08T20:37:13.6303922Z remote: Compressing objects: 69% (1020/1478) 2024-08-08T20:37:13.6304562Z remote: Compressing objects: 70% (1035/1478) 2024-08-08T20:37:13.6305189Z remote: Compressing objects: 71% (1050/1478) 2024-08-08T20:37:13.6305818Z remote: Compressing objects: 72% (1065/1478) 2024-08-08T20:37:13.6306467Z remote: Compressing objects: 73% (1079/1478) 2024-08-08T20:37:13.6307101Z remote: Compressing objects: 74% (1094/1478) 2024-08-08T20:37:13.6307741Z remote: Compressing objects: 75% (1109/1478) 2024-08-08T20:37:13.6308383Z remote: Compressing objects: 76% (1124/1478) 2024-08-08T20:37:13.6309458Z remote: Compressing objects: 77% (1139/1478) 2024-08-08T20:37:13.6310118Z remote: Compressing objects: 78% (1153/1478) 2024-08-08T20:37:13.6310758Z remote: Compressing objects: 79% (1168/1478) 2024-08-08T20:37:13.6311382Z remote: Compressing objects: 80% (1183/1478) 2024-08-08T20:37:13.6312023Z remote: Compressing objects: 81% (1198/1478) 2024-08-08T20:37:13.6312655Z remote: Compressing objects: 82% (1212/1478) 2024-08-08T20:37:13.6313286Z remote: Compressing objects: 83% (1227/1478) 2024-08-08T20:37:13.6313918Z remote: Compressing objects: 84% (1242/1478) 2024-08-08T20:37:13.6332632Z remote: Compressing objects: 85% (1257/1478) 2024-08-08T20:37:13.6333546Z remote: Compressing objects: 86% (1272/1478) 2024-08-08T20:37:13.6334278Z remote: Compressing objects: 87% (1286/1478) 2024-08-08T20:37:13.6334931Z remote: Compressing objects: 88% (1301/1478) 2024-08-08T20:37:13.6335579Z remote: Compressing objects: 89% (1316/1478) 2024-08-08T20:37:13.6336213Z remote: Compressing objects: 90% (1331/1478) 2024-08-08T20:37:13.6336853Z remote: Compressing objects: 91% (1345/1478) 2024-08-08T20:37:13.6337473Z remote: Compressing objects: 92% (1360/1478) 2024-08-08T20:37:13.6338099Z remote: Compressing objects: 93% (1375/1478) 2024-08-08T20:37:13.6339253Z remote: Compressing objects: 94% (1390/1478) 2024-08-08T20:37:13.6340011Z remote: Compressing objects: 95% (1405/1478) 2024-08-08T20:37:13.6340750Z remote: Compressing objects: 96% (1419/1478) 2024-08-08T20:37:13.6341503Z remote: Compressing objects: 97% (1434/1478) 2024-08-08T20:37:13.6342255Z remote: Compressing objects: 98% (1449/1478) 2024-08-08T20:37:13.6342892Z remote: Compressing objects: 99% (1464/1478) 2024-08-08T20:37:13.6343522Z remote: Compressing objects: 100% (1478/1478) 2024-08-08T20:37:13.6344227Z remote: Compressing objects: 100% (1478/1478), done. 2024-08-08T20:37:38.8808162Z remote: Total 1011786 (delta 1841), reused 2052 (delta 1310), pack-reused 1008995 2024-08-08T20:37:50.3904696Z [command]/usr/bin/git rev-parse --verify --quiet b9d86fa89636e301796d4201f36d86c73f6e49bc^{object} 2024-08-08T20:37:50.3940847Z b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:37:50.3946753Z ##[endgroup] 2024-08-08T20:37:50.3947126Z ##[group]Determining the checkout info 2024-08-08T20:37:50.3947545Z ##[endgroup] 2024-08-08T20:37:50.3947883Z ##[group]Checking out the ref 2024-08-08T20:37:50.3948605Z [command]/usr/bin/git checkout --quiet --force b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:37:52.0079576Z ##[endgroup] 2024-08-08T20:37:52.0080266Z ##[group]Setting up auth for fetching submodules 2024-08-08T20:37:52.0087390Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-08T20:37:52.0151700Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-08T20:37:52.0182992Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-08T20:37:52.0224917Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-08T20:37:52.0255160Z ##[endgroup] 2024-08-08T20:37:52.0255872Z ##[group]Fetching submodules 2024-08-08T20:37:52.0258292Z [command]/usr/bin/git submodule sync --recursive 2024-08-08T20:37:52.0589413Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-08T20:37:52.0864559Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-08T20:37:52.0869043Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-08T20:37:52.0871508Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-08T20:37:52.0876125Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-08T20:37:52.0877608Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-08T20:37:52.0881106Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-08T20:37:52.0884635Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-08T20:37:52.0890775Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-08T20:37:52.0891655Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-08T20:37:52.0895719Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-08T20:37:52.0899321Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-08T20:37:52.0903062Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-08T20:37:52.0907158Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-08T20:37:52.0910814Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-08T20:37:52.0914665Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-08T20:37:52.0920553Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-08-08T20:37:52.0924684Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:37:52.0929084Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-08T20:37:52.0933367Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-08T20:37:52.0937628Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-08T20:37:52.0941946Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-08T20:37:52.0946251Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-08T20:37:52.0950706Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-08T20:37:52.0955058Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-08T20:37:52.0959593Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-08T20:37:52.0964135Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-08T20:37:52.0969374Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-08T20:37:52.0974003Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-08T20:37:52.0978754Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-08T20:37:52.0983530Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-08T20:37:52.0988674Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-08T20:37:52.0993741Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-08T20:37:52.0998695Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-08T20:37:52.1003609Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-08T20:37:52.1009159Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-08T20:37:52.1080365Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-08T20:37:53.3888384Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-08T20:37:54.4901310Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-08T20:37:55.6230227Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-08T20:37:56.7467415Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-08T20:37:59.2146345Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-08T20:38:25.3630552Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-08T20:38:26.8674252Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-08T20:38:28.1313925Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-08T20:38:29.6934622Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-08T20:38:32.0228470Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-08T20:38:35.0884145Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-08T20:38:38.9813082Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-08T20:38:40.5301440Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-08T20:38:42.7625235Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-08T20:38:44.1016537Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-08-08T20:38:44.6002991Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-08T20:38:45.3610011Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-08T20:38:46.0592016Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-08T20:38:47.2291099Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-08T20:38:47.9699833Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-08T20:38:48.6297647Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-08T20:38:50.7023456Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-08T20:38:51.7990192Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-08T20:38:52.9112471Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-08T20:39:03.4905860Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-08T20:39:05.8305851Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-08T20:39:09.6179601Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-08T20:39:10.2413132Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-08T20:39:17.4474907Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-08T20:39:17.9411066Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-08T20:39:18.5471357Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-08T20:39:19.8143060Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-08T20:39:20.4689934Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-08T20:39:21.4480279Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-08T20:39:23.6720544Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-08T20:39:23.6990394Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-08T20:39:23.7241483Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-08T20:39:23.7642103Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-08T20:39:23.8147102Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-08T20:39:24.8056625Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-08T20:39:24.8502595Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-08T20:39:24.9166376Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-08T20:39:25.0319943Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-08T20:39:25.0899163Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019' 2024-08-08T20:39:25.6194523Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-08T20:39:25.8756846Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-08T20:39:25.9697237Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-08T20:39:25.9775690Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:39:25.9779138Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:39:25.9784428Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:39:25.9787096Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:39:25.9790869Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:39:25.9849598Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-08T20:39:27.2426588Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-08T20:39:28.2082260Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-08T20:39:30.1689975Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-08T20:39:33.2063256Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-08T20:39:33.8801672Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-08T20:39:33.9935892Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-08T20:39:34.4168185Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-08T20:39:34.5030891Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-08T20:39:34.5370530Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-08T20:39:34.6776371Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-08T20:39:34.7379456Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-08T20:39:34.7673883Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-08-08T20:39:34.8246447Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-08T20:39:34.8712463Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-08T20:39:34.9330771Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-08T20:39:34.9642124Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-08T20:39:34.9698076Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-08T20:39:34.9736644Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-08T20:39:50.5059173Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-08T20:39:50.5500455Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-08T20:39:50.6531102Z Submodule path 'third_party/kineto': checked out 'da2f2682cabaf95d601fa2a9b7e0979f84fe7667' 2024-08-08T20:39:50.6595748Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:39:50.6598488Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:39:50.6601193Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:39:50.6643898Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-08T20:39:51.5145697Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-08T20:39:52.9144323Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-08T20:39:54.1927842Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-08T20:39:54.1987960Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:39:54.1991494Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:39:54.1995466Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:39:54.1997811Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:39:54.2001358Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:39:54.2004624Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:39:54.2008380Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:39:54.2012062Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:39:54.2086260Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-08T20:39:55.5748193Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-08T20:39:56.5110643Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-08T20:39:57.8883979Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-08T20:39:58.6111783Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-08T20:39:59.5771782Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-08T20:40:00.8657629Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-08T20:40:09.8142798Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-08T20:40:10.5882447Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-08T20:40:10.6260305Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-08T20:40:10.6783920Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-08T20:40:10.7058356Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-08T20:40:10.7133410Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:10.7189587Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-08T20:40:11.5059163Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-08T20:40:11.5422719Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-08T20:40:11.5976367Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-08T20:40:11.7143435Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-08T20:40:11.7491892Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-08T20:40:11.8053241Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-08T20:40:11.8764524Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-08T20:40:11.9338443Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-08T20:40:11.9770621Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-08T20:40:12.1013193Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-08T20:40:12.5175236Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-08T20:40:12.5247251Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:12.5249802Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:12.5319446Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-08T20:40:13.3748471Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-08T20:40:14.5930563Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-08T20:40:14.6478450Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-08T20:40:14.7344340Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-08T20:40:14.7391590Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:14.7394798Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:14.7398125Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:14.7401579Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:14.7405261Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:14.7410220Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:14.7412712Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:14.7416177Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:14.7465179Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-08T20:40:15.5234494Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-08T20:40:17.1790906Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-08T20:40:18.0017601Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-08T20:40:24.1438619Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-08T20:40:31.8294090Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-08T20:40:32.4291110Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-08T20:40:33.4386361Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-08T20:40:42.6935889Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-08T20:40:42.7473036Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-08T20:40:42.7787749Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-08T20:40:42.9004191Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-08T20:40:42.9318042Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-08T20:40:42.9600424Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-08T20:40:42.9935771Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-08T20:40:42.9985595Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:42.9991097Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:43.0053585Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-08T20:40:44.9449463Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-08T20:40:46.3836576Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-08T20:40:46.4520462Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-08T20:40:46.9813970Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-08T20:40:47.0226092Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-08T20:40:47.2990708Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-08T20:40:47.3067822Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:47.3070611Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:47.3126453Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-08T20:40:48.1612845Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-08T20:40:49.5182207Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-08T20:40:49.6013900Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-08T20:40:49.6297856Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-08T20:40:49.6609901Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-08T20:40:49.7150404Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-08T20:40:49.7651048Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-08T20:40:49.8235753Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-08T20:40:49.8667630Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-08T20:40:49.8714918Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:49.8719445Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:49.8722088Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:49.8725704Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:49.8775027Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-08T20:40:51.1816681Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-08T20:40:51.8500893Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-08T20:40:53.2964507Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-08T20:40:54.6032339Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-08T20:40:54.6362354Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-08T20:40:54.7169789Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-08T20:40:54.7618839Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-08T20:40:54.7694524Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:54.7738488Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-08T20:40:55.4009414Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-08T20:40:55.4095120Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-08T20:40:55.4364013Z Entering 'android/libs/fbjni' 2024-08-08T20:40:55.4402082Z Entering 'third_party/FP16' 2024-08-08T20:40:55.4439464Z Entering 'third_party/FXdiv' 2024-08-08T20:40:55.4478071Z Entering 'third_party/NNPACK' 2024-08-08T20:40:55.4513465Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T20:40:55.4549917Z Entering 'third_party/XNNPACK' 2024-08-08T20:40:55.4601821Z Entering 'third_party/benchmark' 2024-08-08T20:40:55.4639626Z Entering 'third_party/cpp-httplib' 2024-08-08T20:40:55.4676752Z Entering 'third_party/cpuinfo' 2024-08-08T20:40:55.4711590Z Entering 'third_party/cudnn_frontend' 2024-08-08T20:40:55.4747429Z Entering 'third_party/cutlass' 2024-08-08T20:40:55.4788904Z Entering 'third_party/eigen' 2024-08-08T20:40:55.4833253Z Entering 'third_party/fbgemm' 2024-08-08T20:40:55.4872754Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:40:55.4908306Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:40:55.4944331Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:40:55.4989501Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:40:55.5027857Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:40:55.5062721Z Entering 'third_party/flatbuffers' 2024-08-08T20:40:55.5106156Z Entering 'third_party/fmt' 2024-08-08T20:40:55.5141008Z Entering 'third_party/foxi' 2024-08-08T20:40:55.5180178Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:40:55.5220902Z Entering 'third_party/gloo' 2024-08-08T20:40:55.5266882Z Entering 'third_party/googletest' 2024-08-08T20:40:55.5302859Z Entering 'third_party/ideep' 2024-08-08T20:40:55.5340970Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T20:40:55.5391290Z Entering 'third_party/ittapi' 2024-08-08T20:40:55.5434325Z Entering 'third_party/kineto' 2024-08-08T20:40:55.5482832Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:40:55.5530108Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:40:55.5568182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:40:55.5601053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:40:55.5640893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:40:55.5681644Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:55.5730777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:40:55.5777014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:40:55.5817911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:40:55.5851938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:40:55.5889173Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:40:55.5939875Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:40:55.5983411Z Entering 'third_party/mimalloc' 2024-08-08T20:40:55.6023178Z Entering 'third_party/nccl/nccl' 2024-08-08T20:40:55.6073088Z Entering 'third_party/nlohmann' 2024-08-08T20:40:55.6113624Z Entering 'third_party/onnx' 2024-08-08T20:40:55.6162648Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:55.6201538Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:55.6243477Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T20:40:55.6284708Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:55.6322430Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:55.6358618Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:55.6392546Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:55.6432321Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:55.6465664Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:55.6497822Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:55.6529909Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:55.6570607Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:55.6613115Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:55.6664945Z Entering 'third_party/pocketfft' 2024-08-08T20:40:55.6700228Z Entering 'third_party/protobuf' 2024-08-08T20:40:55.6737443Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:55.6781521Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:55.6822163Z Entering 'third_party/psimd' 2024-08-08T20:40:55.6861790Z Entering 'third_party/pthreadpool' 2024-08-08T20:40:55.6905256Z Entering 'third_party/pybind11' 2024-08-08T20:40:55.6944662Z Entering 'third_party/python-peachpy' 2024-08-08T20:40:55.6979300Z Entering 'third_party/sleef' 2024-08-08T20:40:55.7019101Z Entering 'third_party/tensorpipe' 2024-08-08T20:40:55.7057572Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:55.7098322Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:55.7134313Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:55.7169294Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:55.7202585Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:55.7262736Z ##[endgroup] 2024-08-08T20:40:55.7264956Z ##[group]Persisting credentials for submodules 2024-08-08T20:40:55.7270303Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-08T20:40:55.7529483Z Entering 'android/libs/fbjni' 2024-08-08T20:40:55.7570701Z Entering 'third_party/FP16' 2024-08-08T20:40:55.7614382Z Entering 'third_party/FXdiv' 2024-08-08T20:40:55.7653353Z Entering 'third_party/NNPACK' 2024-08-08T20:40:55.7691859Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T20:40:55.7729693Z Entering 'third_party/XNNPACK' 2024-08-08T20:40:55.7782239Z Entering 'third_party/benchmark' 2024-08-08T20:40:55.7820841Z Entering 'third_party/cpp-httplib' 2024-08-08T20:40:55.7860302Z Entering 'third_party/cpuinfo' 2024-08-08T20:40:55.7899291Z Entering 'third_party/cudnn_frontend' 2024-08-08T20:40:55.7936756Z Entering 'third_party/cutlass' 2024-08-08T20:40:55.7985936Z Entering 'third_party/eigen' 2024-08-08T20:40:55.8025672Z Entering 'third_party/fbgemm' 2024-08-08T20:40:55.8063776Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:40:55.8100541Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:40:55.8141213Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:40:55.8188580Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:40:55.8224032Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:40:55.8267292Z Entering 'third_party/flatbuffers' 2024-08-08T20:40:55.8313790Z Entering 'third_party/fmt' 2024-08-08T20:40:55.8352896Z Entering 'third_party/foxi' 2024-08-08T20:40:55.8390624Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:40:55.8429101Z Entering 'third_party/gloo' 2024-08-08T20:40:55.8469072Z Entering 'third_party/googletest' 2024-08-08T20:40:55.8509726Z Entering 'third_party/ideep' 2024-08-08T20:40:55.8547459Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T20:40:55.8607201Z Entering 'third_party/ittapi' 2024-08-08T20:40:55.8656015Z Entering 'third_party/kineto' 2024-08-08T20:40:55.8698115Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:40:55.8734893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:40:55.8772549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:40:55.8809767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:40:55.8844978Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:40:55.8879285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:55.8930254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:40:55.8968008Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:40:55.9005157Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:40:55.9041461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:40:55.9080401Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:40:55.9125242Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:40:55.9168395Z Entering 'third_party/mimalloc' 2024-08-08T20:40:55.9215085Z Entering 'third_party/nccl/nccl' 2024-08-08T20:40:55.9253731Z Entering 'third_party/nlohmann' 2024-08-08T20:40:55.9296823Z Entering 'third_party/onnx' 2024-08-08T20:40:55.9347395Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:55.9391835Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:55.9432249Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T20:40:55.9473303Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:55.9514202Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:55.9549294Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:55.9586766Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:55.9626721Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:55.9666567Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:55.9706950Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:55.9742072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:55.9782270Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:55.9825128Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:55.9882366Z Entering 'third_party/pocketfft' 2024-08-08T20:40:55.9923236Z Entering 'third_party/protobuf' 2024-08-08T20:40:55.9989156Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:56.0045305Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:56.0098652Z Entering 'third_party/psimd' 2024-08-08T20:40:56.0151884Z Entering 'third_party/pthreadpool' 2024-08-08T20:40:56.0195361Z Entering 'third_party/pybind11' 2024-08-08T20:40:56.0234306Z Entering 'third_party/python-peachpy' 2024-08-08T20:40:56.0280492Z Entering 'third_party/sleef' 2024-08-08T20:40:56.0330289Z Entering 'third_party/tensorpipe' 2024-08-08T20:40:56.0371903Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:56.0413154Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:56.0450826Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:56.0489689Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:56.0531820Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:56.0609983Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-08T20:40:56.0877051Z Entering 'android/libs/fbjni' 2024-08-08T20:40:56.0918256Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-08T20:40:56.0940654Z Entering 'third_party/FP16' 2024-08-08T20:40:56.0973757Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-08T20:40:56.0991886Z Entering 'third_party/FXdiv' 2024-08-08T20:40:56.1024986Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-08T20:40:56.1042883Z Entering 'third_party/NNPACK' 2024-08-08T20:40:56.1078704Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-08T20:40:56.1101456Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T20:40:56.1140030Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-08T20:40:56.1160001Z Entering 'third_party/XNNPACK' 2024-08-08T20:40:56.1195255Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-08T20:40:56.1229284Z Entering 'third_party/benchmark' 2024-08-08T20:40:56.1264774Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-08T20:40:56.1283475Z Entering 'third_party/cpp-httplib' 2024-08-08T20:40:56.1316528Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-08T20:40:56.1333873Z Entering 'third_party/cpuinfo' 2024-08-08T20:40:56.1369772Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-08T20:40:56.1388825Z Entering 'third_party/cudnn_frontend' 2024-08-08T20:40:56.1428870Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-08T20:40:56.1449181Z Entering 'third_party/cutlass' 2024-08-08T20:40:56.1483759Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-08T20:40:56.1508547Z Entering 'third_party/eigen' 2024-08-08T20:40:56.1543358Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-08T20:40:56.1563214Z Entering 'third_party/fbgemm' 2024-08-08T20:40:56.1595195Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-08T20:40:56.1612646Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:40:56.1645306Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-08T20:40:56.1666693Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:40:56.1701595Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-08T20:40:56.1720887Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:40:56.1764129Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-08T20:40:56.1789876Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:40:56.1828063Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.1849120Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:40:56.1890076Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-08T20:40:56.1908736Z Entering 'third_party/flatbuffers' 2024-08-08T20:40:56.1949332Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-08T20:40:56.1974684Z Entering 'third_party/fmt' 2024-08-08T20:40:56.2009033Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-08T20:40:56.2024937Z Entering 'third_party/foxi' 2024-08-08T20:40:56.2060340Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-08-08T20:40:56.2081453Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:40:56.2121804Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-08T20:40:56.2138766Z Entering 'third_party/gloo' 2024-08-08T20:40:56.2174673Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-08T20:40:56.2191090Z Entering 'third_party/googletest' 2024-08-08T20:40:56.2223476Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.2242368Z Entering 'third_party/ideep' 2024-08-08T20:40:56.2285560Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-08T20:40:56.2303171Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T20:40:56.2340594Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-08T20:40:56.2370088Z Entering 'third_party/ittapi' 2024-08-08T20:40:56.2403379Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-08T20:40:56.2420609Z Entering 'third_party/kineto' 2024-08-08T20:40:56.2459422Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-08T20:40:56.2478246Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:40:56.2518659Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-08T20:40:56.2537026Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:40:56.2573432Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-08T20:40:56.2590400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:40:56.2625116Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-08T20:40:56.2640595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:40:56.2683370Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-08T20:40:56.2700480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:40:56.2739517Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-08T20:40:56.2756379Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:56.2799940Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-08T20:40:56.2818007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:40:56.2856896Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-08T20:40:56.2879767Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:40:56.2919600Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.2948850Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:40:56.2983615Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-08T20:40:56.3007707Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:40:56.3050127Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-08T20:40:56.3066937Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:40:56.3103520Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-08T20:40:56.3123506Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:40:56.3155444Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.3175487Z Entering 'third_party/mimalloc' 2024-08-08T20:40:56.3210030Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-08T20:40:56.3227632Z Entering 'third_party/nccl/nccl' 2024-08-08T20:40:56.3263421Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-08T20:40:56.3281365Z Entering 'third_party/nlohmann' 2024-08-08T20:40:56.3319297Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-08T20:40:56.3339341Z Entering 'third_party/onnx' 2024-08-08T20:40:56.3378694Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-08T20:40:56.3411796Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:56.3444430Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-08T20:40:56.3462762Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:56.3503426Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-08T20:40:56.3529175Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T20:40:56.3562722Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-08T20:40:56.3589169Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:56.3620793Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-08T20:40:56.3638496Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:56.3677156Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.3694066Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:56.3726579Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-08T20:40:56.3748647Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:56.3778627Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-08T20:40:56.3800739Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:56.3837754Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-08T20:40:56.3853831Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:56.3889099Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-08T20:40:56.3909871Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:56.3941649Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-08T20:40:56.3958278Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:56.3995118Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-08T20:40:56.4014050Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:56.4050403Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-08T20:40:56.4067463Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:56.4103090Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-08T20:40:56.4138502Z Entering 'third_party/pocketfft' 2024-08-08T20:40:56.4172307Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-08T20:40:56.4190343Z Entering 'third_party/protobuf' 2024-08-08T20:40:56.4225173Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-08T20:40:56.4245187Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:56.4285429Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-08T20:40:56.4303317Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:56.4339988Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.4363422Z Entering 'third_party/psimd' 2024-08-08T20:40:56.4400140Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-08T20:40:56.4417675Z Entering 'third_party/pthreadpool' 2024-08-08T20:40:56.4450704Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-08T20:40:56.4468821Z Entering 'third_party/pybind11' 2024-08-08T20:40:56.4501480Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-08T20:40:56.4519116Z Entering 'third_party/python-peachpy' 2024-08-08T20:40:56.4559515Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-08T20:40:56.4583188Z Entering 'third_party/sleef' 2024-08-08T20:40:56.4623444Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-08T20:40:56.4640598Z Entering 'third_party/tensorpipe' 2024-08-08T20:40:56.4681907Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-08T20:40:56.4699576Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:56.4742960Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-08T20:40:56.4763484Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:56.4809378Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-08T20:40:56.4823399Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:56.4860394Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-08T20:40:56.4883187Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:56.4924536Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-08T20:40:56.4947534Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:56.4978461Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-08T20:40:56.5217920Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-08T20:40:56.5471928Z Entering 'android/libs/fbjni' 2024-08-08T20:40:56.5518555Z Entering 'third_party/FP16' 2024-08-08T20:40:56.5557017Z Entering 'third_party/FXdiv' 2024-08-08T20:40:56.5596934Z Entering 'third_party/NNPACK' 2024-08-08T20:40:56.5633492Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T20:40:56.5668000Z Entering 'third_party/XNNPACK' 2024-08-08T20:40:56.5719058Z Entering 'third_party/benchmark' 2024-08-08T20:40:56.5759434Z Entering 'third_party/cpp-httplib' 2024-08-08T20:40:56.5796199Z Entering 'third_party/cpuinfo' 2024-08-08T20:40:56.5831265Z Entering 'third_party/cudnn_frontend' 2024-08-08T20:40:56.5868648Z Entering 'third_party/cutlass' 2024-08-08T20:40:56.5911105Z Entering 'third_party/eigen' 2024-08-08T20:40:56.5950573Z Entering 'third_party/fbgemm' 2024-08-08T20:40:56.5987453Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:40:56.6025950Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:40:56.6061514Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:40:56.6100573Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:40:56.6138174Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:40:56.6173753Z Entering 'third_party/flatbuffers' 2024-08-08T20:40:56.6213871Z Entering 'third_party/fmt' 2024-08-08T20:40:56.6248324Z Entering 'third_party/foxi' 2024-08-08T20:40:56.6281309Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:40:56.6316640Z Entering 'third_party/gloo' 2024-08-08T20:40:56.6351102Z Entering 'third_party/googletest' 2024-08-08T20:40:56.6388589Z Entering 'third_party/ideep' 2024-08-08T20:40:56.6423174Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T20:40:56.6463207Z Entering 'third_party/ittapi' 2024-08-08T20:40:56.6503243Z Entering 'third_party/kineto' 2024-08-08T20:40:56.6540800Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:40:56.6578727Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:40:56.6618520Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:40:56.6654374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:40:56.6688150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:40:56.6730407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:56.6776244Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:40:56.6813020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:40:56.6845638Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:40:56.6886257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:40:56.6924728Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:40:56.6961724Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:40:56.6999043Z Entering 'third_party/mimalloc' 2024-08-08T20:40:56.7034258Z Entering 'third_party/nccl/nccl' 2024-08-08T20:40:56.7068369Z Entering 'third_party/nlohmann' 2024-08-08T20:40:56.7105414Z Entering 'third_party/onnx' 2024-08-08T20:40:56.7153934Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:56.7203107Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:56.7245307Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T20:40:56.7283285Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:56.7319148Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:56.7355357Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:56.7390733Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:56.7425922Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:56.7456789Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:56.7489713Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:56.7532687Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:56.7571865Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:56.7608644Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:56.7660745Z Entering 'third_party/pocketfft' 2024-08-08T20:40:56.7694787Z Entering 'third_party/protobuf' 2024-08-08T20:40:56.7741049Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:56.7785804Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:56.7826066Z Entering 'third_party/psimd' 2024-08-08T20:40:56.7865233Z Entering 'third_party/pthreadpool' 2024-08-08T20:40:56.7905975Z Entering 'third_party/pybind11' 2024-08-08T20:40:56.7945569Z Entering 'third_party/python-peachpy' 2024-08-08T20:40:56.7983752Z Entering 'third_party/sleef' 2024-08-08T20:40:56.8024244Z Entering 'third_party/tensorpipe' 2024-08-08T20:40:56.8068069Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:56.8105781Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:56.8138638Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:56.8173759Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:56.8211653Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:56.8274902Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-08T20:40:56.8547533Z Entering 'android/libs/fbjni' 2024-08-08T20:40:56.8586781Z Entering 'third_party/FP16' 2024-08-08T20:40:56.8628968Z Entering 'third_party/FXdiv' 2024-08-08T20:40:56.8665473Z Entering 'third_party/NNPACK' 2024-08-08T20:40:56.8701476Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T20:40:56.8742412Z Entering 'third_party/XNNPACK' 2024-08-08T20:40:56.8791165Z Entering 'third_party/benchmark' 2024-08-08T20:40:56.8830244Z Entering 'third_party/cpp-httplib' 2024-08-08T20:40:56.8864336Z Entering 'third_party/cpuinfo' 2024-08-08T20:40:56.8898382Z Entering 'third_party/cudnn_frontend' 2024-08-08T20:40:56.8932714Z Entering 'third_party/cutlass' 2024-08-08T20:40:56.8979298Z Entering 'third_party/eigen' 2024-08-08T20:40:56.9024559Z Entering 'third_party/fbgemm' 2024-08-08T20:40:56.9063536Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T20:40:56.9106721Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T20:40:56.9147147Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T20:40:56.9184700Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T20:40:56.9218131Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T20:40:56.9257483Z Entering 'third_party/flatbuffers' 2024-08-08T20:40:56.9293952Z Entering 'third_party/fmt' 2024-08-08T20:40:56.9331588Z Entering 'third_party/foxi' 2024-08-08T20:40:56.9364650Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T20:40:56.9401003Z Entering 'third_party/gloo' 2024-08-08T20:40:56.9437578Z Entering 'third_party/googletest' 2024-08-08T20:40:56.9481148Z Entering 'third_party/ideep' 2024-08-08T20:40:56.9520042Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T20:40:56.9580685Z Entering 'third_party/ittapi' 2024-08-08T20:40:56.9619931Z Entering 'third_party/kineto' 2024-08-08T20:40:56.9654000Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T20:40:56.9688146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T20:40:56.9725922Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T20:40:56.9760150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T20:40:56.9800634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T20:40:56.9839130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T20:40:56.9883201Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T20:40:56.9918395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T20:40:56.9955901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T20:40:56.9990081Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T20:40:57.0028604Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T20:40:57.0061317Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T20:40:57.0097452Z Entering 'third_party/mimalloc' 2024-08-08T20:40:57.0133525Z Entering 'third_party/nccl/nccl' 2024-08-08T20:40:57.0168090Z Entering 'third_party/nlohmann' 2024-08-08T20:40:57.0202507Z Entering 'third_party/onnx' 2024-08-08T20:40:57.0270349Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T20:40:57.0309863Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T20:40:57.0351386Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T20:40:57.0389157Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T20:40:57.0429751Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T20:40:57.0465651Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T20:40:57.0498896Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T20:40:57.0538915Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T20:40:57.0577419Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T20:40:57.0613228Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T20:40:57.0644821Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T20:40:57.0691529Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T20:40:57.0728894Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T20:40:57.0778946Z Entering 'third_party/pocketfft' 2024-08-08T20:40:57.0816687Z Entering 'third_party/protobuf' 2024-08-08T20:40:57.0855122Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T20:40:57.0897579Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T20:40:57.0934518Z Entering 'third_party/psimd' 2024-08-08T20:40:57.0969165Z Entering 'third_party/pthreadpool' 2024-08-08T20:40:57.1002586Z Entering 'third_party/pybind11' 2024-08-08T20:40:57.1042924Z Entering 'third_party/python-peachpy' 2024-08-08T20:40:57.1079807Z Entering 'third_party/sleef' 2024-08-08T20:40:57.1113041Z Entering 'third_party/tensorpipe' 2024-08-08T20:40:57.1146629Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T20:40:57.1184550Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T20:40:57.1225209Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T20:40:57.1259559Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T20:40:57.1297998Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T20:40:57.1352033Z ##[endgroup] 2024-08-08T20:40:57.1384204Z [command]/usr/bin/git log -1 --format='%H' 2024-08-08T20:40:57.1411889Z 'b9d86fa89636e301796d4201f36d86c73f6e49bc' 2024-08-08T20:40:57.1634401Z Prepare all required actions 2024-08-08T20:40:57.1635137Z Getting action download info 2024-08-08T20:40:57.2025738Z ##[group]Run ./.github/actions/setup-rocm 2024-08-08T20:40:57.2025972Z env: 2024-08-08T20:40:57.2026130Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:40:57.2026317Z ##[endgroup] 2024-08-08T20:40:57.2043187Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-08-08T20:40:57.2043677Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-08-08T20:40:57.2060377Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:40:57.2060661Z env: 2024-08-08T20:40:57.2060822Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:40:57.2061012Z ##[endgroup] 2024-08-08T20:40:57.2140894Z ##[group]Run set -ex 2024-08-08T20:40:57.2141119Z set -ex 2024-08-08T20:40:57.2141277Z  2024-08-08T20:40:57.2141467Z cat ~/.docker/config.json || true 2024-08-08T20:40:57.2142202Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-08-08T20:40:57.2142859Z rm -f ~/.docker/config.json 2024-08-08T20:40:57.2159160Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:40:57.2159435Z env: 2024-08-08T20:40:57.2159597Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:40:57.2159824Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:40:57.2160063Z ##[endgroup] 2024-08-08T20:40:57.2196081Z + cat /home/pytorchci/.docker/config.json 2024-08-08T20:40:57.2203752Z { 2024-08-08T20:40:57.2204162Z "auths": {} 2024-08-08T20:40:57.2204678Z + rm -f /home/pytorchci/.docker/config.json 2024-08-08T20:40:57.2214599Z } 2024-08-08T20:40:57.2233777Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-08T20:40:57.2234188Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-08T20:40:57.2234489Z # shellcheck disable=SC2046 2024-08-08T20:40:57.2234767Z docker stop $(docker ps -q) || true 2024-08-08T20:40:57.2235031Z # Prune all stopped containers. 2024-08-08T20:40:57.2235264Z docker container prune -f 2024-08-08T20:40:57.2248145Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:40:57.2248439Z env: 2024-08-08T20:40:57.2248597Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:40:57.2248833Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:40:57.2249082Z ##[endgroup] 2024-08-08T20:40:58.7166402Z b71503081d68 2024-08-08T20:41:14.9770224Z Deleted Containers: 2024-08-08T20:41:14.9770974Z b71503081d6877cfcc5196c4c2482613be8910925d3b1da429de93b3b7a08dc5 2024-08-08T20:41:14.9771544Z 2024-08-08T20:41:14.9771777Z Total reclaimed space: 6.705GB 2024-08-08T20:41:14.9825504Z ##[group]Run cat /etc/os-release || true 2024-08-08T20:41:14.9825856Z cat /etc/os-release || true 2024-08-08T20:41:14.9826228Z cat /etc/apt/sources.list.d/rocm.list || true 2024-08-08T20:41:14.9826791Z cat /opt/rocm/.info/version || true 2024-08-08T20:41:14.9827439Z whoami 2024-08-08T20:41:14.9862455Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:14.9862801Z env: 2024-08-08T20:41:14.9862999Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:14.9863270Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:14.9863571Z ##[endgroup] 2024-08-08T20:41:14.9939557Z PRETTY_NAME="Ubuntu 22.04.3 LTS" 2024-08-08T20:41:14.9940158Z NAME="Ubuntu" 2024-08-08T20:41:14.9940648Z VERSION_ID="22.04" 2024-08-08T20:41:14.9941074Z VERSION="22.04.3 LTS (Jammy Jellyfish)" 2024-08-08T20:41:14.9941439Z VERSION_CODENAME=jammy 2024-08-08T20:41:14.9941660Z ID=ubuntu 2024-08-08T20:41:14.9941840Z ID_LIKE=debian 2024-08-08T20:41:14.9942102Z HOME_URL="https://www.ubuntu.com/" 2024-08-08T20:41:14.9942402Z SUPPORT_URL="https://help.ubuntu.com/" 2024-08-08T20:41:14.9942746Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-08T20:41:14.9943701Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-08T20:41:14.9944145Z UBUNTU_CODENAME=jammy 2024-08-08T20:41:14.9956029Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.1 jammy main 2024-08-08T20:41:14.9973825Z 6.1.0-82 2024-08-08T20:41:14.9983329Z pytorchci 2024-08-08T20:41:15.0010769Z ##[group]Run rocm-smi 2024-08-08T20:41:15.0011247Z rocm-smi 2024-08-08T20:41:15.0043636Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.0044287Z env: 2024-08-08T20:41:15.0044727Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.0045406Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.0046049Z ##[endgroup] 2024-08-08T20:41:15.0862841Z 2024-08-08T20:41:15.0863002Z 2024-08-08T20:41:15.0863470Z ====================================== ROCm System Management Interface ====================================== 2024-08-08T20:41:15.0863999Z ================================================ Concise Info ================================================ 2024-08-08T20:41:15.0864597Z Device [Model : Revision] Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-08-08T20:41:15.0865585Z  Name (20 chars) (Edge) (Avg) (Mem, Compute)  2024-08-08T20:41:15.0866122Z ============================================================================================================== 2024-08-08T20:41:15.0866867Z 0 [0x0c34 : 0x02] 34.0°C 43.0W N/A, N/A 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-08-08T20:41:15.0874832Z Instinct MI210 2024-08-08T20:41:15.0875510Z 1 [0x0c34 : 0x02] 33.0°C 40.0W N/A, N/A 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-08-08T20:41:15.0886228Z Instinct MI210 2024-08-08T20:41:15.0892538Z ============================================================================================================== 2024-08-08T20:41:15.0893383Z ============================================ End of ROCm SMI Log ============================================= 2024-08-08T20:41:15.1015056Z ##[group]Run rocminfo 2024-08-08T20:41:15.1015508Z rocminfo 2024-08-08T20:41:15.1042521Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.1043193Z env: 2024-08-08T20:41:15.1043566Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.1044495Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.1045212Z ##[endgroup] 2024-08-08T20:41:15.1824358Z ROCk module version 6.7.0 is loaded 2024-08-08T20:41:15.1825042Z ===================== 2024-08-08T20:41:15.1825529Z HSA System Attributes 2024-08-08T20:41:15.1825948Z ===================== 2024-08-08T20:41:15.1826422Z Runtime Version: 1.13 2024-08-08T20:41:15.1826819Z Runtime Ext Version: 1.4 2024-08-08T20:41:15.1827102Z System Timestamp Freq.: 1000.000000MHz 2024-08-08T20:41:15.1827527Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-08-08T20:41:15.1827987Z Machine Model: LARGE 2024-08-08T20:41:15.1828349Z System Endianness: LITTLE 2024-08-08T20:41:15.1828717Z Mwaitx: DISABLED 2024-08-08T20:41:15.1828986Z DMAbuf Support: YES 2024-08-08T20:41:15.1829165Z 2024-08-08T20:41:15.1829260Z ========== 2024-08-08T20:41:15.1829531Z HSA Agents 2024-08-08T20:41:15.1829782Z ========== 2024-08-08T20:41:15.1830035Z ******* 2024-08-08T20:41:15.1830280Z Agent 1 2024-08-08T20:41:15.1830527Z ******* 2024-08-08T20:41:15.1830828Z Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:41:15.1831195Z Uuid: CPU-XX 2024-08-08T20:41:15.1831796Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:41:15.1832162Z Vendor Name: CPU 2024-08-08T20:41:15.1832509Z Feature: None specified 2024-08-08T20:41:15.1832891Z Profile: FULL_PROFILE 2024-08-08T20:41:15.1833246Z Float Round Mode: NEAR 2024-08-08T20:41:15.1833609Z Max Queue Number: 0(0x0) 2024-08-08T20:41:15.1833978Z Queue Min Size: 0(0x0) 2024-08-08T20:41:15.1834311Z Queue Max Size: 0(0x0) 2024-08-08T20:41:15.1834648Z Queue Type: MULTI 2024-08-08T20:41:15.1834961Z Node: 0 2024-08-08T20:41:15.1835273Z Device Type: CPU 2024-08-08T20:41:15.1835591Z Cache Info: 2024-08-08T20:41:15.1835854Z L1: 32768(0x8000) KB 2024-08-08T20:41:15.1836175Z Chip ID: 0(0x0) 2024-08-08T20:41:15.1836512Z ASIC Revision: 0(0x0) 2024-08-08T20:41:15.1836868Z Cacheline Size: 64(0x40) 2024-08-08T20:41:15.1837211Z Max Clock Freq. (MHz): 2600 2024-08-08T20:41:15.1837540Z BDFID: 0 2024-08-08T20:41:15.1837873Z Internal Node ID: 0 2024-08-08T20:41:15.1838222Z Compute Unit: 32 2024-08-08T20:41:15.1838551Z SIMDs per CU: 0 2024-08-08T20:41:15.1838885Z Shader Engines: 0 2024-08-08T20:41:15.1839243Z Shader Arrs. per Eng.: 0 2024-08-08T20:41:15.1839605Z WatchPts on Addr. Ranges:1 2024-08-08T20:41:15.1839923Z Features: None 2024-08-08T20:41:15.1840161Z Pool Info: 2024-08-08T20:41:15.1840385Z Pool 1 2024-08-08T20:41:15.1840676Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:41:15.1841025Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:41:15.1841358Z Allocatable: TRUE 2024-08-08T20:41:15.1845695Z Alloc Granule: 4KB 2024-08-08T20:41:15.1846274Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1847004Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1847702Z Accessible by all: TRUE 2024-08-08T20:41:15.1848432Z Pool 2 2024-08-08T20:41:15.1848999Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-08T20:41:15.1849660Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:41:15.1850307Z Allocatable: TRUE 2024-08-08T20:41:15.1850983Z Alloc Granule: 4KB 2024-08-08T20:41:15.1851697Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1852409Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1853108Z Accessible by all: TRUE 2024-08-08T20:41:15.1853712Z Pool 3 2024-08-08T20:41:15.1854258Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:41:15.1854902Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:41:15.1855544Z Allocatable: TRUE 2024-08-08T20:41:15.1856475Z Alloc Granule: 4KB 2024-08-08T20:41:15.1856846Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1857215Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1857576Z Accessible by all: TRUE 2024-08-08T20:41:15.1857886Z ISA Info: 2024-08-08T20:41:15.1858107Z ******* 2024-08-08T20:41:15.1858323Z Agent 2 2024-08-08T20:41:15.1858539Z ******* 2024-08-08T20:41:15.1858810Z Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:41:15.1859146Z Uuid: CPU-XX 2024-08-08T20:41:15.1859486Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:41:15.1859852Z Vendor Name: CPU 2024-08-08T20:41:15.1860204Z Feature: None specified 2024-08-08T20:41:15.1860548Z Profile: FULL_PROFILE 2024-08-08T20:41:15.1860886Z Float Round Mode: NEAR 2024-08-08T20:41:15.1861239Z Max Queue Number: 0(0x0) 2024-08-08T20:41:15.1861585Z Queue Min Size: 0(0x0) 2024-08-08T20:41:15.1861918Z Queue Max Size: 0(0x0) 2024-08-08T20:41:15.1862253Z Queue Type: MULTI 2024-08-08T20:41:15.1862576Z Node: 1 2024-08-08T20:41:15.1862892Z Device Type: CPU 2024-08-08T20:41:15.1863192Z Cache Info: 2024-08-08T20:41:15.1863444Z L1: 32768(0x8000) KB 2024-08-08T20:41:15.1863742Z Chip ID: 0(0x0) 2024-08-08T20:41:15.1864073Z ASIC Revision: 0(0x0) 2024-08-08T20:41:15.1864420Z Cacheline Size: 64(0x40) 2024-08-08T20:41:15.1864765Z Max Clock Freq. (MHz): 2600 2024-08-08T20:41:15.1865088Z BDFID: 0 2024-08-08T20:41:15.1865427Z Internal Node ID: 1 2024-08-08T20:41:15.1865822Z Compute Unit: 32 2024-08-08T20:41:15.1866386Z SIMDs per CU: 0 2024-08-08T20:41:15.1866784Z Shader Engines: 0 2024-08-08T20:41:15.1867192Z Shader Arrs. per Eng.: 0 2024-08-08T20:41:15.1867592Z WatchPts on Addr. Ranges:1 2024-08-08T20:41:15.1867907Z Features: None 2024-08-08T20:41:15.1868141Z Pool Info: 2024-08-08T20:41:15.1868376Z Pool 1 2024-08-08T20:41:15.1868663Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:41:15.1869002Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:41:15.1869336Z Allocatable: TRUE 2024-08-08T20:41:15.1869708Z Alloc Granule: 4KB 2024-08-08T20:41:15.1870091Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1870490Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1870849Z Accessible by all: TRUE 2024-08-08T20:41:15.1871152Z Pool 2 2024-08-08T20:41:15.1871437Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-08T20:41:15.1871770Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:41:15.1872218Z Allocatable: TRUE 2024-08-08T20:41:15.1872574Z Alloc Granule: 4KB 2024-08-08T20:41:15.1872953Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1873322Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1873675Z Accessible by all: TRUE 2024-08-08T20:41:15.1873986Z Pool 3 2024-08-08T20:41:15.1874275Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:41:15.1874600Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:41:15.1874935Z Allocatable: TRUE 2024-08-08T20:41:15.1875283Z Alloc Granule: 4KB 2024-08-08T20:41:15.1875644Z Alloc Recommended Granule:4KB 2024-08-08T20:41:15.1876072Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1876749Z Accessible by all: TRUE 2024-08-08T20:41:15.1877464Z ISA Info: 2024-08-08T20:41:15.1877962Z ******* 2024-08-08T20:41:15.1878451Z Agent 3 2024-08-08T20:41:15.1878906Z ******* 2024-08-08T20:41:15.1879375Z Name: gfx90a 2024-08-08T20:41:15.1880000Z Uuid: GPU-0741618dc9e991d3 2024-08-08T20:41:15.1880669Z Marketing Name: AMD Instinct MI210 2024-08-08T20:41:15.1881358Z Vendor Name: AMD 2024-08-08T20:41:15.1882018Z Feature: KERNEL_DISPATCH 2024-08-08T20:41:15.1882675Z Profile: BASE_PROFILE 2024-08-08T20:41:15.1883344Z Float Round Mode: NEAR 2024-08-08T20:41:15.1884030Z Max Queue Number: 128(0x80) 2024-08-08T20:41:15.1884681Z Queue Min Size: 64(0x40) 2024-08-08T20:41:15.1885426Z Queue Max Size: 131072(0x20000) 2024-08-08T20:41:15.1886074Z Queue Type: MULTI 2024-08-08T20:41:15.1886677Z Node: 2 2024-08-08T20:41:15.1887288Z Device Type: GPU 2024-08-08T20:41:15.1888178Z Cache Info: 2024-08-08T20:41:15.1888646Z L1: 16(0x10) KB 2024-08-08T20:41:15.1889218Z L2: 8192(0x2000) KB 2024-08-08T20:41:15.1889803Z Chip ID: 29711(0x740f) 2024-08-08T20:41:15.1890440Z ASIC Revision: 1(0x1) 2024-08-08T20:41:15.1891127Z Cacheline Size: 64(0x40) 2024-08-08T20:41:15.1891810Z Max Clock Freq. (MHz): 1700 2024-08-08T20:41:15.1892432Z BDFID: 768 2024-08-08T20:41:15.1893072Z Internal Node ID: 2 2024-08-08T20:41:15.1893739Z Compute Unit: 104 2024-08-08T20:41:15.1894376Z SIMDs per CU: 4 2024-08-08T20:41:15.1895051Z Shader Engines: 8 2024-08-08T20:41:15.1895798Z Shader Arrs. per Eng.: 1 2024-08-08T20:41:15.1896638Z WatchPts on Addr. Ranges:4 2024-08-08T20:41:15.1897479Z Coherent Host Access: FALSE 2024-08-08T20:41:15.1898249Z Features: KERNEL_DISPATCH 2024-08-08T20:41:15.1899000Z Fast F16 Operation: TRUE 2024-08-08T20:41:15.1900153Z Wavefront Size: 64(0x40) 2024-08-08T20:41:15.1900790Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:41:15.1901176Z Workgroup Max Size per Dimension: 2024-08-08T20:41:15.1901458Z x 1024(0x400) 2024-08-08T20:41:15.1901750Z y 1024(0x400) 2024-08-08T20:41:15.1902035Z z 1024(0x400) 2024-08-08T20:41:15.1902353Z Max Waves Per CU: 32(0x20) 2024-08-08T20:41:15.1902711Z Max Work-item Per CU: 2048(0x800) 2024-08-08T20:41:15.1903064Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:41:15.1903378Z Grid Max Size per Dimension: 2024-08-08T20:41:15.1903646Z x 4294967295(0xffffffff) 2024-08-08T20:41:15.1903952Z y 4294967295(0xffffffff) 2024-08-08T20:41:15.1904240Z z 4294967295(0xffffffff) 2024-08-08T20:41:15.1904579Z Max fbarriers/Workgrp: 32 2024-08-08T20:41:15.1912848Z Packet Processor uCode:: 82 2024-08-08T20:41:15.1913665Z SDMA engine uCode:: 8 2024-08-08T20:41:15.1914425Z IOMMU Support:: None 2024-08-08T20:41:15.1915056Z Pool Info: 2024-08-08T20:41:15.1915508Z Pool 1 2024-08-08T20:41:15.1916077Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:41:15.1916756Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:41:15.1917403Z Allocatable: TRUE 2024-08-08T20:41:15.1918089Z Alloc Granule: 4KB 2024-08-08T20:41:15.1918812Z Alloc Recommended Granule:2048KB 2024-08-08T20:41:15.1919532Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1920232Z Accessible by all: FALSE 2024-08-08T20:41:15.1920833Z Pool 2 2024-08-08T20:41:15.1921372Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-08T20:41:15.1922038Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:41:15.1923045Z Allocatable: TRUE 2024-08-08T20:41:15.1923732Z Alloc Granule: 4KB 2024-08-08T20:41:15.1924429Z Alloc Recommended Granule:2048KB 2024-08-08T20:41:15.1925310Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1926010Z Accessible by all: FALSE 2024-08-08T20:41:15.1926613Z Pool 3 2024-08-08T20:41:15.1927146Z Segment: GROUP 2024-08-08T20:41:15.1927759Z Size: 64(0x40) KB 2024-08-08T20:41:15.1928384Z Allocatable: FALSE 2024-08-08T20:41:15.1929063Z Alloc Granule: 0KB 2024-08-08T20:41:15.1929766Z Alloc Recommended Granule:0KB 2024-08-08T20:41:15.1930511Z Alloc Alignment: 0KB 2024-08-08T20:41:15.1931238Z Accessible by all: FALSE 2024-08-08T20:41:15.1931855Z ISA Info: 2024-08-08T20:41:15.1932298Z ISA 1 2024-08-08T20:41:15.1932877Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-08T20:41:15.1933634Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-08T20:41:15.1934658Z Profiles: HSA_PROFILE_BASE 2024-08-08T20:41:15.1935368Z Default Rounding Mode: NEAR 2024-08-08T20:41:15.1936247Z Default Rounding Mode: NEAR 2024-08-08T20:41:15.1937030Z Fast f16: TRUE 2024-08-08T20:41:15.1937826Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:41:15.1938597Z Workgroup Max Size per Dimension: 2024-08-08T20:41:15.1939266Z x 1024(0x400) 2024-08-08T20:41:15.1939933Z y 1024(0x400) 2024-08-08T20:41:15.1940502Z z 1024(0x400) 2024-08-08T20:41:15.1940868Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:41:15.1941209Z Grid Max Size per Dimension: 2024-08-08T20:41:15.1941494Z x 4294967295(0xffffffff) 2024-08-08T20:41:15.1941795Z y 4294967295(0xffffffff) 2024-08-08T20:41:15.1942090Z z 4294967295(0xffffffff) 2024-08-08T20:41:15.1942421Z FBarrier Max Size: 32 2024-08-08T20:41:15.1942722Z ******* 2024-08-08T20:41:15.1942949Z Agent 4 2024-08-08T20:41:15.1943167Z ******* 2024-08-08T20:41:15.1943418Z Name: gfx90a 2024-08-08T20:41:15.1943754Z Uuid: GPU-d2f63e3de4d6eee3 2024-08-08T20:41:15.1944104Z Marketing Name: AMD Instinct MI210 2024-08-08T20:41:15.1944453Z Vendor Name: AMD 2024-08-08T20:41:15.1944792Z Feature: KERNEL_DISPATCH 2024-08-08T20:41:15.1945147Z Profile: BASE_PROFILE 2024-08-08T20:41:15.1945500Z Float Round Mode: NEAR 2024-08-08T20:41:15.1945846Z Max Queue Number: 128(0x80) 2024-08-08T20:41:15.1946210Z Queue Min Size: 64(0x40) 2024-08-08T20:41:15.1946628Z Queue Max Size: 131072(0x20000) 2024-08-08T20:41:15.1947351Z Queue Type: MULTI 2024-08-08T20:41:15.1948344Z Node: 3 2024-08-08T20:41:15.1948963Z Device Type: GPU 2024-08-08T20:41:15.1949544Z Cache Info: 2024-08-08T20:41:15.1950035Z L1: 16(0x10) KB 2024-08-08T20:41:15.1950616Z L2: 8192(0x2000) KB 2024-08-08T20:41:15.1951202Z Chip ID: 29711(0x740f) 2024-08-08T20:41:15.1951848Z ASIC Revision: 1(0x1) 2024-08-08T20:41:15.1952525Z Cacheline Size: 64(0x40) 2024-08-08T20:41:15.1953193Z Max Clock Freq. (MHz): 1700 2024-08-08T20:41:15.1953816Z BDFID: 33536 2024-08-08T20:41:15.1954451Z Internal Node ID: 3 2024-08-08T20:41:15.1955110Z Compute Unit: 104 2024-08-08T20:41:15.1955780Z SIMDs per CU: 4 2024-08-08T20:41:15.1956549Z Shader Engines: 8 2024-08-08T20:41:15.1957353Z Shader Arrs. per Eng.: 1 2024-08-08T20:41:15.1958183Z WatchPts on Addr. Ranges:4 2024-08-08T20:41:15.1958947Z Coherent Host Access: FALSE 2024-08-08T20:41:15.1959839Z Features: KERNEL_DISPATCH 2024-08-08T20:41:15.1960477Z Fast F16 Operation: TRUE 2024-08-08T20:41:15.1961159Z Wavefront Size: 64(0x40) 2024-08-08T20:41:15.1961844Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:41:15.1962472Z Workgroup Max Size per Dimension: 2024-08-08T20:41:15.1963090Z x 1024(0x400) 2024-08-08T20:41:15.1963651Z y 1024(0x400) 2024-08-08T20:41:15.1964197Z z 1024(0x400) 2024-08-08T20:41:15.1964905Z Max Waves Per CU: 32(0x20) 2024-08-08T20:41:15.1965588Z Max Work-item Per CU: 2048(0x800) 2024-08-08T20:41:15.1966263Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:41:15.1966868Z Grid Max Size per Dimension: 2024-08-08T20:41:15.1967380Z x 4294967295(0xffffffff) 2024-08-08T20:41:15.1967931Z y 4294967295(0xffffffff) 2024-08-08T20:41:15.1968491Z z 4294967295(0xffffffff) 2024-08-08T20:41:15.1969137Z Max fbarriers/Workgrp: 32 2024-08-08T20:41:15.1969859Z Packet Processor uCode:: 82 2024-08-08T20:41:15.1970563Z SDMA engine uCode:: 8 2024-08-08T20:41:15.1971253Z IOMMU Support:: None 2024-08-08T20:41:15.1971835Z Pool Info: 2024-08-08T20:41:15.1972274Z Pool 1 2024-08-08T20:41:15.1972850Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:41:15.1973519Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:41:15.1974179Z Allocatable: TRUE 2024-08-08T20:41:15.1974865Z Alloc Granule: 4KB 2024-08-08T20:41:15.1975611Z Alloc Recommended Granule:2048KB 2024-08-08T20:41:15.1976468Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1977292Z Accessible by all: FALSE 2024-08-08T20:41:15.1977995Z Pool 2 2024-08-08T20:41:15.1978639Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-08T20:41:15.1979804Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:41:15.1980505Z Allocatable: TRUE 2024-08-08T20:41:15.1980925Z Alloc Granule: 4KB 2024-08-08T20:41:15.1981357Z Alloc Recommended Granule:2048KB 2024-08-08T20:41:15.1981744Z Alloc Alignment: 4KB 2024-08-08T20:41:15.1982106Z Accessible by all: FALSE 2024-08-08T20:41:15.1982419Z Pool 3 2024-08-08T20:41:15.1982687Z Segment: GROUP 2024-08-08T20:41:15.1983008Z Size: 64(0x40) KB 2024-08-08T20:41:15.1983340Z Allocatable: FALSE 2024-08-08T20:41:15.1983694Z Alloc Granule: 0KB 2024-08-08T20:41:15.1984063Z Alloc Recommended Granule:0KB 2024-08-08T20:41:15.1984440Z Alloc Alignment: 0KB 2024-08-08T20:41:15.1984802Z Accessible by all: FALSE 2024-08-08T20:41:15.1985109Z ISA Info: 2024-08-08T20:41:15.1985342Z ISA 1 2024-08-08T20:41:15.1985789Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-08T20:41:15.1986176Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-08T20:41:15.1986560Z Profiles: HSA_PROFILE_BASE 2024-08-08T20:41:15.1986990Z Default Rounding Mode: NEAR 2024-08-08T20:41:15.1987763Z Default Rounding Mode: NEAR 2024-08-08T20:41:15.1988565Z Fast f16: TRUE 2024-08-08T20:41:15.1989373Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:41:15.1990012Z Workgroup Max Size per Dimension: 2024-08-08T20:41:15.1990571Z x 1024(0x400) 2024-08-08T20:41:15.1991135Z y 1024(0x400) 2024-08-08T20:41:15.1991678Z z 1024(0x400) 2024-08-08T20:41:15.1992300Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:41:15.1992908Z Grid Max Size per Dimension: 2024-08-08T20:41:15.1993416Z x 4294967295(0xffffffff) 2024-08-08T20:41:15.1993978Z y 4294967295(0xffffffff) 2024-08-08T20:41:15.1994545Z z 4294967295(0xffffffff) 2024-08-08T20:41:15.1995170Z FBarrier Max Size: 32 2024-08-08T20:41:15.1995794Z *** Done *** 2024-08-08T20:41:15.2076815Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-08-08T20:41:15.2077180Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-08-08T20:41:15.2077726Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-08T20:41:15.2078249Z if [[ $ngpu -eq 0 ]]; then 2024-08-08T20:41:15.2078548Z  echo "Error: Failed to detect any GPUs on the runner" 2024-08-08T20:41:15.2078820Z  echo "$msg" 2024-08-08T20:41:15.2079003Z  exit 1 2024-08-08T20:41:15.2079174Z fi 2024-08-08T20:41:15.2079344Z if [[ $ngpu -eq 1 ]]; then 2024-08-08T20:41:15.2079677Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-08-08T20:41:15.2080018Z  echo "$msg" 2024-08-08T20:41:15.2080201Z  exit 1 2024-08-08T20:41:15.2080584Z fi 2024-08-08T20:41:15.2096759Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.2097039Z env: 2024-08-08T20:41:15.2097199Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.2097429Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.2097675Z ##[endgroup] 2024-08-08T20:41:15.3097241Z Prepare all required actions 2024-08-08T20:41:15.3138612Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-08-08T20:41:15.3139159Z with: 2024-08-08T20:41:15.3139496Z diskspace-cutoff: 70 2024-08-08T20:41:15.3139876Z env: 2024-08-08T20:41:15.3140203Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.3140675Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.3141188Z ##[endgroup] 2024-08-08T20:41:15.3192910Z ##[group]Run set -ex 2024-08-08T20:41:15.3193282Z set -ex 2024-08-08T20:41:15.3193605Z diskspace_cutoff=70 2024-08-08T20:41:15.3194098Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-08-08T20:41:15.3194906Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-08T20:41:15.3196078Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-08T20:41:15.3197085Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-08-08T20:41:15.3197583Z  docker system prune -af 2024-08-08T20:41:15.3198536Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-08T20:41:15.3199298Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-08-08T20:41:15.3200080Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-08-08T20:41:15.3200760Z  echo "$msg" 2024-08-08T20:41:15.3201106Z  exit 1 2024-08-08T20:41:15.3201424Z  else 2024-08-08T20:41:15.3201802Z  difference=$((diskspace - diskspace_new)) 2024-08-08T20:41:15.3202334Z  echo "Diskspace saved: $difference percent" 2024-08-08T20:41:15.3202774Z  fi 2024-08-08T20:41:15.3203056Z fi 2024-08-08T20:41:15.3227869Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.3228482Z env: 2024-08-08T20:41:15.3228812Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.3229303Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.3229824Z ##[endgroup] 2024-08-08T20:41:15.3282999Z + diskspace_cutoff=70 2024-08-08T20:41:15.3290564Z ++ docker info -f '{{.DockerRootDir}}' 2024-08-08T20:41:15.3777691Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-08-08T20:41:15.3783782Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-08-08T20:41:15.3784394Z ++ sed -n 2p 2024-08-08T20:41:15.3785374Z ++ sed s/%// 2024-08-08T20:41:15.3786953Z ++ sed 's/ //' 2024-08-08T20:41:15.3801192Z + diskspace=27 2024-08-08T20:41:15.3801650Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-08-08T20:41:15.3802138Z + [[ 27 -ge 70 ]] 2024-08-08T20:41:15.3836567Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-08T20:41:15.3837399Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-08T20:41:15.3838091Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-08T20:41:15.3862551Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.3862894Z env: 2024-08-08T20:41:15.3863099Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.3863371Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.3863672Z ##[endgroup] 2024-08-08T20:41:15.3960813Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-08-08T20:41:15.3961338Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-08-08T20:41:15.3962106Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-08-08T20:41:15.3977453Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:15.3977735Z env: 2024-08-08T20:41:15.3977897Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.3978119Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.3978391Z ##[endgroup] 2024-08-08T20:41:15.4074661Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-08-08T20:41:15.4075007Z with: 2024-08-08T20:41:15.4075341Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-08-08T20:41:15.4075813Z aws-region: us-east-1 2024-08-08T20:41:15.4076219Z role-duration-seconds: 18000 2024-08-08T20:41:15.4076800Z audience: sts.amazonaws.com 2024-08-08T20:41:15.4077315Z env: 2024-08-08T20:41:15.4077709Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:15.4078320Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:15.4079228Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:15.4080061Z ##[endgroup] 2024-08-08T20:41:15.8519482Z Assuming role with OIDC 2024-08-08T20:41:16.1888665Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-08-08T20:41:16.3046702Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-08-08T20:41:16.3047359Z with: 2024-08-08T20:41:16.3047779Z mask-password: true 2024-08-08T20:41:16.3048266Z registry-type: private 2024-08-08T20:41:16.3048742Z skip-logout: false 2024-08-08T20:41:16.3049167Z env: 2024-08-08T20:41:16.3049557Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:16.3050119Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:16.3051136Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:16.3052316Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:16.3052987Z AWS_REGION: us-east-1 2024-08-08T20:41:16.3053648Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:16.3054490Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:16.3065948Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:16.3066520Z ##[endgroup] 2024-08-08T20:41:16.7862011Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5088267Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-08T20:41:17.5089187Z with: 2024-08-08T20:41:17.5090398Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5091748Z docker-build-dir: .ci/docker 2024-08-08T20:41:17.5092294Z working-directory: . 2024-08-08T20:41:17.5092936Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5093682Z force-push: false 2024-08-08T20:41:17.5094116Z env: 2024-08-08T20:41:17.5094517Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:17.5095114Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:17.5096142Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:17.5097082Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:17.5097598Z AWS_REGION: us-east-1 2024-08-08T20:41:17.5098381Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:17.5099102Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:17.5110559Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:17.5111043Z ##[endgroup] 2024-08-08T20:41:17.5145429Z ##[group]Run set -ex 2024-08-08T20:41:17.5145978Z set -ex 2024-08-08T20:41:17.5146398Z  2024-08-08T20:41:17.5147114Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-08T20:41:17.5148317Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-08T20:41:17.5149318Z # job could then download the pre-built image as usual 2024-08-08T20:41:17.5150676Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-08T20:41:17.5151504Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5152312Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5153035Z  2024-08-08T20:41:17.5153676Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-08T20:41:17.5154455Z  exit 0 2024-08-08T20:41:17.5154866Z else 2024-08-08T20:41:17.5155354Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5155936Z fi 2024-08-08T20:41:17.5156326Z  2024-08-08T20:41:17.5156932Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-08T20:41:17.5157975Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-08T20:41:17.5158901Z  # use it as it is, but first let's extract the tag 2024-08-08T20:41:17.5159775Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-08T20:41:17.5160672Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5161526Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5162253Z else 2024-08-08T20:41:17.5162820Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-08T20:41:17.5163633Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5164886Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5165863Z fi 2024-08-08T20:41:17.5200481Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:17.5201190Z env: 2024-08-08T20:41:17.5201596Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:17.5202179Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:17.5203220Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:17.5204183Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:17.5204700Z AWS_REGION: us-east-1 2024-08-08T20:41:17.5205424Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:17.5206142Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:17.5217941Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:17.5218465Z REPO_NAME: pytorch 2024-08-08T20:41:17.5219692Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5221001Z DOCKER_BUILD_DIR: .ci/docker 2024-08-08T20:41:17.5221671Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5222369Z ##[endgroup] 2024-08-08T20:41:17.5289459Z + [[ ! -d .ci/docker ]] 2024-08-08T20:41:17.5290005Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-08T20:41:17.5290532Z + echo skip=false 2024-08-08T20:41:17.5292198Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-08T20:41:17.5306042Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5307371Z ++ awk -F '[:,]' '{print $2}' 2024-08-08T20:41:17.5342690Z + DOCKER_TAG=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5343529Z + echo docker-tag=02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5345086Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5404421Z ##[group]Run set +e 2024-08-08T20:41:17.5405095Z set +e 2024-08-08T20:41:17.5405526Z set -x 2024-08-08T20:41:17.5405931Z  2024-08-08T20:41:17.5406317Z login() { 2024-08-08T20:41:17.5407540Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-08T20:41:17.5408500Z } 2024-08-08T20:41:17.5408887Z  2024-08-08T20:41:17.5409272Z retry () { 2024-08-08T20:41:17.5409792Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-08T20:41:17.5410375Z } 2024-08-08T20:41:17.5410765Z  2024-08-08T20:41:17.5411196Z retry login "${DOCKER_REGISTRY}" 2024-08-08T20:41:17.5411753Z  2024-08-08T20:41:17.5412344Z # Check if image already exists, if it does then skip building it 2024-08-08T20:41:17.5413223Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-08T20:41:17.5413878Z  exit 0 2024-08-08T20:41:17.5414289Z fi 2024-08-08T20:41:17.5414676Z  2024-08-08T20:41:17.5415420Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-08T20:41:17.5416622Z # be empty. The default action would be to continue rebuild the image 2024-08-08T20:41:17.5417711Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-08T20:41:17.5418666Z  # if we're on the base branch then use the parent commit 2024-08-08T20:41:17.5419529Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-08T20:41:17.5420205Z else 2024-08-08T20:41:17.5420889Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-08T20:41:17.5421712Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-08T20:41:17.5422305Z fi 2024-08-08T20:41:17.5422664Z  2024-08-08T20:41:17.5423061Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-08T20:41:17.5423654Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5424210Z  2024-08-08T20:41:17.5424978Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-08T20:41:17.5425889Z  exit 0 2024-08-08T20:41:17.5426288Z fi 2024-08-08T20:41:17.5426710Z  2024-08-08T20:41:17.5427378Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-08T20:41:17.5429189Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-08T20:41:17.5430227Z  exit 1 2024-08-08T20:41:17.5430634Z fi 2024-08-08T20:41:17.5431017Z  2024-08-08T20:41:17.5431666Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-08T20:41:17.5432814Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-08T20:41:17.5433838Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-08T20:41:17.5435022Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-08T20:41:17.5436341Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-08T20:41:17.5437132Z fi 2024-08-08T20:41:17.5437512Z  2024-08-08T20:41:17.5437981Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:17.5470497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:17.5471169Z env: 2024-08-08T20:41:17.5471552Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:17.5472094Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:17.5473041Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:17.5473926Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:17.5474406Z AWS_REGION: us-east-1 2024-08-08T20:41:17.5474953Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:17.5475615Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:17.5487041Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:17.5487538Z DOCKER_BUILD_DIR: .ci/docker 2024-08-08T20:41:17.5488516Z BASE_REVISION: b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:41:17.5489901Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5491257Z DOCKER_TAG: 02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:17.5492070Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5492770Z ##[endgroup] 2024-08-08T20:41:17.5548241Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5548865Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:17.5551272Z + aws ecr get-login-password --region us-east-1 2024-08-08T20:41:17.5555211Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:18.9586547Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-08-08T20:41:18.9587699Z Configure a credential helper to remove this warning. See 2024-08-08T20:41:18.9588754Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-08T20:41:18.9589498Z 2024-08-08T20:41:18.9590515Z Login Succeeded 2024-08-08T20:41:18.9621768Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:20.4017181Z { 2024-08-08T20:41:20.4017843Z "schemaVersion": 2, 2024-08-08T20:41:20.4018733Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-08T20:41:20.4019634Z "config": { 2024-08-08T20:41:20.4020294Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-08T20:41:20.4021042Z "size": 23410, 2024-08-08T20:41:20.4021863Z "digest": "sha256:00a70f0483c9ebd97217dd6cf435a0b76d1aa08cf57f57ef3ea0397a5342f0e6" 2024-08-08T20:41:20.4022712Z }, 2024-08-08T20:41:20.4023088Z "layers": [ 2024-08-08T20:41:20.4023481Z { 2024-08-08T20:41:20.4024101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4024924Z "size": 28584223, 2024-08-08T20:41:20.4025880Z "digest": "sha256:560c024910bebac6b404791af28ebd48a8289303b8377d17b67ffdfe52754f2a" 2024-08-08T20:41:20.4026866Z }, 2024-08-08T20:41:20.4027382Z { 2024-08-08T20:41:20.4028940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4029846Z "size": 1822, 2024-08-08T20:41:20.4030734Z "digest": "sha256:a2fe8a18b8204dd2ac1ad97839cabb5c281167ddb5d967b73904da6f26cfa744" 2024-08-08T20:41:20.4031626Z }, 2024-08-08T20:41:20.4031983Z { 2024-08-08T20:41:20.4032568Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4033300Z "size": 312563589, 2024-08-08T20:41:20.4034077Z "digest": "sha256:e9ca0410e6cc4e1df85ef1faddc3574fcd62d9ef5d4f4cd9a69d24d14eae47da" 2024-08-08T20:41:20.4034925Z }, 2024-08-08T20:41:20.4035288Z { 2024-08-08T20:41:20.4035870Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4036616Z "size": 804, 2024-08-08T20:41:20.4037356Z "digest": "sha256:b9f09a29a03fdaddd7a3adc9d6603a99682b41017349a4041774e21f678ff0a8" 2024-08-08T20:41:20.4037886Z }, 2024-08-08T20:41:20.4038075Z { 2024-08-08T20:41:20.4038393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4038799Z "size": 106, 2024-08-08T20:41:20.4039198Z "digest": "sha256:6aedf0bc84e4d193a32995123f9acb43813ac8b02cc948c5679d6783b7af7f57" 2024-08-08T20:41:20.4039641Z }, 2024-08-08T20:41:20.4039834Z { 2024-08-08T20:41:20.4040149Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4040536Z "size": 545, 2024-08-08T20:41:20.4040919Z "digest": "sha256:8388915ed71499a7fb259ee3c910eda0eaed8650f24bc049f2aeac4b88c2350a" 2024-08-08T20:41:20.4041352Z }, 2024-08-08T20:41:20.4041544Z { 2024-08-08T20:41:20.4041850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4042227Z "size": 1280, 2024-08-08T20:41:20.4042868Z "digest": "sha256:2366d00cb0387a87ac06d3eaaa44c714c4adbac24c86968b71dfa04c26f866b1" 2024-08-08T20:41:20.4043312Z }, 2024-08-08T20:41:20.4043504Z { 2024-08-08T20:41:20.4043816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4044203Z "size": 3230, 2024-08-08T20:41:20.4044588Z "digest": "sha256:538f0f042001f7e21d033df6ab93bd29d0e43a95c6bace8269319c31a132cb7f" 2024-08-08T20:41:20.4045174Z }, 2024-08-08T20:41:20.4045403Z { 2024-08-08T20:41:20.4045965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4046850Z "size": 1909, 2024-08-08T20:41:20.4047701Z "digest": "sha256:1e0d9b1ca68f49765014b0ad847a7141faebc074d0f1ee34260693ac87a99208" 2024-08-08T20:41:20.4048571Z }, 2024-08-08T20:41:20.4048926Z { 2024-08-08T20:41:20.4049498Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4050227Z "size": 701, 2024-08-08T20:41:20.4050974Z "digest": "sha256:1d58a2e52f2e7b08b5d5aeb353bda7df6950249e4aa959623646afaff9e294cc" 2024-08-08T20:41:20.4051806Z }, 2024-08-08T20:41:20.4052159Z { 2024-08-08T20:41:20.4052726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4053465Z "size": 2483281422, 2024-08-08T20:41:20.4054243Z "digest": "sha256:b459913b5cb20a5b3ab827e61475d84fc84d5eadaf85a482732928be75796df4" 2024-08-08T20:41:20.4055105Z }, 2024-08-08T20:41:20.4055529Z { 2024-08-08T20:41:20.4056211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4057079Z "size": 381, 2024-08-08T20:41:20.4057922Z "digest": "sha256:ecf45da46ed638573f16baaf2467c12d8245672467505fdff66837276e327395" 2024-08-08T20:41:20.4058886Z }, 2024-08-08T20:41:20.4059302Z { 2024-08-08T20:41:20.4059976Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4060642Z "size": 12119, 2024-08-08T20:41:20.4061081Z "digest": "sha256:0901b3eb762929348ec8ffc0a29fe108c39fad603f0261ef597683223792df06" 2024-08-08T20:41:20.4061518Z }, 2024-08-08T20:41:20.4061702Z { 2024-08-08T20:41:20.4062014Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4062392Z "size": 504, 2024-08-08T20:41:20.4062922Z "digest": "sha256:93a4b311ea200f9c341f502bfbddd9996132bdb955d875a9e6122827de6e024d" 2024-08-08T20:41:20.4063351Z }, 2024-08-08T20:41:20.4063646Z { 2024-08-08T20:41:20.4064006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4064406Z "size": 121477679, 2024-08-08T20:41:20.4064808Z "digest": "sha256:d4768f7775789193cf0a696bdffb4f58850e65044e51063e087ce838150470bb" 2024-08-08T20:41:20.4065247Z }, 2024-08-08T20:41:20.4065434Z { 2024-08-08T20:41:20.4065745Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4066160Z "size": 109, 2024-08-08T20:41:20.4066543Z "digest": "sha256:807fcf62d49420d5492d51ed4f05b5f75c996ff7ba843f2dba8dff57dde944ec" 2024-08-08T20:41:20.4066980Z }, 2024-08-08T20:41:20.4067172Z { 2024-08-08T20:41:20.4067471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4067845Z "size": 488, 2024-08-08T20:41:20.4068223Z "digest": "sha256:90e1e9edf7c2ef3b3c59507bc115b77772ca93ef8b1cdd9c72222b20c46f12ca" 2024-08-08T20:41:20.4068671Z }, 2024-08-08T20:41:20.4068853Z { 2024-08-08T20:41:20.4069162Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4069543Z "size": 294, 2024-08-08T20:41:20.4069923Z "digest": "sha256:fa5c0c43d4013ef819e5ada0925deb93d7d5eb2ceb02ee651c8727e04f44d09f" 2024-08-08T20:41:20.4070359Z }, 2024-08-08T20:41:20.4070561Z { 2024-08-08T20:41:20.4070862Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4071239Z "size": 103, 2024-08-08T20:41:20.4071609Z "digest": "sha256:14e1db5417fa873bebd53e6ea1692c100e415d23804a459d820f010c49a82bb5" 2024-08-08T20:41:20.4072033Z }, 2024-08-08T20:41:20.4072393Z { 2024-08-08T20:41:20.4072690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4073068Z "size": 1474, 2024-08-08T20:41:20.4073448Z "digest": "sha256:d653fc11a6f84a2f2d2290b8b1d646acaa4e0d4f8790e668d1ba60eb334712d9" 2024-08-08T20:41:20.4073882Z }, 2024-08-08T20:41:20.4074086Z { 2024-08-08T20:41:20.4074388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4074770Z "size": 611424883, 2024-08-08T20:41:20.4075161Z "digest": "sha256:9ee8ea11f5c9c4f4b054cf9868c2c06fd511929037f40b4aceaeae0be7871ccf" 2024-08-08T20:41:20.4075589Z }, 2024-08-08T20:41:20.4075782Z { 2024-08-08T20:41:20.4076094Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4076482Z "size": 160, 2024-08-08T20:41:20.4076865Z "digest": "sha256:135e96d81df42d8322b4f12aa8ddc45cfd13ed4959e9256357f76e7116888ce4" 2024-08-08T20:41:20.4077294Z }, 2024-08-08T20:41:20.4077485Z { 2024-08-08T20:41:20.4077801Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4078181Z "size": 1640, 2024-08-08T20:41:20.4078561Z "digest": "sha256:38730fc7b31d55b1b05d15812ddd639e2f5c1c4da80db3f42ad52dc5a9734b4e" 2024-08-08T20:41:20.4078990Z }, 2024-08-08T20:41:20.4079176Z { 2024-08-08T20:41:20.4079481Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4079862Z "size": 6892304620, 2024-08-08T20:41:20.4080244Z "digest": "sha256:13b48384977e39c16d7e546175ec4f85dbae84df45917c780e0d07c482176c1d" 2024-08-08T20:41:20.4080659Z }, 2024-08-08T20:41:20.4080846Z { 2024-08-08T20:41:20.4081146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4081523Z "size": 105, 2024-08-08T20:41:20.4081901Z "digest": "sha256:89d9a967b2f2d67e91d486ded7b5bac4d0772193ead80dd76e6a04bea2a74005" 2024-08-08T20:41:20.4082328Z }, 2024-08-08T20:41:20.4082515Z { 2024-08-08T20:41:20.4082811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4083201Z "size": 974, 2024-08-08T20:41:20.4083577Z "digest": "sha256:e7e645edc562d86d8239a2d0a83015825898cca1c6fb68b32a63789735da0aba" 2024-08-08T20:41:20.4084014Z }, 2024-08-08T20:41:20.4084209Z { 2024-08-08T20:41:20.4084648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4085113Z "size": 320966958, 2024-08-08T20:41:20.4085569Z "digest": "sha256:19e97ecd13078ae1b6999a8084d257312dcc043804caa666aa30c18c3bbb5e5b" 2024-08-08T20:41:20.4086288Z }, 2024-08-08T20:41:20.4086739Z { 2024-08-08T20:41:20.4087422Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4088284Z "size": 111, 2024-08-08T20:41:20.4089030Z "digest": "sha256:fa0997dd0bca80aa81d7f7e1010dfbc96419d8c938bb6dce55ef77659abd5eb9" 2024-08-08T20:41:20.4089939Z + exit 0 2024-08-08T20:41:20.4090313Z }, 2024-08-08T20:41:20.4090671Z { 2024-08-08T20:41:20.4091242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4091977Z "size": 166, 2024-08-08T20:41:20.4092686Z "digest": "sha256:813423a31ace73ea4d8f50b33df8940231e36f0b30655483e59f1d263d2508c9" 2024-08-08T20:41:20.4093492Z }, 2024-08-08T20:41:20.4093841Z { 2024-08-08T20:41:20.4094425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4095160Z "size": 3883361, 2024-08-08T20:41:20.4095875Z "digest": "sha256:7b110335634992dd5186d9b974d451fa243849ab801dacbe15a533442cf1ed81" 2024-08-08T20:41:20.4096858Z }, 2024-08-08T20:41:20.4097276Z { 2024-08-08T20:41:20.4097959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4098819Z "size": 107, 2024-08-08T20:41:20.4099670Z "digest": "sha256:e33e159301dca7e96de8acdb8a043e99626abf553540180ee7a7ea634311434a" 2024-08-08T20:41:20.4100645Z }, 2024-08-08T20:41:20.4101059Z { 2024-08-08T20:41:20.4101694Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4102332Z "size": 565, 2024-08-08T20:41:20.4102722Z "digest": "sha256:2fb52e871fb50a1df019d7b6dd58c4f54a3dfbc0cb32571654e4ce25e90b1206" 2024-08-08T20:41:20.4103157Z }, 2024-08-08T20:41:20.4103343Z { 2024-08-08T20:41:20.4103646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4104033Z "size": 43195784, 2024-08-08T20:41:20.4104430Z "digest": "sha256:303c4cd8be2816eb5403bea1bfc0e72b18e4ba5891b572bdbdc5eb8b008c2819" 2024-08-08T20:41:20.4104868Z }, 2024-08-08T20:41:20.4105055Z { 2024-08-08T20:41:20.4105356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4105739Z "size": 106, 2024-08-08T20:41:20.4106151Z "digest": "sha256:db3ad2049fae7347bb0a1fdeb9dcc20ad8879d3ef833b7ec8519e48ecf1d12da" 2024-08-08T20:41:20.4106587Z }, 2024-08-08T20:41:20.4106777Z { 2024-08-08T20:41:20.4107073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4107454Z "size": 295, 2024-08-08T20:41:20.4107894Z "digest": "sha256:79230c903f0b40cf457521f5bc45616a2147a12f07229e24565276e54c89c9f6" 2024-08-08T20:41:20.4108752Z }, 2024-08-08T20:41:20.4109167Z { 2024-08-08T20:41:20.4109850Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4110681Z "size": 88291, 2024-08-08T20:41:20.4111427Z "digest": "sha256:fb98230de6cd142e30295a00f2e752acb54c1f769a1759568d2a1c4470ff2928" 2024-08-08T20:41:20.4112241Z }, 2024-08-08T20:41:20.4112591Z { 2024-08-08T20:41:20.4113163Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4113889Z "size": 106, 2024-08-08T20:41:20.4114609Z "digest": "sha256:eb0fa0dad3822585a8914f653af60ff6ffc6e84b5d7371b4db0815081bdd6fcd" 2024-08-08T20:41:20.4115442Z }, 2024-08-08T20:41:20.4115797Z { 2024-08-08T20:41:20.4116371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4117091Z "size": 1212, 2024-08-08T20:41:20.4117813Z "digest": "sha256:be20df34c8440bb8e79de96ac448b757068bee3431010a051c1e1831b103846d" 2024-08-08T20:41:20.4118638Z }, 2024-08-08T20:41:20.4118985Z { 2024-08-08T20:41:20.4119557Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4120279Z "size": 701, 2024-08-08T20:41:20.4121297Z "digest": "sha256:1d58a2e52f2e7b08b5d5aeb353bda7df6950249e4aa959623646afaff9e294cc" 2024-08-08T20:41:20.4122138Z }, 2024-08-08T20:41:20.4122489Z { 2024-08-08T20:41:20.4123065Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4123793Z "size": 143, 2024-08-08T20:41:20.4124521Z "digest": "sha256:7c53731b1be3c9c6a7b328e2ec39cbcaa40abbd11f7a152f06ba1ac8e37ac509" 2024-08-08T20:41:20.4125483Z }, 2024-08-08T20:41:20.4125843Z { 2024-08-08T20:41:20.4126409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4127132Z "size": 120, 2024-08-08T20:41:20.4127832Z "digest": "sha256:8653067a26c3efda563955f9bb0980d5612bf24e86539518a9dfa7417d6306d2" 2024-08-08T20:41:20.4128652Z }, 2024-08-08T20:41:20.4129001Z { 2024-08-08T20:41:20.4129565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4130294Z "size": 1950349427, 2024-08-08T20:41:20.4131046Z "digest": "sha256:521e80f6d520a2d3b969e198c27169f27fca979aa048922c2db5ec659a9ca719" 2024-08-08T20:41:20.4131852Z }, 2024-08-08T20:41:20.4132202Z { 2024-08-08T20:41:20.4132768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4133494Z "size": 177, 2024-08-08T20:41:20.4134199Z "digest": "sha256:ed82d826adc8d492280f209268025e7a8d1daffd83a6772bdb4931649fb35cfe" 2024-08-08T20:41:20.4135087Z }, 2024-08-08T20:41:20.4135504Z { 2024-08-08T20:41:20.4136178Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4137030Z "size": 208, 2024-08-08T20:41:20.4137855Z "digest": "sha256:b600990021299db536ef976a85cb2fa93e74a0ccad41e73076ddebf48a22498e" 2024-08-08T20:41:20.4139232Z }, 2024-08-08T20:41:20.4139649Z { 2024-08-08T20:41:20.4140319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4140808Z "size": 701, 2024-08-08T20:41:20.4141208Z "digest": "sha256:1d58a2e52f2e7b08b5d5aeb353bda7df6950249e4aa959623646afaff9e294cc" 2024-08-08T20:41:20.4141644Z }, 2024-08-08T20:41:20.4141834Z { 2024-08-08T20:41:20.4142136Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4142514Z "size": 633, 2024-08-08T20:41:20.4142879Z "digest": "sha256:1cc626898268d254995ed432111cd4b983be24363d961a8be5ec6fbe847581f9" 2024-08-08T20:41:20.4143300Z }, 2024-08-08T20:41:20.4143490Z { 2024-08-08T20:41:20.4143787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4144170Z "size": 229176816, 2024-08-08T20:41:20.4144555Z "digest": "sha256:8a7320138b9341037e3ecb8a054134b9a46a8e3181a255f75e78a3712a61ac05" 2024-08-08T20:41:20.4144975Z }, 2024-08-08T20:41:20.4145177Z { 2024-08-08T20:41:20.4145482Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4145870Z "size": 1841, 2024-08-08T20:41:20.4146251Z "digest": "sha256:0e4d4d20539368402c749f20fb831589fad6876da0354f9e1e6a889a5b0fc07c" 2024-08-08T20:41:20.4146780Z }, 2024-08-08T20:41:20.4147165Z { 2024-08-08T20:41:20.4147859Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-08T20:41:20.4148716Z "size": 15983002, 2024-08-08T20:41:20.4149536Z "digest": "sha256:f9232b4b11bff14ea9e6f52773fab0f4de1cb92e1e895d28e3b39d7b10b5c500" 2024-08-08T20:41:20.4150360Z } 2024-08-08T20:41:20.4150709Z ] 2024-08-08T20:41:20.4151063Z } 2024-08-08T20:41:20.4294470Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-08T20:41:20.4295314Z with: 2024-08-08T20:41:20.4296485Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:20.4297988Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:20.4298694Z env: 2024-08-08T20:41:20.4299111Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:20.4299705Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:20.4300730Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:20.4301687Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:20.4302221Z AWS_REGION: us-east-1 2024-08-08T20:41:20.4302873Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:20.4303578Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:20.4314856Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:20.4315339Z ##[endgroup] 2024-08-08T20:41:20.4344831Z ##[group]Run set -x 2024-08-08T20:41:20.4345364Z set -x 2024-08-08T20:41:20.4345783Z set +e 2024-08-08T20:41:20.4346213Z  2024-08-08T20:41:20.4346618Z login() { 2024-08-08T20:41:20.4347485Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-08T20:41:20.4348417Z } 2024-08-08T20:41:20.4348813Z  2024-08-08T20:41:20.4349206Z retry () { 2024-08-08T20:41:20.4349729Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-08T20:41:20.4350326Z } 2024-08-08T20:41:20.4350711Z  2024-08-08T20:41:20.4351147Z retry login "${DOCKER_REGISTRY}" 2024-08-08T20:41:20.4351709Z  2024-08-08T20:41:20.4352098Z set -e 2024-08-08T20:41:20.4352722Z # ignore output since only exit code is used for conditional 2024-08-08T20:41:20.4353596Z # only pull docker image if it's not available locally 2024-08-08T20:41:20.4354575Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-08T20:41:20.4355467Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-08T20:41:20.4356055Z fi 2024-08-08T20:41:20.4391865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:20.4392906Z env: 2024-08-08T20:41:20.4393316Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:20.4393895Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:20.4394930Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:20.4395912Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:20.4396438Z AWS_REGION: us-east-1 2024-08-08T20:41:20.4397034Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:20.4397812Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:20.4409282Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:20.4410512Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:20.4411943Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:20.4412647Z ##[endgroup] 2024-08-08T20:41:20.4470489Z + set +e 2024-08-08T20:41:20.4471117Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:20.4471896Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:20.4478421Z + aws ecr get-login-password --region us-east-1 2024-08-08T20:41:20.4479839Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T20:41:21.8882861Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-08-08T20:41:21.8883998Z Configure a credential helper to remove this warning. See 2024-08-08T20:41:21.8885334Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-08T20:41:21.8886168Z 2024-08-08T20:41:21.8896249Z Login Succeeded 2024-08-08T20:41:21.8935912Z + set -e 2024-08-08T20:41:21.8938376Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:41:21.9173388Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-08T20:41:21.9174298Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-08T20:41:21.9175061Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-08T20:41:21.9175728Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-08T20:41:21.9202915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:41:21.9203463Z env: 2024-08-08T20:41:21.9203783Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:21.9204234Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:21.9205221Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:21.9205960Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:21.9206372Z AWS_REGION: us-east-1 2024-08-08T20:41:21.9206892Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:21.9207561Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:21.9217001Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:21.9217451Z ##[endgroup] 2024-08-08T20:41:22.1834556Z Defaulting to user installation because normal site-packages is not writeable 2024-08-08T20:41:22.2554932Z Requirement already satisfied: psutil==5.9.1 in /home/pytorchci/.local/lib/python3.10/site-packages (5.9.1) 2024-08-08T20:41:22.2559919Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/pytorchci/.local/lib/python3.10/site-packages (11.525.84) 2024-08-08T20:41:22.3978144Z Prepare all required actions 2024-08-08T20:41:22.3978573Z Getting action download info 2024-08-08T20:41:22.5940279Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-08T20:41:23.1990444Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-08T20:41:23.6471285Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-08T20:41:23.6471958Z with: 2024-08-08T20:41:23.6472396Z name: linux-focal-rocm6.1-py3.8 2024-08-08T20:41:23.6472958Z s3-bucket: gha-artifacts 2024-08-08T20:41:23.6473430Z env: 2024-08-08T20:41:23.6474290Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:23.6474838Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:23.6475842Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:23.6476776Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:23.6477360Z AWS_REGION: us-east-1 2024-08-08T20:41:23.6477993Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:23.6478685Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:23.6490144Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:23.6490619Z ##[endgroup] 2024-08-08T20:41:23.6562692Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-08T20:41:23.6563432Z with: 2024-08-08T20:41:23.6563991Z name: linux-focal-rocm6.1-py3.8 2024-08-08T20:41:23.6564683Z s3-bucket: gha-artifacts 2024-08-08T20:41:23.6565463Z region: us-east-1 2024-08-08T20:41:23.6565996Z env: 2024-08-08T20:41:23.6566464Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:41:23.6567143Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:41:23.6568390Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:41:23.6569452Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:41:23.6569986Z AWS_REGION: us-east-1 2024-08-08T20:41:23.6570566Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:41:23.6571294Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:41:23.6582518Z AWS_SESSION_TOKEN: *** 2024-08-08T20:41:23.6582988Z ##[endgroup] 2024-08-08T20:41:24.0576028Z (node:2898818) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-08T20:41:24.0576525Z 2024-08-08T20:41:24.0576717Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-08T20:41:24.0577205Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-08T20:41:24.0577738Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-08T20:41:24.3153061Z Found 1 objects with prefix pytorch/pytorch/10308807531/linux-focal-rocm6.1-py3.8/ 2024-08-08T20:41:24.3154445Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-08T20:42:23.3588972Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-08T20:42:23.3606349Z Artifact download has finished successfully 2024-08-08T20:42:23.4085894Z ##[group]Run unzip -o artifacts.zip 2024-08-08T20:42:23.4086248Z unzip -o artifacts.zip 2024-08-08T20:42:23.4105665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:23.4106085Z env: 2024-08-08T20:42:23.4106344Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:23.4106699Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:23.4107325Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:23.4107898Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:23.4108183Z AWS_REGION: us-east-1 2024-08-08T20:42:23.4108499Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:23.4108862Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:23.4114729Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:23.4114974Z ##[endgroup] 2024-08-08T20:42:23.4242136Z Archive: artifacts.zip 2024-08-08T20:42:23.4244275Z creating: dist/ 2024-08-08T20:42:24.8156379Z inflating: dist/torch-2.5.0a0+gitb9d86fa-cp38-cp38-linux_x86_64.whl 2024-08-08T20:42:24.8289798Z inflating: dist/.ninja_log 2024-08-08T20:42:24.8292309Z creating: build/custom_test_artifacts/ 2024-08-08T20:42:24.8292972Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-08T20:42:24.8293488Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-08T20:42:24.8294031Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-08T20:42:24.8294682Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-08T20:42:24.8295283Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-08T20:42:24.8295877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-08T20:42:24.8296755Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-08T20:42:24.8297259Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-08T20:42:24.8298104Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-08T20:42:24.8299543Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-08T20:42:24.8300054Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-08T20:42:24.8300560Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-08T20:42:24.8302402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-08T20:42:24.8303422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-08T20:42:24.8304844Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-08T20:42:24.8305458Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-08T20:42:24.8306829Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-08T20:42:24.8307527Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-08T20:42:24.8308070Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-08T20:42:24.8308524Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-08T20:42:24.8308990Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-08T20:42:24.8309472Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-08T20:42:24.8310013Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-08T20:42:24.8310609Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-08T20:42:24.8311192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-08T20:42:24.8311737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-08T20:42:24.8312297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-08T20:42:24.8312859Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-08T20:42:24.8313413Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-08T20:42:24.8313968Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-08T20:42:24.8314534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-08T20:42:24.8332594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-08T20:42:24.8463312Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-08T20:42:24.8463891Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-08T20:42:24.8464480Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-08T20:42:24.8465134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-08T20:42:24.8465794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-08T20:42:24.8466499Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-08T20:42:24.8467387Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-08T20:42:24.8468121Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-08T20:42:24.8468856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-08T20:42:24.8469593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-08T20:42:24.8470330Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-08T20:42:24.8486916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-08T20:42:24.8563126Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-08T20:42:24.8563961Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-08T20:42:24.8564712Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-08T20:42:24.8565451Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-08T20:42:24.8566052Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-08T20:42:24.8566628Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-08T20:42:24.8567211Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2024-08-08T20:42:24.8567768Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-08-08T20:42:24.8568298Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-08T20:42:24.8568822Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-08T20:42:24.8569335Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-08T20:42:24.8677015Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-08T20:42:24.8733593Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-08T20:42:24.8734120Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-08T20:42:24.8734576Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-08T20:42:24.8735103Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-08T20:42:24.8738141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-08T20:42:24.8738791Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-08T20:42:24.8739380Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-08T20:42:24.8740029Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-08T20:42:24.8740656Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-08T20:42:24.8741386Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-08T20:42:24.8742833Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-08T20:42:24.8743511Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-08T20:42:24.8744157Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-08T20:42:24.8745357Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-08T20:42:24.8746811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-08T20:42:24.8748078Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-08T20:42:24.8749014Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-08T20:42:24.8749969Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-08T20:42:24.8750734Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-08T20:42:24.8751360Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-08T20:42:24.8751901Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-08T20:42:24.8752469Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-08T20:42:24.8753066Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-08T20:42:24.8753759Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-08T20:42:24.8754519Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-08T20:42:24.8755259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-08T20:42:24.8755949Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-08T20:42:24.8756659Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-08T20:42:24.8757379Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-08T20:42:24.8758087Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-08T20:42:24.8758791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-08T20:42:24.8759491Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-08T20:42:24.8775901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-08T20:42:24.8834069Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-08T20:42:24.8834916Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-08T20:42:24.8835652Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-08T20:42:24.8836296Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-08T20:42:24.8836888Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-08T20:42:24.8837449Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-08T20:42:24.8838022Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2024-08-08T20:42:24.8838560Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-08-08T20:42:24.8839081Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-08T20:42:24.8839735Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-08T20:42:24.8840554Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-08T20:42:24.8884500Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-08T20:42:24.8885048Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-08T20:42:24.8885531Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-08T20:42:24.8886135Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-08T20:42:24.8889162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-08T20:42:24.8889806Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-08T20:42:24.8890441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-08T20:42:24.8891318Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-08T20:42:24.8891991Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-08T20:42:24.8892815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-08T20:42:24.8893930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-08T20:42:24.8894616Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-08T20:42:24.8895290Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-08T20:42:24.8896755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-08T20:42:24.8898002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-08T20:42:24.8899502Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-08T20:42:24.8900287Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-08T20:42:24.8901346Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-08T20:42:24.8902327Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-08T20:42:24.8903098Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-08T20:42:24.8903707Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-08T20:42:24.8904342Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-08T20:42:24.8905023Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-08T20:42:24.8905768Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-08T20:42:24.8906609Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-08T20:42:24.8907417Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-08T20:42:24.8908158Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-08T20:42:24.8908924Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-08T20:42:24.8909696Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-08T20:42:24.8910469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-08T20:42:24.8911243Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-08T20:42:24.8912003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-08T20:42:24.8913037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-08T20:42:24.9026349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-08T20:42:24.9027184Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-08T20:42:24.9027967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-08T20:42:24.9028830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-08T20:42:24.9029918Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-08T20:42:24.9030712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-08T20:42:24.9031526Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-08T20:42:24.9032335Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-08T20:42:24.9033147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-08T20:42:24.9033962Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-08T20:42:24.9034766Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-08T20:42:24.9050596Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-08T20:42:24.9100940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-08T20:42:24.9101876Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-08T20:42:24.9102663Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-08T20:42:24.9103357Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-08T20:42:24.9104007Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-08T20:42:24.9104620Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-08T20:42:24.9105230Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2024-08-08T20:42:24.9105823Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-08-08T20:42:24.9106380Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-08T20:42:24.9106927Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-08T20:42:24.9107481Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-08T20:42:24.9200019Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-08T20:42:24.9238410Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-08T20:42:24.9238902Z creating: build/lib/ 2024-08-08T20:42:24.9318346Z inflating: build/lib/libprotobuf-lite.a 2024-08-08T20:42:24.9728738Z inflating: build/lib/libprotobuf.a 2024-08-08T20:42:25.0181339Z inflating: build/lib/libprotoc.a 2024-08-08T20:42:25.0189921Z inflating: build/lib/libpthreadpool.a 2024-08-08T20:42:25.0197176Z inflating: build/lib/libcpuinfo.a 2024-08-08T20:42:25.0204181Z inflating: build/lib/libcpuinfo_internals.a 2024-08-08T20:42:25.0204577Z inflating: build/lib/libclog.a 2024-08-08T20:42:25.0222172Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-08T20:42:25.0223858Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-08T20:42:25.0240030Z inflating: build/lib/libnnpack.a 2024-08-08T20:42:25.0298513Z inflating: build/lib/libgtest.a 2024-08-08T20:42:25.0298884Z inflating: build/lib/libgtest_main.a 2024-08-08T20:42:25.0315273Z inflating: build/lib/libgmock.a 2024-08-08T20:42:25.0315639Z inflating: build/lib/libgmock_main.a 2024-08-08T20:42:25.0382861Z inflating: build/lib/libbenchmark.a 2024-08-08T20:42:25.0383238Z inflating: build/lib/libbenchmark_main.a 2024-08-08T20:42:25.0547317Z inflating: build/lib/libXNNPACK.a 2024-08-08T20:42:25.0554036Z inflating: build/lib/libittnotify.a 2024-08-08T20:42:25.0610696Z inflating: build/lib/libasmjit.a 2024-08-08T20:42:25.1753467Z inflating: build/lib/libfbgemm.a 2024-08-08T20:42:25.1778254Z inflating: build/lib/libtensorpipe_uv.a 2024-08-08T20:42:25.2297359Z inflating: build/lib/libtensorpipe.a 2024-08-08T20:42:25.2297739Z inflating: build/lib/libfoxi_loader.a 2024-08-08T20:42:25.2399842Z inflating: build/lib/libgloo.a 2024-08-08T20:42:25.2755682Z inflating: build/lib/libgloo_hip.a 2024-08-08T20:42:25.2793903Z inflating: build/lib/libonnx_proto.a 2024-08-08T20:42:25.3421869Z inflating: build/lib/libonnx.a 2024-08-08T20:42:26.2652869Z inflating: build/lib/libdnnl.a 2024-08-08T20:42:26.2671098Z inflating: build/lib/libfmt.a 2024-08-08T20:42:26.2960271Z inflating: build/lib/libkineto.a 2024-08-08T20:42:26.3043797Z inflating: build/lib/libc10.so 2024-08-08T20:42:26.3044544Z inflating: build/lib/libtorch_global_deps.so 2024-08-08T20:42:26.3049123Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-08T20:42:26.3122329Z inflating: build/lib/libc10_hip.so 2024-08-08T20:42:28.7471352Z inflating: build/lib/libtorch_cpu.so 2024-08-08T20:42:28.7475554Z inflating: build/lib/libunbox_lib.a 2024-08-08T20:42:28.7479225Z inflating: build/lib/libshm.so 2024-08-08T20:42:30.7762269Z inflating: build/lib/libtorch_hip.so 2024-08-08T20:42:30.7763094Z inflating: build/lib/libtorch.so 2024-08-08T20:42:30.7781956Z inflating: build/lib/libjitbackend_test.so 2024-08-08T20:42:30.7804450Z inflating: build/lib/libbackend_with_compiler.so 2024-08-08T20:42:30.7868795Z inflating: build/lib/libtorchbind_test.so 2024-08-08T20:42:30.7892069Z inflating: build/lib/libaoti_custom_ops.so 2024-08-08T20:42:30.9656359Z inflating: build/lib/libtorch_python.so 2024-08-08T20:42:30.9686941Z inflating: build/lib/libnnapi_backend.so 2024-08-08T20:42:30.9687632Z creating: build/bin/ 2024-08-08T20:42:30.9688173Z creating: build/bin/CMakeFiles/ 2024-08-08T20:42:30.9688800Z inflating: build/bin/cmake_install.cmake 2024-08-08T20:42:30.9689568Z inflating: build/bin/CTestTestfile.cmake 2024-08-08T20:42:31.0084633Z inflating: build/bin/protoc-3.13.0.0 2024-08-08T20:42:31.0481177Z inflating: build/bin/protoc 2024-08-08T20:42:31.0525908Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-08T20:42:31.0571682Z inflating: build/bin/c10_DeviceGuard_test 2024-08-08T20:42:31.0616318Z inflating: build/bin/c10_Device_test 2024-08-08T20:42:31.0659895Z inflating: build/bin/c10_StreamGuard_test 2024-08-08T20:42:31.0707403Z inflating: build/bin/c10_Scalar_test 2024-08-08T20:42:31.0759933Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-08T20:42:31.0805344Z inflating: build/bin/c10_SymInt_test 2024-08-08T20:42:31.0853928Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-08T20:42:31.0903513Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-08T20:42:31.0953588Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-08T20:42:31.1016643Z inflating: build/bin/c10_cow_test 2024-08-08T20:42:31.1063790Z inflating: build/bin/c10_Bitset_test 2024-08-08T20:42:31.1107335Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-08T20:42:31.1151490Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-08T20:42:31.1196596Z inflating: build/bin/c10_Half_test 2024-08-08T20:42:31.1247356Z inflating: build/bin/c10_LeftRight_test 2024-08-08T20:42:31.1295606Z inflating: build/bin/c10_Metaprogramming_test 2024-08-08T20:42:31.1339989Z inflating: build/bin/c10_Synchronized_test 2024-08-08T20:42:31.1389524Z inflating: build/bin/c10_ThreadLocal_test 2024-08-08T20:42:31.1435735Z inflating: build/bin/c10_TypeIndex_test 2024-08-08T20:42:31.1481103Z inflating: build/bin/c10_TypeList_test 2024-08-08T20:42:31.1524562Z inflating: build/bin/c10_TypeTraits_test 2024-08-08T20:42:31.1570631Z inflating: build/bin/c10_accumulate_test 2024-08-08T20:42:31.1615476Z inflating: build/bin/c10_bit_cast_test 2024-08-08T20:42:31.1664760Z inflating: build/bin/c10_bfloat16_test 2024-08-08T20:42:31.1714967Z inflating: build/bin/c10_complex_math_test 2024-08-08T20:42:31.1761793Z inflating: build/bin/c10_exception_test 2024-08-08T20:42:31.1811369Z inflating: build/bin/c10_complex_test 2024-08-08T20:42:31.1856171Z inflating: build/bin/c10_flags_test 2024-08-08T20:42:31.1900631Z inflating: build/bin/c10_generic_math_test 2024-08-08T20:42:31.1945750Z inflating: build/bin/c10_irange_test 2024-08-08T20:42:31.1993305Z inflating: build/bin/c10_lazy_test 2024-08-08T20:42:31.2141553Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-08T20:42:31.2192326Z inflating: build/bin/c10_logging_test 2024-08-08T20:42:31.2259393Z inflating: build/bin/c10_optional_test 2024-08-08T20:42:31.2314944Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-08T20:42:31.2362740Z inflating: build/bin/c10_registry_test 2024-08-08T20:42:31.2408668Z inflating: build/bin/c10_ssize_test 2024-08-08T20:42:31.2454926Z inflating: build/bin/c10_string_util_test 2024-08-08T20:42:31.2591036Z inflating: build/bin/c10_small_vector_test 2024-08-08T20:42:31.2643173Z inflating: build/bin/c10_string_view_test 2024-08-08T20:42:31.2688044Z inflating: build/bin/c10_tempfile_test 2024-08-08T20:42:31.2731089Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-08T20:42:31.2780995Z inflating: build/bin/c10_typeid_test 2024-08-08T20:42:31.2824604Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-08-08T20:42:31.2868103Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-08-08T20:42:31.2911632Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-08-08T20:42:31.2954880Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-08-08T20:42:31.2998521Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-08T20:42:31.3041935Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-08T20:42:31.3085467Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-08-08T20:42:31.3128780Z inflating: build/bin/c10_hip_HIPTest 2024-08-08T20:42:31.3425865Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-08T20:42:31.3748509Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-08T20:42:31.4057914Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-08T20:42:31.4105653Z inflating: build/bin/test_edge_op_registration 2024-08-08T20:42:31.4152298Z inflating: build/bin/BackoffTest 2024-08-08T20:42:31.4199677Z inflating: build/bin/FileStoreTest 2024-08-08T20:42:31.4249271Z inflating: build/bin/TCPStoreTest 2024-08-08T20:42:31.4296765Z inflating: build/bin/HashStoreTest 2024-08-08T20:42:31.4355139Z inflating: build/bin/ProcessGroupGlooTest 2024-08-08T20:42:31.4356848Z inflating: build/bin/example_allreduce 2024-08-08T20:42:31.4361324Z inflating: build/bin/torch_shm_manager 2024-08-08T20:42:31.4414326Z inflating: build/bin/static_runtime_bench 2024-08-08T20:42:31.4640412Z inflating: build/bin/static_runtime_test 2024-08-08T20:42:31.4706178Z inflating: build/bin/Dict_test 2024-08-08T20:42:31.4752761Z inflating: build/bin/Dimname_test 2024-08-08T20:42:31.4810540Z inflating: build/bin/MaybeOwned_test 2024-08-08T20:42:31.4854531Z inflating: build/bin/hip_packedtensoraccessor_test 2024-08-08T20:42:31.4905156Z inflating: build/bin/NamedTensor_test 2024-08-08T20:42:31.4957951Z inflating: build/bin/apply_utils_test 2024-08-08T20:42:31.5010423Z inflating: build/bin/atest 2024-08-08T20:42:31.5066545Z inflating: build/bin/basic 2024-08-08T20:42:31.5115384Z inflating: build/bin/broadcast_test 2024-08-08T20:42:31.5160885Z inflating: build/bin/cpu_allocator_test 2024-08-08T20:42:31.5212869Z inflating: build/bin/cpu_generator_test 2024-08-08T20:42:31.5260410Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-08T20:42:31.5344173Z inflating: build/bin/cpu_rng_test 2024-08-08T20:42:31.5388745Z inflating: build/bin/dispatch_key_set_test 2024-08-08T20:42:31.5433501Z inflating: build/bin/dlconvertor_test 2024-08-08T20:42:31.5485450Z inflating: build/bin/extension_backend_test 2024-08-08T20:42:31.5534534Z inflating: build/bin/half_test 2024-08-08T20:42:31.5620082Z inflating: build/bin/ivalue_test 2024-08-08T20:42:31.5664300Z inflating: build/bin/lazy_tensor_test 2024-08-08T20:42:31.5712529Z inflating: build/bin/memory_format_test 2024-08-08T20:42:31.5760791Z inflating: build/bin/math_kernel_test 2024-08-08T20:42:31.5808136Z inflating: build/bin/memory_overlapping_test 2024-08-08T20:42:31.5855475Z inflating: build/bin/mobile_memory_cleanup 2024-08-08T20:42:31.5905186Z inflating: build/bin/native_test 2024-08-08T20:42:31.5950491Z inflating: build/bin/operator_name_test 2024-08-08T20:42:31.5996376Z inflating: build/bin/operators_test 2024-08-08T20:42:31.6047080Z inflating: build/bin/quantized_test 2024-08-08T20:42:31.6093220Z inflating: build/bin/packedtensoraccessor_test 2024-08-08T20:42:31.6152982Z inflating: build/bin/pow_test 2024-08-08T20:42:31.6197496Z inflating: build/bin/reduce_ops_test 2024-08-08T20:42:31.6243029Z inflating: build/bin/reportMemoryUsage_test 2024-08-08T20:42:31.6293352Z inflating: build/bin/scalar_tensor_test 2024-08-08T20:42:31.6339400Z inflating: build/bin/StorageUtils_test 2024-08-08T20:42:31.6391375Z inflating: build/bin/scalar_test 2024-08-08T20:42:31.6438187Z inflating: build/bin/stride_properties_test 2024-08-08T20:42:31.6508708Z inflating: build/bin/tensor_iterator_test 2024-08-08T20:42:31.6557090Z inflating: build/bin/test_parallel 2024-08-08T20:42:31.6606735Z inflating: build/bin/type_ptr_test 2024-08-08T20:42:31.6608517Z inflating: build/bin/thread_init_test 2024-08-08T20:42:31.6662557Z inflating: build/bin/type_test 2024-08-08T20:42:31.6709066Z inflating: build/bin/undefined_tensor_test 2024-08-08T20:42:31.6709868Z inflating: build/bin/verify_api_visibility 2024-08-08T20:42:31.6772311Z inflating: build/bin/legacy_vmap_test 2024-08-08T20:42:31.6818061Z inflating: build/bin/wrapdim_test 2024-08-08T20:42:31.6863909Z inflating: build/bin/weakref_test 2024-08-08T20:42:31.6959196Z inflating: build/bin/List_test 2024-08-08T20:42:31.7005281Z inflating: build/bin/xla_tensor_test 2024-08-08T20:42:31.7058771Z inflating: build/bin/IListRef_test 2024-08-08T20:42:31.7145313Z inflating: build/bin/kernel_function_test 2024-08-08T20:42:31.7254273Z inflating: build/bin/kernel_function_legacy_test 2024-08-08T20:42:31.7313576Z inflating: build/bin/KernelFunction_test 2024-08-08T20:42:31.7428469Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-08T20:42:31.7482802Z inflating: build/bin/kernel_stackbased_test 2024-08-08T20:42:31.7575841Z inflating: build/bin/kernel_lambda_test 2024-08-08T20:42:31.7621354Z inflating: build/bin/CppSignature_test 2024-08-08T20:42:31.7707711Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-08T20:42:31.7751068Z inflating: build/bin/op_allowlist_test 2024-08-08T20:42:31.7807852Z inflating: build/bin/inline_container_test 2024-08-08T20:42:31.7857478Z inflating: build/bin/backend_fallback_test 2024-08-08T20:42:31.8130688Z inflating: build/bin/op_registration_test 2024-08-08T20:42:31.8173744Z inflating: build/bin/hip_complex_math_test 2024-08-08T20:42:31.8220772Z inflating: build/bin/hip_apply_test 2024-08-08T20:42:31.8264498Z inflating: build/bin/hip_complex_test 2024-08-08T20:42:31.8308029Z inflating: build/bin/hip_distributions_test 2024-08-08T20:42:31.8353165Z inflating: build/bin/hip_dlconvertor_test 2024-08-08T20:42:31.8396575Z inflating: build/bin/hip_generator_test 2024-08-08T20:42:31.8440036Z inflating: build/bin/hip_half_test 2024-08-08T20:42:31.8483539Z inflating: build/bin/hip_integer_divider_test 2024-08-08T20:42:31.8527049Z inflating: build/bin/hip_optional_test 2024-08-08T20:42:31.8571122Z inflating: build/bin/hip_vectorized_test 2024-08-08T20:42:31.9089426Z inflating: build/bin/test_jit 2024-08-08T20:42:31.9102396Z inflating: build/bin/tutorial_tensorexpr 2024-08-08T20:42:31.9810175Z inflating: build/bin/test_tensorexpr 2024-08-08T20:42:31.9859115Z inflating: build/bin/test_dist_autograd 2024-08-08T20:42:31.9920356Z inflating: build/bin/test_cpp_rpc 2024-08-08T20:42:31.9921867Z inflating: build/bin/parallel_benchmark 2024-08-08T20:42:31.9931148Z inflating: build/bin/aot_model_compiler_test 2024-08-08T20:42:32.0986099Z inflating: build/bin/test_api 2024-08-08T20:42:32.1046242Z inflating: build/bin/test_mobile_nnc 2024-08-08T20:42:32.1356737Z inflating: build/bin/test_lazy 2024-08-08T20:42:32.1357423Z creating: .additional_ci_files/ 2024-08-08T20:42:32.1411443Z inflating: .additional_ci_files/test-times.json 2024-08-08T20:42:32.1626136Z inflating: .additional_ci_files/test-class-times.json 2024-08-08T20:42:32.1667357Z ##[group]Run rm artifacts.zip 2024-08-08T20:42:32.1667674Z rm artifacts.zip 2024-08-08T20:42:32.1686711Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:32.1687083Z env: 2024-08-08T20:42:32.1687298Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:32.1687628Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:32.1688149Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:32.1688636Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:32.1688906Z AWS_REGION: us-east-1 2024-08-08T20:42:32.1689229Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:32.1689593Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:32.1695333Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:32.1695582Z ##[endgroup] 2024-08-08T20:42:32.2808522Z ##[group]Run df -H 2024-08-08T20:42:32.2809010Z df -H 2024-08-08T20:42:32.2845814Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:32.2846550Z env: 2024-08-08T20:42:32.2846971Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:32.2847560Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:32.2848594Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:32.2849567Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:32.2850093Z AWS_REGION: us-east-1 2024-08-08T20:42:32.2850702Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:32.2851397Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:32.2863652Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:32.2864140Z ##[endgroup] 2024-08-08T20:42:32.2943574Z Filesystem Size Used Avail Use% Mounted on 2024-08-08T20:42:32.2944389Z tmpfs 14G 17M 14G 1% /run 2024-08-08T20:42:32.2945195Z /dev/mapper/ubuntu--vg-ubuntu--lv 1.9T 473G 1.4T 27% / 2024-08-08T20:42:32.2946193Z tmpfs 68G 8.2k 68G 1% /dev/shm 2024-08-08T20:42:32.2947064Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-08-08T20:42:32.2947791Z /dev/sda2 2.1G 306M 1.7G 16% /boot 2024-08-08T20:42:32.2948548Z /dev/sda1 1.2G 6.4M 1.2G 1% /boot/efi 2024-08-08T20:42:32.2949666Z tmpfs 14G 17k 14G 1% /run/user/1001 2024-08-08T20:42:32.3010174Z Prepare all required actions 2024-08-08T20:42:32.3010903Z Getting action download info 2024-08-08T20:42:32.8529012Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-08T20:42:32.8529496Z with: 2024-08-08T20:42:32.8529796Z env: 2024-08-08T20:42:32.8530106Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:32.8530542Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:32.8531299Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:32.8532017Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:32.8532419Z AWS_REGION: us-east-1 2024-08-08T20:42:32.8532872Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:32.8533722Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:32.8542179Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:32.8542548Z ##[endgroup] 2024-08-08T20:42:32.8585249Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-08T20:42:32.8585719Z with: 2024-08-08T20:42:32.8586020Z name: td_results 2024-08-08T20:42:32.8586404Z s3-bucket: gha-artifacts 2024-08-08T20:42:32.8586778Z region: us-east-1 2024-08-08T20:42:32.8587104Z env: 2024-08-08T20:42:32.8587440Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:32.8587869Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:32.8588626Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:32.8589341Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:32.8589733Z AWS_REGION: us-east-1 2024-08-08T20:42:32.8590158Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:32.8590682Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:32.8599150Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:32.8599523Z ##[endgroup] 2024-08-08T20:42:33.2575885Z (node:2898847) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-08T20:42:33.2576800Z 2024-08-08T20:42:33.2577160Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-08T20:42:33.2578132Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-08T20:42:33.2579111Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-08T20:42:33.5254908Z Found 1 objects with prefix pytorch/pytorch/10308807531/td_results/ 2024-08-08T20:42:33.5256141Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-08T20:42:33.8707924Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-08T20:42:33.8719248Z Artifact download has finished successfully 2024-08-08T20:42:33.9081294Z ##[group]Run mkdir -p .additional_ci_files 2024-08-08T20:42:33.9082043Z mkdir -p .additional_ci_files 2024-08-08T20:42:33.9082819Z mv td_results.json .additional_ci_files/td_results.json 2024-08-08T20:42:33.9114067Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:33.9114771Z env: 2024-08-08T20:42:33.9115185Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:33.9115804Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:33.9116842Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:33.9117797Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:33.9118312Z AWS_REGION: us-east-1 2024-08-08T20:42:33.9118904Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:33.9119614Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:33.9131376Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:33.9131857Z ##[endgroup] 2024-08-08T20:42:33.9270604Z ##[group]Run .github/scripts/parse_ref.py 2024-08-08T20:42:33.9271324Z .github/scripts/parse_ref.py 2024-08-08T20:42:33.9303139Z shell: /usr/bin/bash -e {0} 2024-08-08T20:42:33.9303429Z env: 2024-08-08T20:42:33.9303641Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:33.9303938Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:33.9304459Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:33.9304949Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:33.9305221Z AWS_REGION: us-east-1 2024-08-08T20:42:33.9305517Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:33.9305883Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:33.9316807Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:33.9317295Z ##[endgroup] 2024-08-08T20:42:33.9586907Z Prepare all required actions 2024-08-08T20:42:33.9628515Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-08T20:42:33.9629154Z with: 2024-08-08T20:42:33.9629833Z github-token: *** 2024-08-08T20:42:33.9630282Z env: 2024-08-08T20:42:33.9630694Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:33.9631269Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:33.9632847Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:33.9633933Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:33.9634549Z AWS_REGION: us-east-1 2024-08-08T20:42:33.9635190Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:33.9636015Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:33.9648388Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:33.9648878Z ##[endgroup] 2024-08-08T20:42:33.9672718Z ##[group]Run set -eux 2024-08-08T20:42:33.9673208Z set -eux 2024-08-08T20:42:33.9674070Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-08T20:42:33.9702804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:33.9703163Z env: 2024-08-08T20:42:33.9703376Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:33.9703673Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:33.9704352Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:33.9705335Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:33.9705861Z AWS_REGION: us-east-1 2024-08-08T20:42:33.9706442Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:33.9707134Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:33.9718504Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:33.9719230Z GITHUB_TOKEN: *** 2024-08-08T20:42:33.9719675Z ##[endgroup] 2024-08-08T20:42:33.9767999Z + python3 .github/scripts/get_workflow_job_id.py 10308807531 pytorch-rocm-hw-22 2024-08-08T20:42:35.2265156Z setting job-id=28538197052 2024-08-08T20:42:35.2266187Z setting job-name=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:42:35.2597048Z Prepare all required actions 2024-08-08T20:42:35.2597784Z Getting action download info 2024-08-08T20:42:35.4160830Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-08T20:42:35.9245374Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-08T20:42:35.9246159Z with: 2024-08-08T20:42:35.9247039Z github-token: *** 2024-08-08T20:42:35.9248804Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-08-08T20:42:35.9250737Z job-name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:42:35.9251528Z env: 2024-08-08T20:42:35.9251923Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:35.9252477Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:35.9253502Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:35.9254444Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:35.9254786Z AWS_REGION: us-east-1 2024-08-08T20:42:35.9255126Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:35.9255531Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:35.9261333Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:35.9261577Z ##[endgroup] 2024-08-08T20:42:35.9311373Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-08T20:42:35.9311938Z with: 2024-08-08T20:42:35.9312236Z shell: bash 2024-08-08T20:42:35.9312566Z timeout_minutes: 10 2024-08-08T20:42:35.9312980Z max_attempts: 5 2024-08-08T20:42:35.9313378Z retry_wait_seconds: 30 2024-08-08T20:42:35.9314627Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-08T20:42:35.9315849Z polling_interval_seconds: 1 2024-08-08T20:42:35.9316244Z warning_on_retry: true 2024-08-08T20:42:35.9316604Z continue_on_error: false 2024-08-08T20:42:35.9316956Z env: 2024-08-08T20:42:35.9317264Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:35.9317691Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:35.9318783Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:35.9319493Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:35.9319889Z AWS_REGION: us-east-1 2024-08-08T20:42:35.9320328Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:35.9320852Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:35.9330331Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:35.9330850Z GITHUB_TOKEN: *** 2024-08-08T20:42:35.9331185Z ##[endgroup] 2024-08-08T20:42:35.9711779Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-08T20:42:36.2163461Z Defaulting to user installation because normal site-packages is not writeable 2024-08-08T20:42:36.2897030Z Requirement already satisfied: requests==2.27.1 in /home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-08-08T20:42:36.2900629Z Requirement already satisfied: pyyaml==6.0.1 in /home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-08-08T20:42:36.2989797Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-08-08T20:42:36.2997706Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-08-08T20:42:36.3007455Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-08-08T20:42:36.3010849Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-08-08T20:42:36.9715603Z Command completed after 1 attempt(s). 2024-08-08T20:42:36.9829942Z ##[group]Run set -x 2024-08-08T20:42:36.9830914Z set -x 2024-08-08T20:42:36.9831356Z  2024-08-08T20:42:36.9832057Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-08T20:42:36.9832904Z # in runner workspace 2024-08-08T20:42:36.9833606Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-08T20:42:36.9867490Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:36.9868188Z env: 2024-08-08T20:42:36.9868601Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:36.9869175Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:36.9870184Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:36.9871126Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:36.9871645Z AWS_REGION: us-east-1 2024-08-08T20:42:36.9872237Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:36.9872944Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:36.9884196Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:36.9884687Z ##[endgroup] 2024-08-08T20:42:36.9952311Z + python3 /home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-08T20:42:37.0202984Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-08T20:42:37.0203719Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-08T20:42:37.0204351Z echo "Job name: ${JOB_NAME}" 2024-08-08T20:42:37.0205018Z  2024-08-08T20:42:37.0205718Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-08T20:42:37.0206566Z # in runner workspace 2024-08-08T20:42:37.0207346Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-08T20:42:37.0208215Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-08T20:42:37.0208834Z  --job-name "${JOB_NAME}" \ 2024-08-08T20:42:37.0210586Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-08-08T20:42:37.0212341Z  --selected-test-configs "" \ 2024-08-08T20:42:37.0213053Z  --pr-number "${PR_NUMBER}" \ 2024-08-08T20:42:37.0214183Z  --tag "${TAG}" \ 2024-08-08T20:42:37.0214818Z  --event-name "${EVENT_NAME}" \ 2024-08-08T20:42:37.0215506Z  --schedule "${SCHEDULE}" \ 2024-08-08T20:42:37.0216158Z  --branch "${HEAD_BRANCH}" 2024-08-08T20:42:37.0252355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:37.0253184Z env: 2024-08-08T20:42:37.0253673Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:37.0254362Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:37.0255540Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:37.0256703Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:37.0257338Z AWS_REGION: us-east-1 2024-08-08T20:42:37.0258025Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:37.0258794Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:37.0270533Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:37.0271215Z GITHUB_TOKEN: *** 2024-08-08T20:42:37.0271941Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:42:37.0272762Z PR_NUMBER: 2024-08-08T20:42:37.0273192Z TAG: ciflow/trunk/132710 2024-08-08T20:42:37.0273671Z EVENT_NAME: push 2024-08-08T20:42:37.0274092Z SCHEDULE: 2024-08-08T20:42:37.0274484Z HEAD_BRANCH: 2024-08-08T20:42:37.0274903Z ##[endgroup] 2024-08-08T20:42:37.0338891Z Workflow: trunk 2024-08-08T20:42:37.0339723Z Job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:42:37.3559947Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-08T20:42:38.2938612Z ##[group]Run echo "Filtered matrix:" 2024-08-08T20:42:38.2939202Z echo "Filtered matrix:" 2024-08-08T20:42:38.2940059Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-08-08T20:42:38.2940909Z  2024-08-08T20:42:38.2941110Z echo 2024-08-08T20:42:38.2941372Z echo "Is the current job unstable? False" 2024-08-08T20:42:38.2941690Z  2024-08-08T20:42:38.2941888Z echo 2024-08-08T20:42:38.2942157Z echo "Is keep-going label set? False" 2024-08-08T20:42:38.2942517Z  2024-08-08T20:42:38.2942929Z echo 2024-08-08T20:42:38.2943433Z echo "Renabled issues? " 2024-08-08T20:42:38.2975078Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:38.2975912Z env: 2024-08-08T20:42:38.2976393Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:38.2977101Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:38.2977932Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:38.2978445Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:38.2978713Z AWS_REGION: us-east-1 2024-08-08T20:42:38.2979088Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:38.2979450Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:38.2987111Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:38.2987585Z ##[endgroup] 2024-08-08T20:42:38.3051029Z Filtered matrix: 2024-08-08T20:42:38.3052797Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-08-08T20:42:38.3054434Z 2024-08-08T20:42:38.3054698Z Is the current job unstable? False 2024-08-08T20:42:38.3055253Z 2024-08-08T20:42:38.3055634Z Is keep-going label set? False 2024-08-08T20:42:38.3056059Z 2024-08-08T20:42:38.3056317Z Renabled issues? 2024-08-08T20:42:38.3125605Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-08T20:42:38.3126637Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-08T20:42:38.3157757Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T20:42:38.3158847Z env: 2024-08-08T20:42:38.3159264Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:38.3159844Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:38.3160874Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:38.3161825Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:38.3162347Z AWS_REGION: us-east-1 2024-08-08T20:42:38.3162941Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:38.3163638Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:38.3175657Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:38.3176231Z JOB_TIMEOUT: 300 2024-08-08T20:42:38.3176747Z ##[endgroup] 2024-08-08T20:42:38.3299479Z ##[group]Run set -x 2024-08-08T20:42:38.3299804Z set -x 2024-08-08T20:42:38.3300034Z  2024-08-08T20:42:38.3300292Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-08T20:42:38.3300670Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-08T20:42:38.3301049Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-08T20:42:38.3301411Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-08-08T20:42:38.3301705Z else 2024-08-08T20:42:38.3301950Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-08T20:42:38.3302240Z fi 2024-08-08T20:42:38.3302451Z  2024-08-08T20:42:38.3302775Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-08T20:42:38.3303266Z # TODO: Stop building test binaries as part of the build phase 2024-08-08T20:42:38.3303695Z # Used for GPU_FLAG since that doesn't play nice 2024-08-08T20:42:38.3304086Z # shellcheck disable=SC2086,SC2090 2024-08-08T20:42:38.3304414Z container_name=$(docker run \ 2024-08-08T20:42:38.3304725Z  ${GPU_FLAG:-} \ 2024-08-08T20:42:38.3305005Z  -e BUILD_ENVIRONMENT \ 2024-08-08T20:42:38.3305302Z  -e PR_NUMBER \ 2024-08-08T20:42:38.3305568Z  -e GITHUB_ACTIONS \ 2024-08-08T20:42:38.3305855Z  -e GITHUB_REPOSITORY \ 2024-08-08T20:42:38.3306144Z  -e GITHUB_WORKFLOW \ 2024-08-08T20:42:38.3306421Z  -e GITHUB_JOB \ 2024-08-08T20:42:38.3306684Z  -e GITHUB_RUN_ID \ 2024-08-08T20:42:38.3306959Z  -e GITHUB_RUN_NUMBER \ 2024-08-08T20:42:38.3307244Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-08T20:42:38.3307527Z  -e JOB_ID \ 2024-08-08T20:42:38.3307812Z  -e JOB_NAME \ 2024-08-08T20:42:38.3308102Z  -e BRANCH \ 2024-08-08T20:42:38.3308566Z  -e SHA1 \ 2024-08-08T20:42:38.3309119Z  -e AWS_DEFAULT_REGION \ 2024-08-08T20:42:38.3309766Z  -e IN_WHEEL_TEST \ 2024-08-08T20:42:38.3310358Z  -e SHARD_NUMBER \ 2024-08-08T20:42:38.3310927Z  -e TEST_CONFIG \ 2024-08-08T20:42:38.3311422Z  -e NUM_TEST_SHARDS \ 2024-08-08T20:42:38.3311961Z  -e REENABLED_ISSUES \ 2024-08-08T20:42:38.3312529Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-08T20:42:38.3313115Z  -e VERBOSE_TEST_LOGS \ 2024-08-08T20:42:38.3313665Z  -e TEST_SHOWLOCALS \ 2024-08-08T20:42:38.3314186Z  -e NO_TEST_TIMEOUT \ 2024-08-08T20:42:38.3314693Z  -e NO_TD \ 2024-08-08T20:42:38.3315220Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-08T20:42:38.3315860Z  -e SCCACHE_BUCKET \ 2024-08-08T20:42:38.3316520Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-08T20:42:38.3317334Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-08T20:42:38.3318135Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-08T20:42:38.3318895Z  -e TESTS_TO_INCLUDE \ 2024-08-08T20:42:38.3319660Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-08T20:42:38.3320475Z  --ulimit stack=10485760:83886080 \ 2024-08-08T20:42:38.3321176Z  --ulimit core=0 \ 2024-08-08T20:42:38.3321828Z  --security-opt seccomp=unconfined \ 2024-08-08T20:42:38.3322446Z  --cap-add=SYS_PTRACE \ 2024-08-08T20:42:38.3323379Z  --shm-size="8g" \ 2024-08-08T20:42:38.3323871Z  --tty \ 2024-08-08T20:42:38.3324312Z  --detach \ 2024-08-08T20:42:38.3324920Z  --name="${container_name}" \ 2024-08-08T20:42:38.3325496Z  --user jenkins \ 2024-08-08T20:42:38.3326136Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-08T20:42:38.3326891Z  -w /var/lib/jenkins/workspace \ 2024-08-08T20:42:38.3327574Z  "${DOCKER_IMAGE}" 2024-08-08T20:42:38.3328147Z ) 2024-08-08T20:42:38.3328691Z # save container name for later step 2024-08-08T20:42:38.3329975Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-08-08T20:42:38.3331296Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-08-08T20:42:38.3332960Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-08-08T20:42:38.3360237Z shell: /usr/bin/bash -e {0} 2024-08-08T20:42:38.3360783Z env: 2024-08-08T20:42:38.3361202Z GIT_DEFAULT_BRANCH: main 2024-08-08T20:42:38.3361786Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T20:42:38.3362817Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T20:42:38.3363757Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T20:42:38.3364277Z AWS_REGION: us-east-1 2024-08-08T20:42:38.3365250Z AWS_ACCESS_KEY_ID: *** 2024-08-08T20:42:38.3365999Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T20:42:38.3377969Z AWS_SESSION_TOKEN: *** 2024-08-08T20:42:38.3378318Z BUILD_ENVIRONMENT: linux-focal-rocm6.1-py3.8 2024-08-08T20:42:38.3378645Z PR_NUMBER: 2024-08-08T20:42:38.3378886Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-08T20:42:38.3379192Z GITHUB_WORKFLOW: trunk 2024-08-08T20:42:38.3379439Z GITHUB_JOB: test 2024-08-08T20:42:38.3379679Z GITHUB_RUN_ID: 10308807531 2024-08-08T20:42:38.3379940Z GITHUB_RUN_NUMBER: 90547 2024-08-08T20:42:38.3380197Z GITHUB_RUN_ATTEMPT: 1 2024-08-08T20:42:38.3380441Z JOB_ID: 28538197052 2024-08-08T20:42:38.3380813Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:42:38.3381224Z BRANCH: 2024-08-08T20:42:38.3381466Z SHA1: b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:42:38.3381780Z CONTINUE_THROUGH_ERROR: False 2024-08-08T20:42:38.3382054Z VERBOSE_TEST_LOGS: False 2024-08-08T20:42:38.3382308Z TEST_SHOWLOCALS: False 2024-08-08T20:42:38.3382555Z NO_TEST_TIMEOUT: False 2024-08-08T20:42:38.3382794Z NO_TD: False 2024-08-08T20:42:38.3383012Z TEST_CONFIG: distributed 2024-08-08T20:42:38.3383256Z SHARD_NUMBER: 1 2024-08-08T20:42:38.3383513Z NUM_TEST_SHARDS: 1 2024-08-08T20:42:38.3383796Z REENABLED_ISSUES: 2024-08-08T20:42:38.3384297Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-08T20:42:38.3385526Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:42:38.3386765Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-08T20:42:38.3387373Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-08T20:42:38.3387811Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-08T20:42:38.3388952Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-08T20:42:38.3390074Z ##[endgroup] 2024-08-08T20:42:38.3440247Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-08T20:42:38.3440994Z + [[ linux-focal-rocm6.1-py3.8 == *onnx* ]] 2024-08-08T20:42:38.3441647Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-08T20:42:38.3457801Z +++ nproc --ignore=2 2024-08-08T20:42:38.3486984Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_10308807531 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:02ec4fbd5adcb3fb91cf5ce431dec18b633de7d9 2024-08-08T20:42:41.3927843Z + container_name=aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T20:42:41.3928419Z + echo CONTAINER_NAME=aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T20:42:41.3929229Z + docker exec -t aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-08-08T20:42:51.9466096Z Processing ./dist/torch-2.5.0a0+gitb9d86fa-cp38-cp38-linux_x86_64.whl 2024-08-08T20:42:52.1635041Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (3.13.1) 2024-08-08T20:42:52.1640216Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (4.12.2) 2024-08-08T20:42:52.1641967Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (2.8.8) 2024-08-08T20:42:52.1643338Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (3.1.4) 2024-08-08T20:42:52.1647270Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (2024.6.1) 2024-08-08T20:42:52.1655264Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+gitb9d86fa) (1.12.1) 2024-08-08T20:42:52.1672991Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch==2.5.0a0+gitb9d86fa) (1.3.0) 2024-08-08T20:42:52.2375015Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.5.0a0+gitb9d86fa) (2.1.5) 2024-08-08T20:42:52.8780786Z Installing collected packages: torch 2024-08-08T20:43:01.0773535Z Successfully installed torch-2.5.0a0+gitb9d86fa 2024-08-08T20:43:01.1274143Z ++ dirname .ci/pytorch/test.sh 2024-08-08T20:43:01.1297370Z + source .ci/pytorch/common.sh 2024-08-08T20:43:01.1306243Z +++ dirname .ci/pytorch/common.sh 2024-08-08T20:43:01.1318402Z ++ source .ci/pytorch/common_utils.sh 2024-08-08T20:43:01.1321733Z +++ declare -f -t trap_add 2024-08-08T20:43:01.1326448Z ++ set -ex 2024-08-08T20:43:01.1326936Z ++ [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-08T20:43:01.1327275Z ++ unset HIP_PLATFORM 2024-08-08T20:43:01.1327547Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-08-08T20:43:01.1327836Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-08-08T20:43:01.1328113Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-08-08T20:43:01.1328405Z ++ HSAKMT_DEBUG_LEVEL=4 2024-08-08T20:43:01.1328665Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-08T20:43:01.1328954Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-08T20:43:01.1329224Z ++ BUILD_TEST_LIBTORCH=0 2024-08-08T20:43:01.1329493Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-08-08T20:43:01.1329807Z + echo 'Environment variables:' 2024-08-08T20:43:01.1330579Z Environment variables: 2024-08-08T20:43:01.1330834Z + env 2024-08-08T20:43:01.1343224Z INSTALLED_DB=yes 2024-08-08T20:43:01.1343662Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T20:43:01.1344138Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-08-08T20:43:01.1344465Z CONTINUE_THROUGH_ERROR=False 2024-08-08T20:43:01.1344769Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-08-08T20:43:01.1345085Z HOSTNAME=aa07fab653b1 2024-08-08T20:43:01.1345636Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1346263Z GITHUB_ACTION=__self 2024-08-08T20:43:01.1346927Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-08T20:43:01.1347224Z GITHUB_RUN_NUMBER=90547 2024-08-08T20:43:01.1347471Z TEST_CONFIG=distributed 2024-08-08T20:43:01.1347727Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-08T20:43:01.1348018Z AWS_DEFAULT_REGION=us-east-1 2024-08-08T20:43:01.1348306Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-08T20:43:01.1348614Z GITHUB_REF_TYPE=tag 2024-08-08T20:43:01.1349126Z *** 2024-08-08T20:43:01.1349358Z GITHUB_REPOSITORY_ID=65600975 2024-08-08T20:43:01.1349632Z GITHUB_ACTIONS=true 2024-08-08T20:43:01.1349899Z SHA1=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1350246Z GITHUB_SHA=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1350786Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/132710 2024-08-08T20:43:01.1351270Z VERBOSE_TEST_LOGS=False 2024-08-08T20:43:01.1351536Z GITHUB_REF=refs/tags/ciflow/trunk/132710 2024-08-08T20:43:01.1351827Z SHARD_NUMBER=1 2024-08-08T20:43:01.1352060Z GITHUB_REF_PROTECTED=false 2024-08-08T20:43:01.1352313Z HOME=/var/lib/jenkins 2024-08-08T20:43:01.1352579Z GITHUB_API_URL=https://api.github.com 2024-08-08T20:43:01.1352897Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-08T20:43:01.1353203Z LANG=C.UTF-8 2024-08-08T20:43:01.1353423Z PYTORCH_TEST_WITH_ROCM=1 2024-08-08T20:43:01.1353667Z NUM_TEST_SHARDS=1 2024-08-08T20:43:01.1354209Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1354931Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:43:01.1355351Z MAGMA_HOME=/opt/rocm/magma 2024-08-08T20:43:01.1355878Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1356433Z HSAKMT_DEBUG_LEVEL=4 2024-08-08T20:43:01.1356846Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-08T20:43:01.1357316Z GITHUB_EVENT_NAME=push 2024-08-08T20:43:01.1357556Z GITHUB_RUN_ID=10308807531 2024-08-08T20:43:01.1358140Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1358765Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-08T20:43:01.1359028Z PR_NUMBER= 2024-08-08T20:43:01.1359243Z GITHUB_RUN_ATTEMPT=1 2024-08-08T20:43:01.1359487Z ANACONDA_PYTHON_VERSION=3.8 2024-08-08T20:43:01.1359795Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-08T20:43:01.1360120Z TERM=xterm 2024-08-08T20:43:01.1360327Z INSTALLED_VISION=yes 2024-08-08T20:43:01.1360550Z BRANCH= 2024-08-08T20:43:01.1361250Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-08T20:43:01.1362250Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-08-08T20:43:01.1362784Z GITHUB_SERVER_URL=https://github.com 2024-08-08T20:43:01.1363080Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-08-08T20:43:01.1363342Z REENABLED_ISSUES= 2024-08-08T20:43:01.1363559Z SHLVL=1 2024-08-08T20:43:01.1363754Z MAX_JOBS=62 2024-08-08T20:43:01.1363964Z GITHUB_ACTOR_ID=54816060 2024-08-08T20:43:01.1364277Z GITHUB_WORKFLOW_SHA=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1364872Z GITHUB_REF_NAME=ciflow/trunk/132710 2024-08-08T20:43:01.1365154Z ROCM_PATH=/opt/rocm 2024-08-08T20:43:01.1365508Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-08T20:43:01.1365900Z GITHUB_JOB=test 2024-08-08T20:43:01.1366123Z NO_TEST_TIMEOUT=False 2024-08-08T20:43:01.1366375Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-08T20:43:01.1366663Z LC_ALL=C.UTF-8 2024-08-08T20:43:01.1366882Z GITHUB_RETENTION_DAYS=90 2024-08-08T20:43:01.1367133Z GITHUB_ACTION_REPOSITORY= 2024-08-08T20:43:01.1368138Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-08T20:43:01.1369004Z GITHUB_BASE_REF= 2024-08-08T20:43:01.1369216Z CI=true 2024-08-08T20:43:01.1369420Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-08T20:43:01.1369686Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-08T20:43:01.1369946Z JOB_ID=28538197052 2024-08-08T20:43:01.1370178Z INSTALLED_PROTOBUF=yes 2024-08-08T20:43:01.1370412Z GITHUB_HEAD_REF= 2024-08-08T20:43:01.1370627Z GITHUB_ACTION_REF= 2024-08-08T20:43:01.1370896Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-08T20:43:01.1371219Z TEST_SHOWLOCALS=False 2024-08-08T20:43:01.1371455Z GITHUB_WORKFLOW=trunk 2024-08-08T20:43:01.1371696Z DEBIAN_FRONTEND=noninteractive 2024-08-08T20:43:01.1372280Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1372859Z NO_TD=False 2024-08-08T20:43:01.1373067Z OLDPWD=/var/lib/jenkins 2024-08-08T20:43:01.1373306Z _=/usr/bin/env 2024-08-08T20:43:01.1373600Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-08T20:43:01.1502074Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-08-08T20:43:01.1502663Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T20:43:01.1503189Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-08-08T20:43:01.1503741Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-08-08T20:43:01.1504140Z + BUILD_DIR=build 2024-08-08T20:43:01.1504378Z + BUILD_RENAMED_DIR=build_renamed 2024-08-08T20:43:01.1504663Z + BUILD_BIN_DIR=build/bin 2024-08-08T20:43:01.1504901Z + SHARD_NUMBER=1 2024-08-08T20:43:01.1505120Z + NUM_TEST_SHARDS=1 2024-08-08T20:43:01.1505344Z + export VALGRIND=ON 2024-08-08T20:43:01.1505569Z + VALGRIND=ON 2024-08-08T20:43:01.1505813Z + [[ linux-focal-rocm6.1-py3.8 == *clang9* ]] 2024-08-08T20:43:01.1506110Z + [[ 0 == \1 ]] 2024-08-08T20:43:01.1506339Z + [[ False == \1 ]] 2024-08-08T20:43:01.1506589Z + [[ linux-focal-rocm6.1-py3.8 != *bazel* ]] 2024-08-08T20:43:01.1507242Z ++ realpath build/custom_test_artifacts 2024-08-08T20:43:01.1517578Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-08-08T20:43:01.1518560Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-08-08T20:43:01.1519327Z + echo 'Setting INCLUDE_CLAUSE' 2024-08-08T20:43:01.1519627Z Setting INCLUDE_CLAUSE 2024-08-08T20:43:01.1520399Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-08-08T20:43:01.1521210Z + echo 'Environment variables' 2024-08-08T20:43:01.1521473Z Environment variables 2024-08-08T20:43:01.1521699Z + env 2024-08-08T20:43:01.1527317Z INSTALLED_DB=yes 2024-08-08T20:43:01.1527800Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T20:43:01.1528273Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-08-08T20:43:01.1528598Z CONTINUE_THROUGH_ERROR=False 2024-08-08T20:43:01.1528896Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-08-08T20:43:01.1529212Z HOSTNAME=aa07fab653b1 2024-08-08T20:43:01.1529996Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1530576Z GITHUB_ACTION=__self 2024-08-08T20:43:01.1530832Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-08T20:43:01.1531115Z GITHUB_RUN_NUMBER=90547 2024-08-08T20:43:01.1531359Z TEST_CONFIG=distributed 2024-08-08T20:43:01.1531655Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-08T20:43:01.1531950Z AWS_DEFAULT_REGION=us-east-1 2024-08-08T20:43:01.1532239Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-08T20:43:01.1532536Z GITHUB_REF_TYPE=tag 2024-08-08T20:43:01.1532824Z *** 2024-08-08T20:43:01.1533208Z GITHUB_REPOSITORY_ID=65600975 2024-08-08T20:43:01.1533485Z GITHUB_ACTIONS=true 2024-08-08T20:43:01.1533745Z SHA1=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1534091Z GITHUB_SHA=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1534611Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/132710 2024-08-08T20:43:01.1535103Z VERBOSE_TEST_LOGS=False 2024-08-08T20:43:01.1535367Z GITHUB_REF=refs/tags/ciflow/trunk/132710 2024-08-08T20:43:01.1535656Z SHARD_NUMBER=1 2024-08-08T20:43:01.1535884Z GITHUB_REF_PROTECTED=false 2024-08-08T20:43:01.1536135Z HOME=/var/lib/jenkins 2024-08-08T20:43:01.1536407Z GITHUB_API_URL=https://api.github.com 2024-08-08T20:43:01.1536722Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-08T20:43:01.1536996Z LANG=C.UTF-8 2024-08-08T20:43:01.1537213Z PYTORCH_TEST_WITH_ROCM=1 2024-08-08T20:43:01.1537459Z NUM_TEST_SHARDS=1 2024-08-08T20:43:01.1538008Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1538721Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-08T20:43:01.1539140Z MAGMA_HOME=/opt/rocm/magma 2024-08-08T20:43:01.1539672Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1540234Z HSAKMT_DEBUG_LEVEL=4 2024-08-08T20:43:01.1540649Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-08T20:43:01.1541102Z GITHUB_EVENT_NAME=push 2024-08-08T20:43:01.1541341Z GITHUB_RUN_ID=10308807531 2024-08-08T20:43:01.1541921Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1542540Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-08T20:43:01.1542802Z PR_NUMBER= 2024-08-08T20:43:01.1543013Z GITHUB_RUN_ATTEMPT=1 2024-08-08T20:43:01.1543237Z VALGRIND=ON 2024-08-08T20:43:01.1543466Z ANACONDA_PYTHON_VERSION=3.8 2024-08-08T20:43:01.1543774Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-08T20:43:01.1544092Z TERM=xterm 2024-08-08T20:43:01.1544299Z INSTALLED_VISION=yes 2024-08-08T20:43:01.1544520Z BRANCH= 2024-08-08T20:43:01.1545214Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-08T20:43:01.1546213Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-08-08T20:43:01.1546741Z GITHUB_SERVER_URL=https://github.com 2024-08-08T20:43:01.1547040Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-08-08T20:43:01.1547299Z REENABLED_ISSUES= 2024-08-08T20:43:01.1547513Z SHLVL=1 2024-08-08T20:43:01.1547703Z MAX_JOBS=62 2024-08-08T20:43:01.1547909Z GITHUB_ACTOR_ID=54816060 2024-08-08T20:43:01.1548223Z GITHUB_WORKFLOW_SHA=b9d86fa89636e301796d4201f36d86c73f6e49bc 2024-08-08T20:43:01.1548588Z GITHUB_REF_NAME=ciflow/trunk/132710 2024-08-08T20:43:01.1548861Z ROCM_PATH=/opt/rocm 2024-08-08T20:43:01.1549210Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-08T20:43:01.1549594Z GITHUB_JOB=test 2024-08-08T20:43:01.1549811Z NO_TEST_TIMEOUT=False 2024-08-08T20:43:01.1550063Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-08T20:43:01.1550483Z LC_ALL=C.UTF-8 2024-08-08T20:43:01.1550708Z GITHUB_RETENTION_DAYS=90 2024-08-08T20:43:01.1550963Z GITHUB_ACTION_REPOSITORY= 2024-08-08T20:43:01.1551809Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-08T20:43:01.1553108Z GITHUB_BASE_REF= 2024-08-08T20:43:01.1553430Z CI=true 2024-08-08T20:43:01.1553743Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-08T20:43:01.1554148Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-08T20:43:01.1554563Z JOB_ID=28538197052 2024-08-08T20:43:01.1555135Z INSTALLED_PROTOBUF=yes 2024-08-08T20:43:01.1555500Z GITHUB_HEAD_REF= 2024-08-08T20:43:01.1555825Z GITHUB_ACTION_REF= 2024-08-08T20:43:01.1556239Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-08T20:43:01.1556740Z TEST_SHOWLOCALS=False 2024-08-08T20:43:01.1557095Z GITHUB_WORKFLOW=trunk 2024-08-08T20:43:01.1557466Z DEBIAN_FRONTEND=noninteractive 2024-08-08T20:43:01.1558404Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_a59851c8-67cc-4130-94dd-099d0e47bee6 2024-08-08T20:43:01.1559280Z NO_TD=False 2024-08-08T20:43:01.1559596Z OLDPWD=/var/lib/jenkins 2024-08-08T20:43:01.1559942Z _=/usr/bin/env 2024-08-08T20:43:01.1560272Z + echo 'Testing pytorch' 2024-08-08T20:43:01.1560636Z Testing pytorch 2024-08-08T20:43:01.1560968Z + export LANG=C.UTF-8 2024-08-08T20:43:01.1561312Z + LANG=C.UTF-8 2024-08-08T20:43:01.1561622Z + PR_NUMBER= 2024-08-08T20:43:01.1561954Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-08T20:43:01.1562410Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-08T20:43:01.1562892Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-08T20:43:01.1563358Z + export HIP_VISIBLE_DEVICES=0,1 2024-08-08T20:43:01.1563767Z + HIP_VISIBLE_DEVICES=0,1 2024-08-08T20:43:01.1564158Z + [[ distributed == \s\l\o\w ]] 2024-08-08T20:43:01.1564626Z + [[ linux-focal-rocm6.1-py3.8 == *slow-gradcheck* ]] 2024-08-08T20:43:01.1565271Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-08T20:43:01.1565747Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-08T20:43:01.1566231Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-08T20:43:01.1566716Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-08T20:43:01.1567161Z + [[ distributed == *crossref* ]] 2024-08-08T20:43:01.1567592Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-08T20:43:01.1568026Z + export VALGRIND=OFF 2024-08-08T20:43:01.1568368Z + VALGRIND=OFF 2024-08-08T20:43:01.1568680Z + rocminfo 2024-08-08T20:43:01.1662064Z ROCk module version 6.7.0 is loaded 2024-08-08T20:43:01.2273476Z ===================== 2024-08-08T20:43:01.2274171Z HSA System Attributes 2024-08-08T20:43:01.2274695Z ===================== 2024-08-08T20:43:01.2275179Z Runtime Version: 1.13 2024-08-08T20:43:01.2275699Z Runtime Ext Version: 1.4 2024-08-08T20:43:01.2276228Z System Timestamp Freq.: 1000.000000MHz 2024-08-08T20:43:01.2277098Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-08-08T20:43:01.2278035Z Machine Model: LARGE 2024-08-08T20:43:01.2278777Z System Endianness: LITTLE 2024-08-08T20:43:01.2279425Z Mwaitx: DISABLED 2024-08-08T20:43:01.2279940Z DMAbuf Support: YES 2024-08-08T20:43:01.2280255Z 2024-08-08T20:43:01.2280420Z ========== 2024-08-08T20:43:01.2280871Z HSA Agents 2024-08-08T20:43:01.2281309Z ========== 2024-08-08T20:43:01.2281837Z ******* 2024-08-08T20:43:01.2282317Z Agent 1 2024-08-08T20:43:01.2282756Z ******* 2024-08-08T20:43:01.2283315Z Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.2283997Z Uuid: CPU-XX 2024-08-08T20:43:01.2284700Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.2285639Z Vendor Name: CPU 2024-08-08T20:43:01.2287002Z Feature: None specified 2024-08-08T20:43:01.2287493Z Profile: FULL_PROFILE 2024-08-08T20:43:01.2287856Z Float Round Mode: NEAR 2024-08-08T20:43:01.2288219Z Max Queue Number: 0(0x0) 2024-08-08T20:43:01.2288582Z Queue Min Size: 0(0x0) 2024-08-08T20:43:01.2288935Z Queue Max Size: 0(0x0) 2024-08-08T20:43:01.2289456Z Queue Type: MULTI 2024-08-08T20:43:01.2289794Z Node: 0 2024-08-08T20:43:01.2290130Z Device Type: CPU 2024-08-08T20:43:01.2290451Z Cache Info: 2024-08-08T20:43:01.2290728Z L1: 32768(0x8000) KB 2024-08-08T20:43:01.2291067Z Chip ID: 0(0x0) 2024-08-08T20:43:01.2291421Z ASIC Revision: 0(0x0) 2024-08-08T20:43:01.2291788Z Cacheline Size: 64(0x40) 2024-08-08T20:43:01.2292151Z Max Clock Freq. (MHz): 2600 2024-08-08T20:43:01.2292489Z BDFID: 0 2024-08-08T20:43:01.2292829Z Internal Node ID: 0 2024-08-08T20:43:01.2293186Z Compute Unit: 32 2024-08-08T20:43:01.2293545Z SIMDs per CU: 0 2024-08-08T20:43:01.2293897Z Shader Engines: 0 2024-08-08T20:43:01.2294261Z Shader Arrs. per Eng.: 0 2024-08-08T20:43:01.2294635Z WatchPts on Addr. Ranges:1 2024-08-08T20:43:01.2294966Z Features: None 2024-08-08T20:43:01.2295220Z Pool Info: 2024-08-08T20:43:01.2295471Z Pool 1 2024-08-08T20:43:01.2295778Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:43:01.2296143Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:43:01.2296507Z Allocatable: TRUE 2024-08-08T20:43:01.2296875Z Alloc Granule: 4KB 2024-08-08T20:43:01.2297260Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2297670Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2298081Z Accessible by all: TRUE 2024-08-08T20:43:01.2298422Z Pool 2 2024-08-08T20:43:01.2298734Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-08T20:43:01.2299093Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:43:01.2299536Z Allocatable: TRUE 2024-08-08T20:43:01.2299911Z Alloc Granule: 4KB 2024-08-08T20:43:01.2300292Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2300677Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2301051Z Accessible by all: TRUE 2024-08-08T20:43:01.2301380Z Pool 3 2024-08-08T20:43:01.2301678Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:43:01.2302027Z Size: 65790812(0x3ebe35c) KB 2024-08-08T20:43:01.2302373Z Allocatable: TRUE 2024-08-08T20:43:01.2302734Z Alloc Granule: 4KB 2024-08-08T20:43:01.2303113Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2303663Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2304031Z Accessible by all: TRUE 2024-08-08T20:43:01.2304359Z ISA Info: 2024-08-08T20:43:01.2304600Z ******* 2024-08-08T20:43:01.2304842Z Agent 2 2024-08-08T20:43:01.2305069Z ******* 2024-08-08T20:43:01.2305343Z Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.2305697Z Uuid: CPU-XX 2024-08-08T20:43:01.2306299Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.2307154Z Vendor Name: CPU 2024-08-08T20:43:01.2307963Z Feature: None specified 2024-08-08T20:43:01.2308754Z Profile: FULL_PROFILE 2024-08-08T20:43:01.2309460Z Float Round Mode: NEAR 2024-08-08T20:43:01.2310169Z Max Queue Number: 0(0x0) 2024-08-08T20:43:01.2310858Z Queue Min Size: 0(0x0) 2024-08-08T20:43:01.2311540Z Queue Max Size: 0(0x0) 2024-08-08T20:43:01.2312213Z Queue Type: MULTI 2024-08-08T20:43:01.2312850Z Node: 1 2024-08-08T20:43:01.2313495Z Device Type: CPU 2024-08-08T20:43:01.2314112Z Cache Info: 2024-08-08T20:43:01.2314628Z L1: 32768(0x8000) KB 2024-08-08T20:43:01.2315243Z Chip ID: 0(0x0) 2024-08-08T20:43:01.2315905Z ASIC Revision: 0(0x0) 2024-08-08T20:43:01.2316605Z Cacheline Size: 64(0x40) 2024-08-08T20:43:01.2317319Z Max Clock Freq. (MHz): 2600 2024-08-08T20:43:01.2317976Z BDFID: 0 2024-08-08T20:43:01.2318633Z Internal Node ID: 1 2024-08-08T20:43:01.2319321Z Compute Unit: 32 2024-08-08T20:43:01.2319990Z SIMDs per CU: 0 2024-08-08T20:43:01.2320669Z Shader Engines: 0 2024-08-08T20:43:01.2321378Z Shader Arrs. per Eng.: 0 2024-08-08T20:43:01.2322107Z WatchPts on Addr. Ranges:1 2024-08-08T20:43:01.2322742Z Features: None 2024-08-08T20:43:01.2323231Z Pool Info: 2024-08-08T20:43:01.2323700Z Pool 1 2024-08-08T20:43:01.2324288Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:43:01.2325094Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:43:01.2325771Z Allocatable: TRUE 2024-08-08T20:43:01.2326471Z Alloc Granule: 4KB 2024-08-08T20:43:01.2327220Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2327968Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2328688Z Accessible by all: TRUE 2024-08-08T20:43:01.2329316Z Pool 2 2024-08-08T20:43:01.2329897Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-08T20:43:01.2330572Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:43:01.2331239Z Allocatable: TRUE 2024-08-08T20:43:01.2331939Z Alloc Granule: 4KB 2024-08-08T20:43:01.2332989Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2333729Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2334447Z Accessible by all: TRUE 2024-08-08T20:43:01.2335070Z Pool 3 2024-08-08T20:43:01.2335637Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:43:01.2336346Z Size: 66046452(0x3efc9f4) KB 2024-08-08T20:43:01.2337130Z Allocatable: TRUE 2024-08-08T20:43:01.2338297Z Alloc Granule: 4KB 2024-08-08T20:43:01.2339187Z Alloc Recommended Granule:4KB 2024-08-08T20:43:01.2340063Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2340919Z Accessible by all: TRUE 2024-08-08T20:43:01.2341434Z ISA Info: 2024-08-08T20:43:01.2341711Z ******* 2024-08-08T20:43:01.2341962Z Agent 3 2024-08-08T20:43:01.2342187Z ******* 2024-08-08T20:43:01.2342443Z Name: gfx90a 2024-08-08T20:43:01.2342774Z Uuid: GPU-0741618dc9e991d3 2024-08-08T20:43:01.2343132Z Marketing Name: AMD Instinct MI210 2024-08-08T20:43:01.2343497Z Vendor Name: AMD 2024-08-08T20:43:01.2343860Z Feature: KERNEL_DISPATCH 2024-08-08T20:43:01.2344214Z Profile: BASE_PROFILE 2024-08-08T20:43:01.2344583Z Float Round Mode: NEAR 2024-08-08T20:43:01.2344945Z Max Queue Number: 128(0x80) 2024-08-08T20:43:01.2345300Z Queue Min Size: 64(0x40) 2024-08-08T20:43:01.2345659Z Queue Max Size: 131072(0x20000) 2024-08-08T20:43:01.2346008Z Queue Type: MULTI 2024-08-08T20:43:01.2346340Z Node: 2 2024-08-08T20:43:01.2346715Z Device Type: GPU 2024-08-08T20:43:01.2347158Z Cache Info: 2024-08-08T20:43:01.2347752Z L1: 16(0x10) KB 2024-08-08T20:43:01.2348445Z L2: 8192(0x2000) KB 2024-08-08T20:43:01.2349173Z Chip ID: 29711(0x740f) 2024-08-08T20:43:01.2349880Z ASIC Revision: 1(0x1) 2024-08-08T20:43:01.2350577Z Cacheline Size: 64(0x40) 2024-08-08T20:43:01.2351279Z Max Clock Freq. (MHz): 1700 2024-08-08T20:43:01.2351934Z BDFID: 768 2024-08-08T20:43:01.2352590Z Internal Node ID: 2 2024-08-08T20:43:01.2353277Z Compute Unit: 104 2024-08-08T20:43:01.2353944Z SIMDs per CU: 4 2024-08-08T20:43:01.2354622Z Shader Engines: 8 2024-08-08T20:43:01.2355323Z Shader Arrs. per Eng.: 1 2024-08-08T20:43:01.2356046Z WatchPts on Addr. Ranges:4 2024-08-08T20:43:01.2356785Z Coherent Host Access: FALSE 2024-08-08T20:43:01.2357459Z Features: KERNEL_DISPATCH 2024-08-08T20:43:01.2358110Z Fast F16 Operation: TRUE 2024-08-08T20:43:01.2358820Z Wavefront Size: 64(0x40) 2024-08-08T20:43:01.2359844Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:43:01.2360499Z Workgroup Max Size per Dimension: 2024-08-08T20:43:01.2361061Z x 1024(0x400) 2024-08-08T20:43:01.2361634Z y 1024(0x400) 2024-08-08T20:43:01.2362204Z z 1024(0x400) 2024-08-08T20:43:01.2362832Z Max Waves Per CU: 32(0x20) 2024-08-08T20:43:01.2363546Z Max Work-item Per CU: 2048(0x800) 2024-08-08T20:43:01.2369393Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:43:01.2370092Z Grid Max Size per Dimension: 2024-08-08T20:43:01.2370616Z x 4294967295(0xffffffff) 2024-08-08T20:43:01.2371209Z y 4294967295(0xffffffff) 2024-08-08T20:43:01.2371794Z z 4294967295(0xffffffff) 2024-08-08T20:43:01.2372474Z Max fbarriers/Workgrp: 32 2024-08-08T20:43:01.2373275Z Packet Processor uCode:: 82 2024-08-08T20:43:01.2374014Z SDMA engine uCode:: 8 2024-08-08T20:43:01.2374722Z IOMMU Support:: None 2024-08-08T20:43:01.2375339Z Pool Info: 2024-08-08T20:43:01.2375809Z Pool 1 2024-08-08T20:43:01.2376453Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:43:01.2377288Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2378085Z Allocatable: TRUE 2024-08-08T20:43:01.2378924Z Alloc Granule: 4KB 2024-08-08T20:43:01.2379801Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2380697Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2381345Z Accessible by all: FALSE 2024-08-08T20:43:01.2381725Z Pool 2 2024-08-08T20:43:01.2382036Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-08T20:43:01.2382391Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2382749Z Allocatable: TRUE 2024-08-08T20:43:01.2383119Z Alloc Granule: 4KB 2024-08-08T20:43:01.2383507Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2383901Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2384274Z Accessible by all: FALSE 2024-08-08T20:43:01.2384595Z Pool 3 2024-08-08T20:43:01.2384894Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:43:01.2385242Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2385584Z Allocatable: TRUE 2024-08-08T20:43:01.2385948Z Alloc Granule: 4KB 2024-08-08T20:43:01.2386329Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2386769Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2387471Z Accessible by all: FALSE 2024-08-08T20:43:01.2388251Z Pool 4 2024-08-08T20:43:01.2388934Z Segment: GROUP 2024-08-08T20:43:01.2389662Z Size: 64(0x40) KB 2024-08-08T20:43:01.2390340Z Allocatable: FALSE 2024-08-08T20:43:01.2391047Z Alloc Granule: 0KB 2024-08-08T20:43:01.2392114Z Alloc Recommended Granule:0KB 2024-08-08T20:43:01.2392869Z Alloc Alignment: 0KB 2024-08-08T20:43:01.2393589Z Accessible by all: FALSE 2024-08-08T20:43:01.2394224Z ISA Info: 2024-08-08T20:43:01.2394694Z ISA 1 2024-08-08T20:43:01.2395302Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-08T20:43:01.2396067Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-08T20:43:01.2397093Z Profiles: HSA_PROFILE_BASE 2024-08-08T20:43:01.2397845Z Default Rounding Mode: NEAR 2024-08-08T20:43:01.2398590Z Default Rounding Mode: NEAR 2024-08-08T20:43:01.2399300Z Fast f16: TRUE 2024-08-08T20:43:01.2400010Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:43:01.2400686Z Workgroup Max Size per Dimension: 2024-08-08T20:43:01.2401273Z x 1024(0x400) 2024-08-08T20:43:01.2401885Z y 1024(0x400) 2024-08-08T20:43:01.2402503Z z 1024(0x400) 2024-08-08T20:43:01.2403160Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:43:01.2403816Z Grid Max Size per Dimension: 2024-08-08T20:43:01.2404380Z x 4294967295(0xffffffff) 2024-08-08T20:43:01.2405102Z y 4294967295(0xffffffff) 2024-08-08T20:43:01.2405691Z z 4294967295(0xffffffff) 2024-08-08T20:43:01.2406362Z FBarrier Max Size: 32 2024-08-08T20:43:01.2406976Z ******* 2024-08-08T20:43:01.2407432Z Agent 4 2024-08-08T20:43:01.2407867Z ******* 2024-08-08T20:43:01.2408375Z Name: gfx90a 2024-08-08T20:43:01.2409042Z Uuid: GPU-d2f63e3de4d6eee3 2024-08-08T20:43:01.2409749Z Marketing Name: AMD Instinct MI210 2024-08-08T20:43:01.2410465Z Vendor Name: AMD 2024-08-08T20:43:01.2411151Z Feature: KERNEL_DISPATCH 2024-08-08T20:43:01.2411847Z Profile: BASE_PROFILE 2024-08-08T20:43:01.2412536Z Float Round Mode: NEAR 2024-08-08T20:43:01.2413237Z Max Queue Number: 128(0x80) 2024-08-08T20:43:01.2413926Z Queue Min Size: 64(0x40) 2024-08-08T20:43:01.2414601Z Queue Max Size: 131072(0x20000) 2024-08-08T20:43:01.2415278Z Queue Type: MULTI 2024-08-08T20:43:01.2415914Z Node: 3 2024-08-08T20:43:01.2416655Z Device Type: GPU 2024-08-08T20:43:01.2417376Z Cache Info: 2024-08-08T20:43:01.2417992Z L1: 16(0x10) KB 2024-08-08T20:43:01.2418701Z L2: 8192(0x2000) KB 2024-08-08T20:43:01.2419428Z Chip ID: 29711(0x740f) 2024-08-08T20:43:01.2420231Z ASIC Revision: 1(0x1) 2024-08-08T20:43:01.2421053Z Cacheline Size: 64(0x40) 2024-08-08T20:43:01.2421492Z Max Clock Freq. (MHz): 1700 2024-08-08T20:43:01.2421868Z BDFID: 33536 2024-08-08T20:43:01.2422385Z Internal Node ID: 3 2024-08-08T20:43:01.2422738Z Compute Unit: 104 2024-08-08T20:43:01.2423085Z SIMDs per CU: 4 2024-08-08T20:43:01.2423436Z Shader Engines: 8 2024-08-08T20:43:01.2423800Z Shader Arrs. per Eng.: 1 2024-08-08T20:43:01.2424176Z WatchPts on Addr. Ranges:4 2024-08-08T20:43:01.2424556Z Coherent Host Access: FALSE 2024-08-08T20:43:01.2425058Z Features: KERNEL_DISPATCH 2024-08-08T20:43:01.2425403Z Fast F16 Operation: TRUE 2024-08-08T20:43:01.2425774Z Wavefront Size: 64(0x40) 2024-08-08T20:43:01.2426145Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:43:01.2426518Z Workgroup Max Size per Dimension: 2024-08-08T20:43:01.2426900Z x 1024(0x400) 2024-08-08T20:43:01.2427564Z y 1024(0x400) 2024-08-08T20:43:01.2428232Z z 1024(0x400) 2024-08-08T20:43:01.2428980Z Max Waves Per CU: 32(0x20) 2024-08-08T20:43:01.2429734Z Max Work-item Per CU: 2048(0x800) 2024-08-08T20:43:01.2430429Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:43:01.2431049Z Grid Max Size per Dimension: 2024-08-08T20:43:01.2431576Z x 4294967295(0xffffffff) 2024-08-08T20:43:01.2432158Z y 4294967295(0xffffffff) 2024-08-08T20:43:01.2432737Z z 4294967295(0xffffffff) 2024-08-08T20:43:01.2433401Z Max fbarriers/Workgrp: 32 2024-08-08T20:43:01.2434168Z Packet Processor uCode:: 82 2024-08-08T20:43:01.2434899Z SDMA engine uCode:: 8 2024-08-08T20:43:01.2435615Z IOMMU Support:: None 2024-08-08T20:43:01.2436258Z Pool Info: 2024-08-08T20:43:01.2436734Z Pool 1 2024-08-08T20:43:01.2437327Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-08T20:43:01.2438029Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2438716Z Allocatable: TRUE 2024-08-08T20:43:01.2439435Z Alloc Granule: 4KB 2024-08-08T20:43:01.2440179Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2440931Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2441656Z Accessible by all: FALSE 2024-08-08T20:43:01.2442290Z Pool 2 2024-08-08T20:43:01.2442869Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-08T20:43:01.2443554Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2444226Z Allocatable: TRUE 2024-08-08T20:43:01.2445119Z Alloc Granule: 4KB 2024-08-08T20:43:01.2445857Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2446600Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2447334Z Accessible by all: FALSE 2024-08-08T20:43:01.2447963Z Pool 3 2024-08-08T20:43:01.2448527Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-08T20:43:01.2449191Z Size: 67092480(0x3ffc000) KB 2024-08-08T20:43:01.2450177Z Allocatable: TRUE 2024-08-08T20:43:01.2450884Z Alloc Granule: 4KB 2024-08-08T20:43:01.2451628Z Alloc Recommended Granule:2048KB 2024-08-08T20:43:01.2452376Z Alloc Alignment: 4KB 2024-08-08T20:43:01.2453101Z Accessible by all: FALSE 2024-08-08T20:43:01.2453725Z Pool 4 2024-08-08T20:43:01.2454268Z Segment: GROUP 2024-08-08T20:43:01.2455254Z Size: 64(0x40) KB 2024-08-08T20:43:01.2455931Z Allocatable: FALSE 2024-08-08T20:43:01.2456694Z Alloc Granule: 0KB 2024-08-08T20:43:01.2457563Z Alloc Recommended Granule:0KB 2024-08-08T20:43:01.2458456Z Alloc Alignment: 0KB 2024-08-08T20:43:01.2459313Z Accessible by all: FALSE 2024-08-08T20:43:01.2460053Z ISA Info: 2024-08-08T20:43:01.2460601Z ISA 1 2024-08-08T20:43:01.2461291Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-08T20:43:01.2461817Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-08T20:43:01.2462232Z Profiles: HSA_PROFILE_BASE 2024-08-08T20:43:01.2462615Z Default Rounding Mode: NEAR 2024-08-08T20:43:01.2463004Z Default Rounding Mode: NEAR 2024-08-08T20:43:01.2463364Z Fast f16: TRUE 2024-08-08T20:43:01.2463727Z Workgroup Max Size: 1024(0x400) 2024-08-08T20:43:01.2464069Z Workgroup Max Size per Dimension: 2024-08-08T20:43:01.2464375Z x 1024(0x400) 2024-08-08T20:43:01.2464679Z y 1024(0x400) 2024-08-08T20:43:01.2464978Z z 1024(0x400) 2024-08-08T20:43:01.2465313Z Grid Max Size: 4294967295(0xffffffff) 2024-08-08T20:43:01.2465643Z Grid Max Size per Dimension: 2024-08-08T20:43:01.2465924Z x 4294967295(0xffffffff) 2024-08-08T20:43:01.2466231Z y 4294967295(0xffffffff) 2024-08-08T20:43:01.2466543Z z 4294967295(0xffffffff) 2024-08-08T20:43:01.2466937Z FBarrier Max Size: 32 2024-08-08T20:43:01.2467479Z *** Done *** 2024-08-08T20:43:01.2501444Z + rocminfo 2024-08-08T20:43:01.2509666Z + grep -E 'Name:.*\sgfx|Marketing' 2024-08-08T20:43:01.3135817Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.3136773Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-08T20:43:01.3137552Z Name: gfx90a 2024-08-08T20:43:01.3138416Z Marketing Name: AMD Instinct MI210 2024-08-08T20:43:01.3139234Z Name: gfx90a 2024-08-08T20:43:01.3140045Z Marketing Name: AMD Instinct MI210 2024-08-08T20:43:01.3306883Z + [[ linux-focal-rocm6.1-py3.8 == *xpu* ]] 2024-08-08T20:43:01.3307761Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-08-08T20:43:01.3308442Z + pip_install --user ninja==1.10.2 2024-08-08T20:43:01.3309192Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-08T20:43:01.6955527Z Collecting ninja==1.10.2 2024-08-08T20:43:01.7175848Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-08T20:43:01.7303134Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-08T20:43:02.4043930Z Installing collected packages: ninja 2024-08-08T20:43:02.4124622Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-08T20:43:02.4126522Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-08T20:43:02.4290080Z Successfully installed ninja-1.10.2 2024-08-08T20:43:02.4839679Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-08T20:43:02.4843318Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-08T20:43:02.4845464Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-08-08T20:43:02.4846079Z + install_tlparse 2024-08-08T20:43:02.4846585Z + pip_install --user tlparse==0.3.7 2024-08-08T20:43:02.4847286Z + pip install --progress-bar off --user tlparse==0.3.7 2024-08-08T20:43:02.8462857Z Collecting tlparse==0.3.7 2024-08-08T20:43:02.8678403Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-08-08T20:43:02.8770827Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-08T20:43:03.5945641Z Installing collected packages: tlparse 2024-08-08T20:43:03.6292197Z Successfully installed tlparse-0.3.7 2024-08-08T20:43:03.6804909Z ++ python -m site --user-base 2024-08-08T20:43:03.6983509Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-08T20:43:03.6985814Z + [[ linux-focal-rocm6.1-py3.8 == *asan* ]] 2024-08-08T20:43:03.6986478Z + [[ linux-focal-rocm6.1-py3.8 == *-debug* ]] 2024-08-08T20:43:03.6987126Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-08-08T20:43:03.6988036Z + echo 'We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass' 2024-08-08T20:43:03.6989161Z We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass 2024-08-08T20:43:03.6990021Z + cd test 2024-08-08T20:43:03.6990670Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-08T20:43:05.2964176Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-08T20:43:05.2964999Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-08T20:43:05.2971069Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-08T20:43:05.2971665Z + [[ distributed == *dynamo_eager* ]] 2024-08-08T20:43:05.2972175Z + [[ distributed == *aot_eager* ]] 2024-08-08T20:43:05.2972548Z + [[ distributed == *aot_inductor* ]] 2024-08-08T20:43:05.2972793Z + [[ distributed == *inductor* ]] 2024-08-08T20:43:05.2973017Z + [[ distributed == *dynamic* ]] 2024-08-08T20:43:05.2973253Z + [[ distributed == *cpu* ]] 2024-08-08T20:43:05.2973506Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-08T20:43:05.2997128Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-08-08T20:43:05.2997455Z + [[ linux-focal-rocm6.1-py3.8 == *-bazel-* ]] 2024-08-08T20:43:05.3001711Z + cd test 2024-08-08T20:43:05.3002033Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-08T20:43:07.4254479Z PyTorch built with: 2024-08-08T20:43:07.4255076Z - GCC 9.4 2024-08-08T20:43:07.4255544Z - C++ Version: 201703 2024-08-08T20:43:07.4256562Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-08T20:43:07.4257826Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-08T20:43:07.4259287Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-08T20:43:07.4259941Z - LAPACK is enabled (usually provided by MKL) 2024-08-08T20:43:07.4260553Z - NNPACK is enabled 2024-08-08T20:43:07.4261038Z - CPU capability usage: AVX2 2024-08-08T20:43:07.4261550Z - HIP Runtime 6.1.40091 2024-08-08T20:43:07.4262017Z - MIOpen 3.1.0 2024-08-08T20:43:07.4262436Z - Magma 2.7.2 2024-08-08T20:43:07.4270719Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-08T20:43:07.4278844Z 2024-08-08T20:43:07.8819969Z + cd test 2024-08-08T20:43:07.8820757Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-08T20:43:09.0330009Z ATen/Parallel: 2024-08-08T20:43:09.0330607Z at::get_num_threads() : 64 2024-08-08T20:43:09.0331191Z at::get_num_interop_threads() : 64 2024-08-08T20:43:09.0331786Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-08T20:43:09.0332360Z omp_get_max_threads() : 64 2024-08-08T20:43:09.0333425Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-08T20:43:09.0334461Z mkl_get_max_threads() : 64 2024-08-08T20:43:09.0335172Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-08T20:43:09.0335963Z std::thread::hardware_concurrency() : 64 2024-08-08T20:43:09.0336545Z Environment variables: 2024-08-08T20:43:09.0337029Z OMP_NUM_THREADS : [not set] 2024-08-08T20:43:09.0337529Z MKL_NUM_THREADS : [not set] 2024-08-08T20:43:09.0338038Z ATen parallel backend: OpenMP 2024-08-08T20:43:09.0338373Z 2024-08-08T20:43:09.4643563Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-08-08T20:43:09.4644313Z + [[ distributed == *backward* ]] 2024-08-08T20:43:09.4644992Z + [[ distributed == *xla* ]] 2024-08-08T20:43:09.4645574Z + [[ distributed == *executorch* ]] 2024-08-08T20:43:09.4646157Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-08T20:43:09.4646799Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-08-08T20:43:09.4647414Z + [[ distributed == distributed ]] 2024-08-08T20:43:09.4647710Z + test_distributed 2024-08-08T20:43:09.4647963Z + echo 'Testing distributed python tests' 2024-08-08T20:43:09.4648272Z Testing distributed python tests 2024-08-08T20:43:09.4649254Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-08-08T20:43:09.5473188Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-08T20:43:09.5474499Z import pkg_resources 2024-08-08T20:43:13.2860264Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-08-08T20:43:13.7648117Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-08T20:43:14.3086392Z Ignoring disabled issues: [''] 2024-08-08T20:43:14.3297744Z Found test times from artifacts 2024-08-08T20:43:14.3808719Z Found test times from artifacts 2024-08-08T20:43:14.3821371Z Running all tests 2024-08-08T20:43:14.3824731Z Running parallel tests on 2 processes 2024-08-08T20:43:14.3825419Z Name: tests to run (est. time: 26.73min) 2024-08-08T20:43:14.3826025Z Serial tests (4): 2024-08-08T20:43:14.3826523Z distributed/test_c10d_common 1/1 2024-08-08T20:43:14.3827124Z distributed/test_c10d_nccl 1/3 2024-08-08T20:43:14.3828255Z distributed/test_c10d_nccl 2/3 2024-08-08T20:43:14.3828805Z distributed/test_c10d_nccl 3/3 2024-08-08T20:43:14.3829327Z Parallel tests (0): 2024-08-08T20:43:14.3829804Z Name: excluded (est. time: 0.0min) 2024-08-08T20:43:14.3830340Z Serial tests (0): 2024-08-08T20:43:14.3830783Z Parallel tests (0): 2024-08-08T20:43:14.3831465Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-08T20:43:14.4083751Z Running distributed/test_c10d_common 1/1 ... [2024-08-08 20:43:14.407766] 2024-08-08T20:43:14.4090955Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-08 20:43:14.408521] 2024-08-08T20:46:08.7875451Z 2024-08-08T20:46:08.7876420Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_fa574fd2589a55d6_.log 2024-08-08T20:46:08.7891937Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-08-08T20:46:08.7900070Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-08-08T20:46:08.7900862Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-08-08T20:46:08.7902167Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-08-08T20:46:08.7903025Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-08-08T20:46:08.7903887Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-08-08T20:46:08.7904651Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-08-08T20:46:08.7905617Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-08-08T20:46:08.7907170Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-08-08T20:46:08.7908831Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-08-08T20:46:08.7910439Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-08-08T20:46:08.7912262Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-08-08T20:46:08.7914128Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-08-08T20:46:08.7915750Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-08-08T20:46:08.7917585Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-08-08T20:46:08.7919720Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-08-08T20:46:08.7921558Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-08-08T20:46:08.7922976Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-08-08T20:46:08.7924332Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-08-08T20:46:08.7925800Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-08-08T20:46:08.7929961Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-08-08T20:46:08.7931537Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-08-08T20:46:08.7932983Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-08-08T20:46:08.7934362Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-08-08T20:46:08.7935179Z 2024-08-08T20:46:08.7935575Z Running distributed/test_c10d_nccl 1/3 ... [2024-08-08 20:46:08.788004] 2024-08-08T20:46:08.7937341Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-08 20:46:08.788345] 2024-08-08T21:00:02.8500760Z 2024-08-08T21:00:02.8508814Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_771af61b8af102a7_.log 2024-08-08T21:00:02.8536018Z Running 69 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-08-08T21:00:02.8563856Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-08-08T21:00:02.8564615Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-08-08T21:00:02.8565599Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-08-08T21:00:02.8566439Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-08-08T21:00:02.8567660Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-08-08T21:00:02.8569047Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg 2024-08-08T21:00:02.8570192Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-08-08T21:00:02.8571422Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization 2024-08-08T21:00:02.8572669Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-08-08T21:00:02.8573946Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-08-08T21:00:02.8575232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-08-08T21:00:02.8576452Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-08-08T21:00:02.8578273Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-08-08T21:00:02.8579866Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-08-08T21:00:02.8581171Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-08-08T21:00:02.8582249Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-08-08T21:00:02.8583237Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-08-08T21:00:02.8584247Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-08-08T21:00:02.8585264Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-08-08T21:00:02.8586148Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-08-08T21:00:02.8587045Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-08-08T21:00:02.8588647Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-08-08T21:00:02.8590198Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-08-08T21:00:02.8591649Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-08-08T21:00:02.8593187Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-08-08T21:00:02.8594793Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-08-08T21:00:02.8596136Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-08-08T21:00:02.8597343Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-08-08T21:00:02.8601245Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-08-08T21:00:02.8602698Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-08-08T21:00:02.8604144Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-08-08T21:00:02.8605560Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-08-08T21:00:02.8606991Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-08-08T21:00:02.8608786Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-08-08T21:00:02.8610531Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-08-08T21:00:02.8611957Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-08-08T21:00:02.8612891Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-08-08T21:00:02.8613969Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-08-08T21:00:02.8614802Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-08-08T21:00:02.8615574Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-08-08T21:00:02.8616316Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-08-08T21:00:02.8617199Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-08-08T21:00:02.8618034Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-08-08T21:00:02.8619034Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-08-08T21:00:02.8620026Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-08-08T21:00:02.8620832Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-08-08T21:00:02.8621573Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-08-08T21:00:02.8622300Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-08-08T21:00:02.8623066Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-08-08T21:00:02.8623842Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-08-08T21:00:02.8624590Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-08-08T21:00:02.8625446Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-08-08T21:00:02.8626462Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-08-08T21:00:02.8627563Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-08-08T21:00:02.8628477Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-08-08T21:00:02.8629178Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-08-08T21:00:02.8629870Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-08-08T21:00:02.8630547Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-08-08T21:00:02.8631270Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-08-08T21:00:02.8631963Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-08-08T21:00:02.8632748Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-08-08T21:00:02.8633508Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-08-08T21:00:02.8634307Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-08-08T21:00:02.8635278Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-08-08T21:00:02.8636380Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-08-08T21:00:02.8637477Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-08-08T21:00:02.8638905Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-08-08T21:00:02.8640190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-08-08T21:00:02.8641668Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-08-08T21:00:02.8642460Z 2024-08-08T21:00:02.8642770Z Running distributed/test_c10d_nccl 2/3 ... [2024-08-08 21:00:02.851832] 2024-08-08T21:00:02.8644409Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-08 21:00:02.852232] 2024-08-08T21:10:29.8967266Z 2024-08-08T21:10:29.8969025Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_664ea6a0c573ac6d_.log 2024-08-08T21:10:29.9002730Z Running 61 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-08-08T21:10:29.9024561Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-08-08T21:10:29.9025349Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-08-08T21:10:29.9026150Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-08-08T21:10:29.9026989Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-08-08T21:10:29.9028012Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-08-08T21:10:29.9028807Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-08-08T21:10:29.9029628Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-08-08T21:10:29.9030477Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-08-08T21:10:29.9031554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-08-08T21:10:29.9032525Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-08-08T21:10:29.9033417Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-08-08T21:10:29.9034352Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-08-08T21:10:29.9035306Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-08-08T21:10:29.9036271Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-08-08T21:10:29.9037294Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-08-08T21:10:29.9038315Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-08-08T21:10:29.9039323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-08-08T21:10:29.9040301Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-08-08T21:10:29.9041226Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-08-08T21:10:29.9042078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-08-08T21:10:29.9042913Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-08-08T21:10:29.9043833Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-08-08T21:10:29.9044824Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-08-08T21:10:29.9045676Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-08-08T21:10:29.9046566Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-08-08T21:10:29.9047542Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-08-08T21:10:29.9048582Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-08-08T21:10:29.9049609Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-08-08T21:10:29.9050503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-08-08T21:10:29.9051566Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-08-08T21:10:29.9052424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-08-08T21:10:29.9053217Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-08-08T21:10:29.9054044Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-08-08T21:10:29.9054975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-08-08T21:10:29.9055746Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-08-08T21:10:29.9056476Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-08-08T21:10:29.9057192Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-08-08T21:10:29.9057851Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-08-08T21:10:29.9058508Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-08-08T21:10:29.9059218Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-08-08T21:10:29.9059929Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-08-08T21:10:29.9060664Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-08-08T21:10:29.9061393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-08-08T21:10:29.9062180Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-08-08T21:10:29.9063184Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-08-08T21:10:29.9064084Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-08-08T21:10:29.9064933Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-08-08T21:10:29.9065716Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-08-08T21:10:29.9066457Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-08-08T21:10:29.9067237Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-08-08T21:10:29.9068117Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-08-08T21:10:29.9069084Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-08-08T21:10:29.9070054Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-08-08T21:10:29.9070996Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-08-08T21:10:29.9071900Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-08-08T21:10:29.9073102Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-08-08T21:10:29.9074749Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-08-08T21:10:29.9076183Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-08-08T21:10:29.9077614Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-08-08T21:10:29.9079275Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestTimeoutDumpOnStuckRanks::test_timeout_dumps_on_stuck_ranks 2024-08-08T21:10:29.9080717Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-08-08T21:10:29.9081481Z 2024-08-08T21:10:29.9081791Z Running distributed/test_c10d_nccl 3/3 ... [2024-08-08 21:10:29.898902] 2024-08-08T21:10:29.9083228Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-08 21:10:29.899310] 2024-08-08T21:22:05.9859851Z 2024-08-08T21:22:05.9861451Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_99b8ba91cbb7cca4_.log 2024-08-08T21:22:05.9890873Z Running 59 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True, test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-08-08T21:22:05.9912908Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-08-08T21:22:05.9913752Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-08-08T21:22:05.9914519Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-08-08T21:22:05.9915688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-08-08T21:22:05.9917292Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-08-08T21:22:05.9918643Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-08-08T21:22:05.9920081Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-08-08T21:22:05.9921510Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-08-08T21:22:05.9923019Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-08-08T21:22:05.9924327Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-08-08T21:22:05.9925839Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-08-08T21:22:05.9927437Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-08-08T21:22:05.9929073Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-08-08T21:22:05.9930647Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-08-08T21:22:05.9932150Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-08-08T21:22:05.9933672Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-08-08T21:22:05.9935402Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-08-08T21:22:05.9936848Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-08-08T21:22:05.9937713Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-08-08T21:22:05.9938505Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-08-08T21:22:05.9939421Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-08-08T21:22:05.9940397Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-08-08T21:22:05.9941375Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-08-08T21:22:05.9942390Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-08-08T21:22:05.9943328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-08-08T21:22:05.9944233Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-08-08T21:22:05.9945331Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-08-08T21:22:05.9946751Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-08-08T21:22:05.9947909Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-08-08T21:22:05.9949254Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-08-08T21:22:05.9950401Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-08-08T21:22:05.9951588Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-08-08T21:22:05.9952740Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-08-08T21:22:05.9954166Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-08-08T21:22:05.9955397Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-08-08T21:22:05.9956479Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-08-08T21:22:05.9957535Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-08-08T21:22:05.9958800Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-08-08T21:22:05.9960403Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-08-08T21:22:05.9962005Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-08-08T21:22:05.9963631Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-08-08T21:22:05.9965165Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-08-08T21:22:05.9966383Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-08-08T21:22:05.9967554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-08-08T21:22:05.9968650Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-08-08T21:22:05.9969817Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0 2024-08-08T21:22:05.9971125Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1 2024-08-08T21:22:05.9972509Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-08-08T21:22:05.9973944Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-08-08T21:22:05.9975475Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-08-08T21:22:05.9976879Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-08-08T21:22:05.9977833Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-08-08T21:22:05.9978780Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-08-08T21:22:05.9979719Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-08-08T21:22:05.9980606Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-08-08T21:22:05.9981589Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-08-08T21:22:05.9982450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_False 2024-08-08T21:22:05.9983377Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTestDumpOnTimeout::test_timeout_dumps_timing_enabled_True 2024-08-08T21:22:05.9984191Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorDumpTest::test_nccl_errors_dump 2024-08-08T21:22:05.9984801Z 2024-08-08T21:22:07.2971670Z 2024-08-08T21:22:07.2972019Z real 38m57.833s 2024-08-08T21:22:07.2972310Z user 53m56.886s 2024-08-08T21:22:07.2972563Z sys 57m6.284s 2024-08-08T21:22:07.2972797Z + assert_git_not_dirty 2024-08-08T21:22:07.2973125Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-08-08T21:22:07.2973469Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-08T21:22:07.2973825Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-08T21:22:07.2974117Z + [[ 1 == 1 ]] 2024-08-08T21:22:07.2974355Z + echo 'Testing distributed C++ tests' 2024-08-08T21:22:07.2974661Z Testing distributed C++ tests 2024-08-08T21:22:07.2976781Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:07.3011399Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:07.3035386Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:07.3036552Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:07.3037486Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-08-08T21:22:07.3848743Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-08T21:22:07.3850076Z import pkg_resources 2024-08-08T21:22:11.1717354Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-08-08T21:22:11.1719505Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-08T21:22:11.1801283Z Found test times from artifacts 2024-08-08T21:22:11.2155096Z Found test times from artifacts 2024-08-08T21:22:11.2168185Z Running all tests 2024-08-08T21:22:11.2173794Z Running parallel tests on 2 processes 2024-08-08T21:22:11.2174528Z Name: tests to run (est. time: 0.0min) 2024-08-08T21:22:11.2175106Z Serial tests (1): 2024-08-08T21:22:11.2175610Z cpp/FileStoreTest 1/1 2024-08-08T21:22:11.2176114Z Parallel tests (0): 2024-08-08T21:22:11.2176597Z Name: excluded (est. time: 0.0min) 2024-08-08T21:22:11.2177123Z Serial tests (0): 2024-08-08T21:22:11.2177557Z Parallel tests (0): 2024-08-08T21:22:11.2178221Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-08T21:22:11.2428652Z Running cpp/FileStoreTest 1/1 ... [2024-08-08 21:22:11.242339] 2024-08-08T21:22:11.2441730Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e03ea4a9bf1eca81.xml', '-x', '--reruns=2'] ... [2024-08-08 21:22:11.243601] 2024-08-08T21:22:13.7271561Z 2024-08-08T21:22:13.7272366Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_cc05a842fc61f928_.log 2024-08-08T21:22:13.7272981Z 2024-08-08T21:22:15.4619085Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-08-08T21:22:15.5452761Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-08T21:22:15.5454078Z import pkg_resources 2024-08-08T21:22:19.2767785Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-08-08T21:22:19.2770537Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-08T21:22:19.2851713Z Found test times from artifacts 2024-08-08T21:22:19.3219606Z Found test times from artifacts 2024-08-08T21:22:19.3234133Z Running all tests 2024-08-08T21:22:19.3243949Z Running parallel tests on 2 processes 2024-08-08T21:22:19.3245241Z Name: tests to run (est. time: 0.0min) 2024-08-08T21:22:19.3246127Z Serial tests (1): 2024-08-08T21:22:19.3246860Z cpp/HashStoreTest 1/1 2024-08-08T21:22:19.3247654Z Parallel tests (0): 2024-08-08T21:22:19.3248402Z Name: excluded (est. time: 0.0min) 2024-08-08T21:22:19.3249201Z Serial tests (0): 2024-08-08T21:22:19.3249881Z Parallel tests (0): 2024-08-08T21:22:19.3250723Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-08T21:22:19.3493072Z Running cpp/HashStoreTest 1/1 ... [2024-08-08 21:22:19.348843] 2024-08-08T21:22:19.3505806Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-20fa308a7cbe5e17.xml', '-x', '--reruns=2'] ... [2024-08-08 21:22:19.349993] 2024-08-08T21:22:21.6828764Z 2024-08-08T21:22:21.6830403Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_645506bade0c4446_.log 2024-08-08T21:22:21.6831969Z 2024-08-08T21:22:23.6339927Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-08-08T21:22:23.7169070Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-08T21:22:23.7170799Z import pkg_resources 2024-08-08T21:22:27.4907448Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-08-08T21:22:27.4909820Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-08T21:22:27.4992158Z Found test times from artifacts 2024-08-08T21:22:27.5363855Z Found test times from artifacts 2024-08-08T21:22:27.5376673Z Running all tests 2024-08-08T21:22:27.5379257Z Running parallel tests on 2 processes 2024-08-08T21:22:27.5380217Z Name: tests to run (est. time: 0.0min) 2024-08-08T21:22:27.5381010Z Serial tests (1): 2024-08-08T21:22:27.5381735Z cpp/TCPStoreTest 1/1 2024-08-08T21:22:27.5382510Z Parallel tests (0): 2024-08-08T21:22:27.5383234Z Name: excluded (est. time: 0.0min) 2024-08-08T21:22:27.5383926Z Serial tests (0): 2024-08-08T21:22:27.5384648Z Parallel tests (0): 2024-08-08T21:22:27.5385556Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-08T21:22:27.5612692Z Running cpp/TCPStoreTest 1/1 ... [2024-08-08 21:22:27.560750] 2024-08-08T21:22:27.5628129Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5eac546d21a7970d.xml', '-x', '--reruns=2'] ... [2024-08-08 21:22:27.562219] 2024-08-08T21:22:41.4887065Z 2024-08-08T21:22:41.4889243Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_6bd00598bd138e93_.log 2024-08-08T21:22:41.4891450Z 2024-08-08T21:22:42.7226557Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-08T21:22:42.7227837Z + [[ 1 == 1 ]] 2024-08-08T21:22:42.7228618Z + test_rpc 2024-08-08T21:22:42.7229798Z + echo 'Testing RPC C++ tests' 2024-08-08T21:22:42.7230679Z Testing RPC C++ tests 2024-08-08T21:22:42.7235961Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:42.7271127Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:42.7301648Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-08T21:22:42.7303367Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-08-08T21:22:42.8113428Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-08T21:22:42.8114952Z import pkg_resources 2024-08-08T21:22:46.5529892Z Downloading https://ossci-metrics.s3.amazonaws.com/slow-tests.json to /var/lib/jenkins/pytorch/test/.pytorch-slow-tests.json 2024-08-08T21:22:46.5532404Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-08T21:22:46.5614434Z Found test times from artifacts 2024-08-08T21:22:46.5967770Z Found test times from artifacts 2024-08-08T21:22:46.5980000Z Running all tests 2024-08-08T21:22:46.5983494Z Running parallel tests on 2 processes 2024-08-08T21:22:46.5984332Z Name: tests to run (est. time: 0.0min) 2024-08-08T21:22:46.5985344Z Serial tests (1): 2024-08-08T21:22:46.5986017Z cpp/test_cpp_rpc 1/1 2024-08-08T21:22:46.5986795Z Parallel tests (0): 2024-08-08T21:22:46.5987574Z Name: excluded (est. time: 0.0min) 2024-08-08T21:22:46.5988311Z Serial tests (0): 2024-08-08T21:22:46.5989057Z Parallel tests (0): 2024-08-08T21:22:46.5989938Z Starting test batch 'tests to run' 0.0 seconds after initiating testing 2024-08-08T21:22:46.6231740Z Running cpp/test_cpp_rpc 1/1 ... [2024-08-08 21:22:46.622637] 2024-08-08T21:22:46.6242969Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7997f29f0f19f274.xml', '-x', '--reruns=2'] ... [2024-08-08 21:22:46.623764] 2024-08-08T21:22:55.0212135Z 2024-08-08T21:22:55.0214220Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_a5e8d3f05cf51133_.log 2024-08-08T21:22:55.0215538Z 2024-08-08T21:22:56.2594057Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-08-08T21:22:56.2594969Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-08-08T21:22:56.2595986Z docker exec -t "aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-08-08T21:22:56.2621751Z shell: /usr/bin/bash -e {0} 2024-08-08T21:22:56.2622201Z env: 2024-08-08T21:22:56.2622551Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:56.2622948Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:56.2623663Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:56.2624510Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:56.2624911Z AWS_REGION: us-east-1 2024-08-08T21:22:56.2625369Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:56.2626100Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:56.2633538Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:56.2634328Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:56.2635235Z ##[endgroup] 2024-08-08T21:22:56.3055739Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-08T21:22:56.3304557Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-08T21:22:56.3304870Z cat test/**/*_toprint.log || true 2024-08-08T21:22:56.3321313Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T21:22:56.3321605Z env: 2024-08-08T21:22:56.3321779Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:56.3322021Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:56.3322447Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:56.3322836Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:56.3323056Z AWS_REGION: us-east-1 2024-08-08T21:22:56.3323306Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:56.3323599Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:56.3328276Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:56.3328606Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:56.3328961Z ##[endgroup] 2024-08-08T21:22:56.3424482Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-08T21:22:56.3448942Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-08T21:22:56.3449267Z kill "$MONITOR_SCRIPT_PID" 2024-08-08T21:22:56.3462020Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T21:22:56.3462312Z env: 2024-08-08T21:22:56.3462486Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:56.3462727Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:56.3463165Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:56.3463565Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:56.3463787Z AWS_REGION: us-east-1 2024-08-08T21:22:56.3464025Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:56.3464321Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:56.3468908Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:56.3469243Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:56.3469613Z MONITOR_SCRIPT_PID: 2898806 2024-08-08T21:22:56.3469836Z ##[endgroup] 2024-08-08T21:22:56.3584558Z Prepare all required actions 2024-08-08T21:22:56.3584901Z Getting action download info 2024-08-08T21:22:56.8009354Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-08T21:22:57.4208653Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-08T21:22:57.4208928Z with: 2024-08-08T21:22:57.4209101Z use-gha: true 2024-08-08T21:22:57.4209370Z file-suffix: test-distributed-1-1-linux.rocm.gpu_28538197052 2024-08-08T21:22:57.4209679Z s3-bucket: gha-artifacts 2024-08-08T21:22:57.4209885Z env: 2024-08-08T21:22:57.4210056Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:57.4210301Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:57.4210732Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:57.4211132Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:57.4211364Z AWS_REGION: us-east-1 2024-08-08T21:22:57.4211646Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:57.4211950Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:57.4216587Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:57.4216929Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:57.4217293Z ##[endgroup] 2024-08-08T21:22:57.4272923Z ##[group]Run actions/upload-artifact@v3 2024-08-08T21:22:57.4273175Z with: 2024-08-08T21:22:57.4273714Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip 2024-08-08T21:22:57.4274096Z retention-days: 14 2024-08-08T21:22:57.4274300Z if-no-files-found: warn 2024-08-08T21:22:57.4274512Z path: test/**/*.json 2024-08-08T21:22:57.4274699Z env: 2024-08-08T21:22:57.4274865Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:57.4275097Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:57.4275507Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:57.4275891Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:57.4276112Z AWS_REGION: us-east-1 2024-08-08T21:22:57.4276347Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:57.4276634Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:57.4281229Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:57.4281565Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:57.4281919Z ##[endgroup] 2024-08-08T21:22:57.8281221Z With the provided path, there will be 9 files uploaded 2024-08-08T21:22:57.8283190Z Starting artifact upload 2024-08-08T21:22:57.8285169Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-08T21:22:57.8287025Z Artifact name is valid! 2024-08-08T21:22:58.1594920Z Container for artifact "test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip" successfully created. Starting upload of file(s) 2024-08-08T21:22:58.9748258Z Total size of all the files uploaded is 35790 bytes 2024-08-08T21:22:58.9749202Z File upload process has finished. Finalizing the artifact upload 2024-08-08T21:22:59.0512914Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-08T21:22:59.0513551Z 2024-08-08T21:22:59.0513955Z The raw size of all the files that were specified for upload is 331810 bytes 2024-08-08T21:22:59.0515428Z The size of all the files that were uploaded is 35790 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-08T21:22:59.0516530Z 2024-08-08T21:22:59.0517664Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-08T21:22:59.0518961Z 2024-08-08T21:22:59.0519781Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip has been successfully uploaded! 2024-08-08T21:22:59.0726493Z ##[group]Run actions/upload-artifact@v3 2024-08-08T21:22:59.0727611Z with: 2024-08-08T21:22:59.0728458Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip 2024-08-08T21:22:59.0729394Z retention-days: 14 2024-08-08T21:22:59.0729874Z if-no-files-found: ignore 2024-08-08T21:22:59.0730395Z path: test/**/*.xml test/**/*.csv 2024-08-08T21:22:59.0730928Z env: 2024-08-08T21:22:59.0731321Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:22:59.0731929Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:22:59.0732943Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:22:59.0733892Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:22:59.0734412Z AWS_REGION: us-east-1 2024-08-08T21:22:59.0734992Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:22:59.0735687Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:22:59.0746969Z AWS_SESSION_TOKEN: *** 2024-08-08T21:22:59.0747769Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:22:59.0748643Z ##[endgroup] 2024-08-08T21:22:59.5185493Z With the provided path, there will be 216 files uploaded 2024-08-08T21:22:59.5186639Z Starting artifact upload 2024-08-08T21:22:59.5188444Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-08T21:22:59.5190887Z Artifact name is valid! 2024-08-08T21:22:59.8741408Z Container for artifact "test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip" successfully created. Starting upload of file(s) 2024-08-08T21:23:09.8801348Z Total file count: 216 ---- Processed file #151 (69.9%) 2024-08-08T21:23:13.6536269Z Total size of all the files uploaded is 53791 bytes 2024-08-08T21:23:13.6537211Z File upload process has finished. Finalizing the artifact upload 2024-08-08T21:23:13.7386569Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-08T21:23:13.7387269Z 2024-08-08T21:23:13.7387727Z The raw size of all the files that were specified for upload is 82794 bytes 2024-08-08T21:23:13.7389159Z The size of all the files that were uploaded is 53791 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-08T21:23:13.7390200Z 2024-08-08T21:23:13.7391359Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-08T21:23:13.7392662Z 2024-08-08T21:23:13.7393499Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip has been successfully uploaded! 2024-08-08T21:23:13.7621166Z ##[group]Run actions/upload-artifact@v3 2024-08-08T21:23:13.7621758Z with: 2024-08-08T21:23:13.7622458Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip 2024-08-08T21:23:13.7623304Z retention-days: 14 2024-08-08T21:23:13.7623784Z if-no-files-found: ignore 2024-08-08T21:23:13.7624355Z path: usage_log.txt test/**/*.log 2024-08-08T21:23:13.7624889Z env: 2024-08-08T21:23:13.7625287Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:13.7625862Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:13.7626868Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:13.7627804Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:13.7628350Z AWS_REGION: us-east-1 2024-08-08T21:23:13.7628921Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:13.7629616Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:13.7640930Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:13.7641711Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:13.7642563Z ##[endgroup] 2024-08-08T21:23:14.1952989Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-08-08T21:23:14.1955780Z The least common ancestor is /home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-08-08T21:23:14.1957147Z With the provided path, there will be 9 files uploaded 2024-08-08T21:23:14.1958128Z Starting artifact upload 2024-08-08T21:23:14.1959913Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-08T21:23:14.1961749Z Artifact name is valid! 2024-08-08T21:23:14.5493770Z Container for artifact "logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip" successfully created. Starting upload of file(s) 2024-08-08T21:23:15.3534743Z Total size of all the files uploaded is 50608 bytes 2024-08-08T21:23:15.3535686Z File upload process has finished. Finalizing the artifact upload 2024-08-08T21:23:15.4385762Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-08T21:23:15.4386422Z 2024-08-08T21:23:15.4386895Z The raw size of all the files that were specified for upload is 802501 bytes 2024-08-08T21:23:15.4388379Z The size of all the files that were uploaded is 50608 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-08T21:23:15.4389439Z 2024-08-08T21:23:15.4390582Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-08T21:23:15.4392492Z 2024-08-08T21:23:15.4393233Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_28538197052.zip has been successfully uploaded! 2024-08-08T21:23:15.4599680Z ##[group]Run # shellcheck disable=SC2156 2024-08-08T21:23:15.4600384Z # shellcheck disable=SC2156 2024-08-08T21:23:15.4601435Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-08T21:23:15.4637227Z shell: /usr/bin/bash -e {0} 2024-08-08T21:23:15.4637761Z env: 2024-08-08T21:23:15.4638167Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:15.4638755Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:15.4639775Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:15.4640717Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:15.4641239Z AWS_REGION: us-east-1 2024-08-08T21:23:15.4641846Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:15.4642568Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:15.4654080Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:15.4654923Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:15.4655899Z ##[endgroup] 2024-08-08T21:23:15.7455114Z Prepare all required actions 2024-08-08T21:23:15.7455943Z Getting action download info 2024-08-08T21:23:15.7507325Z ##[group]Run ./.github/actions/teardown-rocm 2024-08-08T21:23:15.7507918Z env: 2024-08-08T21:23:15.7508327Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:15.7508889Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:15.7509894Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:15.7510841Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:15.7511382Z AWS_REGION: us-east-1 2024-08-08T21:23:15.7512062Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:15.7512899Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:15.7524416Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:15.7525370Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:15.7526237Z ##[endgroup] 2024-08-08T21:23:15.7556837Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-08T21:23:15.7557842Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-08T21:23:15.7558604Z # shellcheck disable=SC2046 2024-08-08T21:23:15.7559228Z docker stop $(docker ps -q) || true 2024-08-08T21:23:15.7559860Z # Prune all stopped containers. 2024-08-08T21:23:15.7560453Z docker container prune -f 2024-08-08T21:23:15.7593053Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T21:23:15.7593909Z env: 2024-08-08T21:23:15.7594409Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:15.7595078Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:15.7596212Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:15.7597154Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:15.7597680Z AWS_REGION: us-east-1 2024-08-08T21:23:15.7598237Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:15.7598937Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:15.7610390Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:15.7611187Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:15.7612035Z ##[endgroup] 2024-08-08T21:23:16.3359260Z aa07fab653b1 2024-08-08T21:23:32.3848763Z Deleted Containers: 2024-08-08T21:23:32.3849611Z aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:32.3850234Z 2024-08-08T21:23:32.3850461Z Total reclaimed space: 6.663GB 2024-08-08T21:23:32.3906409Z Prepare all required actions 2024-08-08T21:23:32.3934295Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-08-08T21:23:32.3934619Z with: 2024-08-08T21:23:32.3934834Z diskspace-cutoff: 70 2024-08-08T21:23:32.3935072Z env: 2024-08-08T21:23:32.3935284Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:32.3935818Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:32.3936336Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:32.3936845Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:32.3937116Z AWS_REGION: us-east-1 2024-08-08T21:23:32.3937418Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:32.3937782Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:32.3945348Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:32.3945965Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:32.3946640Z ##[endgroup] 2024-08-08T21:23:32.3969917Z ##[group]Run set -ex 2024-08-08T21:23:32.3970329Z set -ex 2024-08-08T21:23:32.3970694Z diskspace_cutoff=70 2024-08-08T21:23:32.3971233Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-08-08T21:23:32.3972180Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-08T21:23:32.3973711Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-08T21:23:32.3975445Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-08-08T21:23:32.3976120Z  docker system prune -af 2024-08-08T21:23:32.3976915Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-08T21:23:32.3977551Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-08-08T21:23:32.3978119Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-08-08T21:23:32.3978616Z  echo "$msg" 2024-08-08T21:23:32.3978883Z  exit 1 2024-08-08T21:23:32.3979131Z  else 2024-08-08T21:23:32.3979414Z  difference=$((diskspace - diskspace_new)) 2024-08-08T21:23:32.3979802Z  echo "Diskspace saved: $difference percent" 2024-08-08T21:23:32.3980137Z  fi 2024-08-08T21:23:32.3980357Z fi 2024-08-08T21:23:32.4007156Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-08T21:23:32.4007718Z env: 2024-08-08T21:23:32.4008040Z GIT_DEFAULT_BRANCH: main 2024-08-08T21:23:32.4008492Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-08T21:23:32.4009282Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-08T21:23:32.4010013Z AWS_DEFAULT_REGION: us-east-1 2024-08-08T21:23:32.4010430Z AWS_REGION: us-east-1 2024-08-08T21:23:32.4010873Z AWS_ACCESS_KEY_ID: *** 2024-08-08T21:23:32.4011422Z AWS_SECRET_ACCESS_KEY: *** 2024-08-08T21:23:32.4019605Z AWS_SESSION_TOKEN: *** 2024-08-08T21:23:32.4020023Z CONTAINER_NAME: aa07fab653b190e6165dccc85be33c248ac06c65c4fc7f9f855183d97a67baee 2024-08-08T21:23:32.4020471Z ##[endgroup] 2024-08-08T21:23:32.4077812Z + diskspace_cutoff=70 2024-08-08T21:23:32.4085940Z ++ docker info -f '{{.DockerRootDir}}' 2024-08-08T21:23:32.4582365Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-08-08T21:23:32.4590346Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-08-08T21:23:32.4591021Z ++ sed -n 2p 2024-08-08T21:23:32.4591366Z ++ sed s/%// 2024-08-08T21:23:32.4592236Z ++ sed 's/ //' 2024-08-08T21:23:32.4608533Z + diskspace=27 2024-08-08T21:23:32.4609036Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-08-08T21:23:32.4609540Z + [[ 27 -ge 70 ]] 2024-08-08T21:23:32.4642750Z Post job cleanup. 2024-08-08T21:23:32.4674004Z Post job cleanup. 2024-08-08T21:23:32.4997545Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-08T21:23:32.5321978Z Post job cleanup. 2024-08-08T21:23:32.6488397Z Post job cleanup. 2024-08-08T21:23:32.6570383Z Post job cleanup. 2024-08-08T21:23:32.6647954Z Post job cleanup. 2024-08-08T21:23:32.7300172Z [command]/usr/bin/git version 2024-08-08T21:23:32.7329572Z git version 2.34.1 2024-08-08T21:23:32.7364797Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/5d25a5e6-d75d-44ce-8090-5eb4da98a9f3' before making global git config changes 2024-08-08T21:23:32.7365503Z Adding repository directory to the temporary git global config as a safe directory 2024-08-08T21:23:32.7367973Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-08T21:23:32.7393214Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-08T21:23:32.7415996Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-08T21:23:32.7750301Z Entering 'android/libs/fbjni' 2024-08-08T21:23:32.7799987Z Entering 'third_party/FP16' 2024-08-08T21:23:32.7850943Z Entering 'third_party/FXdiv' 2024-08-08T21:23:32.7891361Z Entering 'third_party/NNPACK' 2024-08-08T21:23:32.7931749Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T21:23:32.7972479Z Entering 'third_party/XNNPACK' 2024-08-08T21:23:32.8026907Z Entering 'third_party/benchmark' 2024-08-08T21:23:32.8066335Z Entering 'third_party/cpp-httplib' 2024-08-08T21:23:32.8104587Z Entering 'third_party/cpuinfo' 2024-08-08T21:23:32.8144134Z Entering 'third_party/cudnn_frontend' 2024-08-08T21:23:32.8183892Z Entering 'third_party/cutlass' 2024-08-08T21:23:32.8243306Z Entering 'third_party/eigen' 2024-08-08T21:23:32.8296769Z Entering 'third_party/fbgemm' 2024-08-08T21:23:32.8337962Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T21:23:32.8384307Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T21:23:32.8425386Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T21:23:32.8470876Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T21:23:32.8510847Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T21:23:32.8561312Z Entering 'third_party/flatbuffers' 2024-08-08T21:23:32.8614723Z Entering 'third_party/fmt' 2024-08-08T21:23:32.8652268Z Entering 'third_party/foxi' 2024-08-08T21:23:32.8691558Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T21:23:32.8736482Z Entering 'third_party/gloo' 2024-08-08T21:23:32.8779502Z Entering 'third_party/googletest' 2024-08-08T21:23:32.8823749Z Entering 'third_party/ideep' 2024-08-08T21:23:32.8862991Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T21:23:32.8910354Z Entering 'third_party/ittapi' 2024-08-08T21:23:32.8955920Z Entering 'third_party/kineto' 2024-08-08T21:23:32.8997053Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T21:23:32.9041877Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T21:23:32.9090159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T21:23:32.9130866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T21:23:32.9171297Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T21:23:32.9210693Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T21:23:32.9258571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T21:23:32.9301567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T21:23:32.9340187Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T21:23:32.9383366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T21:23:32.9432264Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T21:23:32.9470136Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T21:23:32.9511994Z Entering 'third_party/mimalloc' 2024-08-08T21:23:32.9560387Z Entering 'third_party/nccl/nccl' 2024-08-08T21:23:32.9605691Z Entering 'third_party/nlohmann' 2024-08-08T21:23:32.9653927Z Entering 'third_party/onnx' 2024-08-08T21:23:32.9706410Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T21:23:32.9753682Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T21:23:32.9796777Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T21:23:32.9839176Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T21:23:32.9882164Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T21:23:32.9930372Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T21:23:32.9967233Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T21:23:33.0003891Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T21:23:33.0041309Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T21:23:33.0081495Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T21:23:33.0119541Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T21:23:33.0164975Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T21:23:33.0210798Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T21:23:33.0272639Z Entering 'third_party/pocketfft' 2024-08-08T21:23:33.0333435Z Entering 'third_party/protobuf' 2024-08-08T21:23:33.0375272Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T21:23:33.0422550Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T21:23:33.0467453Z Entering 'third_party/psimd' 2024-08-08T21:23:33.0507232Z Entering 'third_party/pthreadpool' 2024-08-08T21:23:33.0545736Z Entering 'third_party/pybind11' 2024-08-08T21:23:33.0584862Z Entering 'third_party/python-peachpy' 2024-08-08T21:23:33.0625353Z Entering 'third_party/sleef' 2024-08-08T21:23:33.0667424Z Entering 'third_party/tensorpipe' 2024-08-08T21:23:33.0708600Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T21:23:33.0748103Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T21:23:33.0795844Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T21:23:33.0840903Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T21:23:33.0880711Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T21:23:33.0947274Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-08T21:23:33.0963439Z http.https://github.com/.extraheader 2024-08-08T21:23:33.0971823Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-08T21:23:33.0995192Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-08T21:23:33.1262201Z Entering 'android/libs/fbjni' 2024-08-08T21:23:33.1293481Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1330258Z Entering 'third_party/FP16' 2024-08-08T21:23:33.1351943Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1386265Z Entering 'third_party/FXdiv' 2024-08-08T21:23:33.1409683Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1443774Z Entering 'third_party/NNPACK' 2024-08-08T21:23:33.1470812Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1506197Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-08T21:23:33.1529810Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1654783Z Entering 'third_party/XNNPACK' 2024-08-08T21:23:33.1675725Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1729500Z Entering 'third_party/benchmark' 2024-08-08T21:23:33.1748092Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1786395Z Entering 'third_party/cpp-httplib' 2024-08-08T21:23:33.1809079Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1839876Z Entering 'third_party/cpuinfo' 2024-08-08T21:23:33.1864395Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1894740Z Entering 'third_party/cudnn_frontend' 2024-08-08T21:23:33.1915027Z http.https://github.com/.extraheader 2024-08-08T21:23:33.1953508Z Entering 'third_party/cutlass' 2024-08-08T21:23:33.1973618Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2011299Z Entering 'third_party/eigen' 2024-08-08T21:23:33.2030656Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2063712Z Entering 'third_party/fbgemm' 2024-08-08T21:23:33.2090328Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2120821Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-08T21:23:33.2141301Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2314055Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-08T21:23:33.2335396Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2369147Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-08T21:23:33.2389140Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2424500Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-08T21:23:33.2445209Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2547938Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-08T21:23:33.2572182Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2604922Z Entering 'third_party/flatbuffers' 2024-08-08T21:23:33.2625282Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2661572Z Entering 'third_party/fmt' 2024-08-08T21:23:33.2696659Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2744584Z Entering 'third_party/foxi' 2024-08-08T21:23:33.2769490Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2800710Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-08T21:23:33.2832660Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2868182Z Entering 'third_party/gloo' 2024-08-08T21:23:33.2897651Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2930190Z Entering 'third_party/googletest' 2024-08-08T21:23:33.2949341Z http.https://github.com/.extraheader 2024-08-08T21:23:33.2987082Z Entering 'third_party/ideep' 2024-08-08T21:23:33.3010146Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3039456Z Entering 'third_party/ideep/mkl-dnn' 2024-08-08T21:23:33.3062717Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3110258Z Entering 'third_party/ittapi' 2024-08-08T21:23:33.3133494Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3164330Z Entering 'third_party/kineto' 2024-08-08T21:23:33.3188796Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3218692Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-08T21:23:33.3240555Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3279840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-08T21:23:33.3308946Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3346527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-08T21:23:33.3369484Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3397396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-08T21:23:33.3423360Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3462960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-08T21:23:33.3489636Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3516922Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-08T21:23:33.3540132Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3576999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-08T21:23:33.3597232Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3650461Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-08T21:23:33.3651945Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3690802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-08T21:23:33.3710986Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3746403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-08T21:23:33.3766581Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3812692Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-08T21:23:33.3832659Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3864368Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-08T21:23:33.3883705Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3924986Z Entering 'third_party/mimalloc' 2024-08-08T21:23:33.3952571Z http.https://github.com/.extraheader 2024-08-08T21:23:33.3984053Z Entering 'third_party/nccl/nccl' 2024-08-08T21:23:33.4015872Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4049761Z Entering 'third_party/nlohmann' 2024-08-08T21:23:33.4067151Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4103748Z Entering 'third_party/onnx' 2024-08-08T21:23:33.4129345Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4205861Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-08T21:23:33.4236982Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4266729Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-08T21:23:33.4290291Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4335223Z Entering 'third_party/opentelemetry-cpp' 2024-08-08T21:23:33.4373006Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4412282Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-08T21:23:33.4436011Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4470201Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-08T21:23:33.4489053Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4521439Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-08T21:23:33.4548005Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4583757Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-08T21:23:33.4609563Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4650614Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-08T21:23:33.4671419Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4705435Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-08T21:23:33.4735852Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4765586Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-08T21:23:33.4786468Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4817010Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-08T21:23:33.4837894Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4874033Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-08T21:23:33.4893771Z http.https://github.com/.extraheader 2024-08-08T21:23:33.4929548Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-08T21:23:33.4947021Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5001061Z Entering 'third_party/pocketfft' 2024-08-08T21:23:33.5024630Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5057469Z Entering 'third_party/protobuf' 2024-08-08T21:23:33.5081315Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5110352Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-08T21:23:33.5132201Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5172830Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-08T21:23:33.5193799Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5230893Z Entering 'third_party/psimd' 2024-08-08T21:23:33.5250445Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5282648Z Entering 'third_party/pthreadpool' 2024-08-08T21:23:33.5307579Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5334907Z Entering 'third_party/pybind11' 2024-08-08T21:23:33.5358673Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5396072Z Entering 'third_party/python-peachpy' 2024-08-08T21:23:33.5418654Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5452823Z Entering 'third_party/sleef' 2024-08-08T21:23:33.5478391Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5514972Z Entering 'third_party/tensorpipe' 2024-08-08T21:23:33.5535657Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5569665Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-08T21:23:33.5592606Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5625611Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-08T21:23:33.5649563Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5680997Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-08T21:23:33.5711379Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5742553Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-08T21:23:33.5764692Z http.https://github.com/.extraheader 2024-08-08T21:23:33.5796225Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-08T21:23:33.5817577Z http.https://github.com/.extraheader 2024-08-08T21:23:33.6034794Z Cleaning up orphan processes