2024-12-18T03:29:59.4566357Z Current runner version: '2.321.0' 2024-12-18T03:29:59.4571338Z Runner name: 'pytorch-rocm-hw-43' 2024-12-18T03:29:59.4572099Z Runner group name: 'linux.rocm.gpu.group' 2024-12-18T03:29:59.4572866Z Machine name: 'pytorch-rocm-hw-43' 2024-12-18T03:29:59.4575589Z ##[group]GITHUB_TOKEN Permissions 2024-12-18T03:29:59.4577260Z Contents: read 2024-12-18T03:29:59.4577864Z Metadata: read 2024-12-18T03:29:59.4578305Z ##[endgroup] 2024-12-18T03:29:59.4580793Z Secret source: Actions 2024-12-18T03:29:59.4581447Z Prepare workflow directory 2024-12-18T03:29:59.5420973Z Prepare all required actions 2024-12-18T03:29:59.5459731Z Getting action download info 2024-12-18T03:29:59.8411733Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T03:30:06.7763591Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-12-18T03:30:07.3292507Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-12-18T03:30:07.8807976Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-18T03:30:08.8800555Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T03:30:09.5755499Z Getting action download info 2024-12-18T03:30:09.7648739Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-18T03:30:10.3489944Z Getting action download info 2024-12-18T03:30:10.5302781Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-18T03:30:11.0498507Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T03:30:11.0500306Z ##[group] Inputs 2024-12-18T03:30:11.0500580Z build-environment: linux-focal-rocm6.2-py3.10 2024-12-18T03:30:11.0501375Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-12-18T03:30:11.0502335Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:30:11.0502844Z sync-tag: 2024-12-18T03:30:11.0503473Z timeout-minutes: 300 2024-12-18T03:30:11.0504063Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-12-18T03:30:11.0504968Z disable-monitor: true 2024-12-18T03:30:11.0505167Z ##[endgroup] 2024-12-18T03:30:11.0505458Z Complete job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:30:11.1498779Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-18T03:30:11.1499407Z with: 2024-12-18T03:30:11.1499566Z no-sudo: true 2024-12-18T03:30:11.1499736Z submodules: recursive 2024-12-18T03:30:11.1499916Z fetch-depth: 0 2024-12-18T03:30:11.1500203Z env: 2024-12-18T03:30:11.1500360Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:30:11.1500534Z ##[endgroup] 2024-12-18T03:30:11.1571251Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T03:30:11.1572002Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T03:30:11.1599877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:30:11.1600163Z env: 2024-12-18T03:30:11.1600326Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:30:11.1600501Z ##[endgroup] 2024-12-18T03:30:11.1911396Z ##[group]Run retry () { 2024-12-18T03:30:11.1911918Z retry () { 2024-12-18T03:30:11.1913011Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-18T03:30:11.1913635Z } 2024-12-18T03:30:11.1914013Z echo "${GITHUB_WORKSPACE}" 2024-12-18T03:30:11.1914526Z if [ -z "${NO_SUDO}" ]; then 2024-12-18T03:30:11.1915101Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T03:30:11.1915636Z else 2024-12-18T03:30:11.1916163Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T03:30:11.1916662Z fi 2024-12-18T03:30:11.1917035Z mkdir "${GITHUB_WORKSPACE}" 2024-12-18T03:30:11.1961665Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:30:11.1962304Z env: 2024-12-18T03:30:11.1962654Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:30:11.1963071Z NO_SUDO: true 2024-12-18T03:30:11.1963428Z ##[endgroup] 2024-12-18T03:30:11.2065582Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:30:17.5191146Z ##[group]Run malfet/checkout@silent-checkout 2024-12-18T03:30:17.5191642Z with: 2024-12-18T03:30:17.5191991Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:30:17.5192426Z fetch-depth: 0 2024-12-18T03:30:17.5192741Z submodules: recursive 2024-12-18T03:30:17.5193071Z quiet-checkout: true 2024-12-18T03:30:17.5193405Z repository: pytorch/pytorch 2024-12-18T03:30:17.5193933Z token: *** 2024-12-18T03:30:17.5194232Z ssh-strict: true 2024-12-18T03:30:17.5194552Z persist-credentials: true 2024-12-18T03:30:17.5194908Z clean: true 2024-12-18T03:30:17.5195232Z sparse-checkout-cone-mode: true 2024-12-18T03:30:17.5195603Z lfs: false 2024-12-18T03:30:17.5195896Z set-safe-directory: true 2024-12-18T03:30:17.5196255Z env: 2024-12-18T03:30:17.5196551Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:30:17.5196887Z ##[endgroup] 2024-12-18T03:30:17.5971015Z Syncing repository: pytorch/pytorch 2024-12-18T03:30:17.5972327Z ##[group]Getting Git version info 2024-12-18T03:30:17.5972795Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-12-18T03:30:17.5973406Z [command]/usr/bin/git version 2024-12-18T03:30:17.5973657Z git version 2.34.1 2024-12-18T03:30:17.5980714Z ##[endgroup] 2024-12-18T03:30:17.5991666Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/e3d8686f-2009-4d1a-a5e3-20eb973bf889' before making global git config changes 2024-12-18T03:30:17.5992403Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T03:30:17.5994995Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:30:17.6019887Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-12-18T03:30:17.6022313Z ##[group]Initializing the repository 2024-12-18T03:30:17.6024415Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:30:17.6048310Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-18T03:30:17.6049331Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-18T03:30:17.6050115Z hint: of your new repositories, which will suppress this warning, call: 2024-12-18T03:30:17.6050639Z hint: 2024-12-18T03:30:17.6051039Z hint: git config --global init.defaultBranch 2024-12-18T03:30:17.6051497Z hint: 2024-12-18T03:30:17.6051926Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-18T03:30:17.6052649Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-18T03:30:17.6053204Z hint: 2024-12-18T03:30:17.6053481Z hint: git branch -m 2024-12-18T03:30:17.6054156Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-18T03:30:17.6060256Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-18T03:30:17.6082608Z ##[endgroup] 2024-12-18T03:30:17.6083228Z ##[group]Disabling automatic garbage collection 2024-12-18T03:30:17.6084326Z [command]/usr/bin/git config --local gc.auto 0 2024-12-18T03:30:17.6106136Z ##[endgroup] 2024-12-18T03:30:17.6106648Z ##[group]Setting up auth 2024-12-18T03:30:17.6109760Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T03:30:17.6130361Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T03:30:17.6389782Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T03:30:17.6412725Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T03:30:17.6655292Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T03:30:17.6685020Z ##[endgroup] 2024-12-18T03:30:17.6685349Z ##[group]Fetching the repository 2024-12-18T03:30:17.6689882Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-18T03:30:23.7699591Z remote: Enumerating objects: 1055511 2024-12-18T03:30:23.7700632Z remote: Enumerating objects: 1056587, done. 2024-12-18T03:30:23.7701274Z remote: Counting objects: 0% (1/1076) 2024-12-18T03:30:23.7701992Z remote: Counting objects: 1% (11/1076) 2024-12-18T03:30:23.7702787Z remote: Counting objects: 2% (22/1076) 2024-12-18T03:30:23.7703498Z remote: Counting objects: 3% (33/1076) 2024-12-18T03:30:23.7704176Z remote: Counting objects: 4% (44/1076) 2024-12-18T03:30:23.7704834Z remote: Counting objects: 5% (54/1076) 2024-12-18T03:30:23.7705509Z remote: Counting objects: 6% (65/1076) 2024-12-18T03:30:23.7706235Z remote: Counting objects: 7% (76/1076) 2024-12-18T03:30:23.7706913Z remote: Counting objects: 8% (87/1076) 2024-12-18T03:30:23.7707591Z remote: Counting objects: 9% (97/1076) 2024-12-18T03:30:23.7708273Z remote: Counting objects: 10% (108/1076) 2024-12-18T03:30:23.7708890Z remote: Counting objects: 11% (119/1076) 2024-12-18T03:30:23.7709452Z remote: Counting objects: 12% (130/1076) 2024-12-18T03:30:23.7710016Z remote: Counting objects: 13% (140/1076) 2024-12-18T03:30:23.7710584Z remote: Counting objects: 14% (151/1076) 2024-12-18T03:30:23.7711265Z remote: Counting objects: 15% (162/1076) 2024-12-18T03:30:23.7711928Z remote: Counting objects: 16% (173/1076) 2024-12-18T03:30:23.7712557Z remote: Counting objects: 17% (183/1076) 2024-12-18T03:30:23.7713117Z remote: Counting objects: 18% (194/1076) 2024-12-18T03:30:23.7713689Z remote: Counting objects: 19% (205/1076) 2024-12-18T03:30:23.7714961Z remote: Counting objects: 20% (216/1076) 2024-12-18T03:30:23.7715576Z remote: Counting objects: 21% (226/1076) 2024-12-18T03:30:23.7716136Z remote: Counting objects: 22% (237/1076) 2024-12-18T03:30:23.7716685Z remote: Counting objects: 23% (248/1076) 2024-12-18T03:30:23.7717249Z remote: Counting objects: 24% (259/1076) 2024-12-18T03:30:23.7717813Z remote: Counting objects: 25% (269/1076) 2024-12-18T03:30:23.7718375Z remote: Counting objects: 26% (280/1076) 2024-12-18T03:30:23.7719062Z remote: Counting objects: 27% (291/1076) 2024-12-18T03:30:23.7719846Z remote: Counting objects: 28% (302/1076) 2024-12-18T03:30:23.7720402Z remote: Counting objects: 29% (313/1076) 2024-12-18T03:30:23.7720965Z remote: Counting objects: 30% (323/1076) 2024-12-18T03:30:23.7721534Z remote: Counting objects: 31% (334/1076) 2024-12-18T03:30:23.7722104Z remote: Counting objects: 32% (345/1076) 2024-12-18T03:30:23.7722677Z remote: Counting objects: 33% (356/1076) 2024-12-18T03:30:23.7723635Z remote: Counting objects: 34% (366/1076) 2024-12-18T03:30:23.7724251Z remote: Counting objects: 35% (377/1076) 2024-12-18T03:30:23.7724807Z remote: Counting objects: 36% (388/1076) 2024-12-18T03:30:23.7725378Z remote: Counting objects: 37% (399/1076) 2024-12-18T03:30:23.7725937Z remote: Counting objects: 38% (409/1076) 2024-12-18T03:30:23.7726493Z remote: Counting objects: 39% (420/1076) 2024-12-18T03:30:23.7727052Z remote: Counting objects: 40% (431/1076) 2024-12-18T03:30:23.7727613Z remote: Counting objects: 41% (442/1076) 2024-12-18T03:30:23.7728280Z remote: Counting objects: 42% (452/1076) 2024-12-18T03:30:23.7728942Z remote: Counting objects: 43% (463/1076) 2024-12-18T03:30:23.7729613Z remote: Counting objects: 44% (474/1076) 2024-12-18T03:30:23.7730265Z remote: Counting objects: 45% (485/1076) 2024-12-18T03:30:23.7730934Z remote: Counting objects: 46% (495/1076) 2024-12-18T03:30:23.7731607Z remote: Counting objects: 47% (506/1076) 2024-12-18T03:30:23.7732257Z remote: Counting objects: 48% (517/1076) 2024-12-18T03:30:23.7732825Z remote: Counting objects: 49% (528/1076) 2024-12-18T03:30:23.7733387Z remote: Counting objects: 50% (538/1076) 2024-12-18T03:30:23.7733942Z remote: Counting objects: 51% (549/1076) 2024-12-18T03:30:23.7734509Z remote: Counting objects: 52% (560/1076) 2024-12-18T03:30:23.7735068Z remote: Counting objects: 53% (571/1076) 2024-12-18T03:30:23.7735622Z remote: Counting objects: 54% (582/1076) 2024-12-18T03:30:23.7736191Z remote: Counting objects: 55% (592/1076) 2024-12-18T03:30:23.7736753Z remote: Counting objects: 56% (603/1076) 2024-12-18T03:30:23.7737310Z remote: Counting objects: 57% (614/1076) 2024-12-18T03:30:23.7737876Z remote: Counting objects: 58% (625/1076) 2024-12-18T03:30:23.7738443Z remote: Counting objects: 59% (635/1076) 2024-12-18T03:30:23.7739007Z remote: Counting objects: 60% (646/1076) 2024-12-18T03:30:23.7739567Z remote: Counting objects: 61% (657/1076) 2024-12-18T03:30:23.7740131Z remote: Counting objects: 62% (668/1076) 2024-12-18T03:30:23.7740850Z remote: Counting objects: 63% (678/1076) 2024-12-18T03:30:23.7741453Z remote: Counting objects: 64% (689/1076) 2024-12-18T03:30:23.7742014Z remote: Counting objects: 65% (700/1076) 2024-12-18T03:30:23.7742569Z remote: Counting objects: 66% (711/1076) 2024-12-18T03:30:23.7743118Z remote: Counting objects: 67% (721/1076) 2024-12-18T03:30:23.7743676Z remote: Counting objects: 68% (732/1076) 2024-12-18T03:30:23.7744235Z remote: Counting objects: 69% (743/1076) 2024-12-18T03:30:23.7744797Z remote: Counting objects: 70% (754/1076) 2024-12-18T03:30:23.7745357Z remote: Counting objects: 71% (764/1076) 2024-12-18T03:30:23.7746240Z remote: Counting objects: 72% (775/1076) 2024-12-18T03:30:23.7746828Z remote: Counting objects: 73% (786/1076) 2024-12-18T03:30:23.7747390Z remote: Counting objects: 74% (797/1076) 2024-12-18T03:30:23.7747959Z remote: Counting objects: 75% (807/1076) 2024-12-18T03:30:23.7748518Z remote: Counting objects: 76% (818/1076) 2024-12-18T03:30:23.7749074Z remote: Counting objects: 77% (829/1076) 2024-12-18T03:30:23.7749656Z remote: Counting objects: 78% (840/1076) 2024-12-18T03:30:23.7750211Z remote: Counting objects: 79% (851/1076) 2024-12-18T03:30:23.7750769Z remote: Counting objects: 80% (861/1076) 2024-12-18T03:30:23.7751329Z remote: Counting objects: 81% (872/1076) 2024-12-18T03:30:23.7751881Z remote: Counting objects: 82% (883/1076) 2024-12-18T03:30:23.7752440Z remote: Counting objects: 83% (894/1076) 2024-12-18T03:30:23.8427134Z remote: Counting objects: 84% (904/1076) 2024-12-18T03:30:23.8427945Z remote: Counting objects: 85% (915/1076) 2024-12-18T03:30:23.8429346Z remote: Counting objects: 86% (926/1076) 2024-12-18T03:30:23.8429941Z remote: Counting objects: 87% (937/1076) 2024-12-18T03:30:23.8430521Z remote: Counting objects: 88% (947/1076) 2024-12-18T03:30:23.8431072Z remote: Counting objects: 89% (958/1076) 2024-12-18T03:30:23.8431634Z remote: Counting objects: 90% (969/1076) 2024-12-18T03:30:23.8432212Z remote: Counting objects: 91% (980/1076) 2024-12-18T03:30:23.8432759Z remote: Counting objects: 92% (990/1076) 2024-12-18T03:30:23.8433331Z remote: Counting objects: 93% (1001/1076) 2024-12-18T03:30:23.8433919Z remote: Counting objects: 94% (1012/1076) 2024-12-18T03:30:23.8434483Z remote: Counting objects: 95% (1023/1076) 2024-12-18T03:30:23.8435073Z remote: Counting objects: 96% (1033/1076) 2024-12-18T03:30:23.8435648Z remote: Counting objects: 97% (1044/1076) 2024-12-18T03:30:23.8436204Z remote: Counting objects: 98% (1055/1076) 2024-12-18T03:30:23.8436774Z remote: Counting objects: 99% (1066/1076) 2024-12-18T03:30:23.8437344Z remote: Counting objects: 100% (1076/1076) 2024-12-18T03:30:23.8437966Z remote: Counting objects: 100% (1076/1076), done. 2024-12-18T03:30:23.8438623Z remote: Compressing objects: 0% (1/393) 2024-12-18T03:30:23.8439207Z remote: Compressing objects: 1% (4/393) 2024-12-18T03:30:23.8439944Z remote: Compressing objects: 2% (8/393) 2024-12-18T03:30:23.8440533Z remote: Compressing objects: 3% (12/393) 2024-12-18T03:30:23.9160166Z remote: Compressing objects: 4% (16/393) 2024-12-18T03:30:23.9160920Z remote: Compressing objects: 5% (20/393) 2024-12-18T03:30:23.9894249Z remote: Compressing objects: 6% (24/393) 2024-12-18T03:30:24.0627286Z remote: Compressing objects: 7% (28/393) 2024-12-18T03:30:24.1365627Z remote: Compressing objects: 8% (32/393) 2024-12-18T03:30:24.1366412Z remote: Compressing objects: 9% (36/393) 2024-12-18T03:30:24.1628404Z remote: Compressing objects: 10% (40/393) 2024-12-18T03:30:24.1629254Z remote: Compressing objects: 11% (44/393) 2024-12-18T03:30:24.1629916Z remote: Compressing objects: 12% (48/393) 2024-12-18T03:30:24.1630530Z remote: Compressing objects: 13% (52/393) 2024-12-18T03:30:24.1631112Z remote: Compressing objects: 14% (56/393) 2024-12-18T03:30:24.1631677Z remote: Compressing objects: 15% (59/393) 2024-12-18T03:30:24.1632261Z remote: Compressing objects: 16% (63/393) 2024-12-18T03:30:24.1632838Z remote: Compressing objects: 17% (67/393) 2024-12-18T03:30:24.1633429Z remote: Compressing objects: 18% (71/393) 2024-12-18T03:30:24.1634020Z remote: Compressing objects: 19% (75/393) 2024-12-18T03:30:24.1634602Z remote: Compressing objects: 20% (79/393) 2024-12-18T03:30:24.1635660Z remote: Compressing objects: 21% (83/393) 2024-12-18T03:30:24.1636317Z remote: Compressing objects: 22% (87/393) 2024-12-18T03:30:24.1636928Z remote: Compressing objects: 23% (91/393) 2024-12-18T03:30:24.1637512Z remote: Compressing objects: 24% (95/393) 2024-12-18T03:30:24.1638085Z remote: Compressing objects: 25% (99/393) 2024-12-18T03:30:24.1638675Z remote: Compressing objects: 26% (103/393) 2024-12-18T03:30:24.1639273Z remote: Compressing objects: 27% (107/393) 2024-12-18T03:30:24.1640065Z remote: Compressing objects: 28% (111/393) 2024-12-18T03:30:24.1640651Z remote: Compressing objects: 29% (114/393) 2024-12-18T03:30:24.1728250Z remote: Compressing objects: 30% (118/393) 2024-12-18T03:30:24.1729057Z remote: Compressing objects: 31% (122/393) 2024-12-18T03:30:24.1729706Z remote: Compressing objects: 32% (126/393) 2024-12-18T03:30:24.1730307Z remote: Compressing objects: 33% (130/393) 2024-12-18T03:30:24.1730912Z remote: Compressing objects: 34% (134/393) 2024-12-18T03:30:24.1732038Z remote: Compressing objects: 35% (138/393) 2024-12-18T03:30:24.1732622Z remote: Compressing objects: 36% (142/393) 2024-12-18T03:30:24.1733194Z remote: Compressing objects: 37% (146/393) 2024-12-18T03:30:24.1733778Z remote: Compressing objects: 38% (150/393) 2024-12-18T03:30:24.1734366Z remote: Compressing objects: 39% (154/393) 2024-12-18T03:30:24.1734930Z remote: Compressing objects: 40% (158/393) 2024-12-18T03:30:24.1735504Z remote: Compressing objects: 41% (162/393) 2024-12-18T03:30:24.1736077Z remote: Compressing objects: 42% (166/393) 2024-12-18T03:30:24.1736647Z remote: Compressing objects: 43% (169/393) 2024-12-18T03:30:24.1737223Z remote: Compressing objects: 44% (173/393) 2024-12-18T03:30:24.1737800Z remote: Compressing objects: 45% (177/393) 2024-12-18T03:30:24.1738372Z remote: Compressing objects: 46% (181/393) 2024-12-18T03:30:24.1738949Z remote: Compressing objects: 47% (185/393) 2024-12-18T03:30:24.1739537Z remote: Compressing objects: 48% (189/393) 2024-12-18T03:30:24.1740102Z remote: Compressing objects: 49% (193/393) 2024-12-18T03:30:24.1740685Z remote: Compressing objects: 50% (197/393) 2024-12-18T03:30:24.1789467Z remote: Compressing objects: 51% (201/393) 2024-12-18T03:30:24.1790255Z remote: Compressing objects: 52% (205/393) 2024-12-18T03:30:24.1790865Z remote: Compressing objects: 53% (209/393) 2024-12-18T03:30:24.1791460Z remote: Compressing objects: 54% (213/393) 2024-12-18T03:30:24.1792075Z remote: Compressing objects: 55% (217/393) 2024-12-18T03:30:24.1792672Z remote: Compressing objects: 56% (221/393) 2024-12-18T03:30:24.1793283Z remote: Compressing objects: 57% (225/393) 2024-12-18T03:30:24.1793890Z remote: Compressing objects: 58% (228/393) 2024-12-18T03:30:24.1794513Z remote: Compressing objects: 59% (232/393) 2024-12-18T03:30:24.1795104Z remote: Compressing objects: 60% (236/393) 2024-12-18T03:30:24.1795698Z remote: Compressing objects: 61% (240/393) 2024-12-18T03:30:24.1796270Z remote: Compressing objects: 62% (244/393) 2024-12-18T03:30:24.1796848Z remote: Compressing objects: 63% (248/393) 2024-12-18T03:30:24.1797427Z remote: Compressing objects: 64% (252/393) 2024-12-18T03:30:24.1797999Z remote: Compressing objects: 65% (256/393) 2024-12-18T03:30:24.1798580Z remote: Compressing objects: 66% (260/393) 2024-12-18T03:30:24.1799169Z remote: Compressing objects: 67% (264/393) 2024-12-18T03:30:24.1799897Z remote: Compressing objects: 68% (268/393) 2024-12-18T03:30:24.1800476Z remote: Compressing objects: 69% (272/393) 2024-12-18T03:30:24.1801055Z remote: Compressing objects: 70% (276/393) 2024-12-18T03:30:24.1801627Z remote: Compressing objects: 71% (280/393) 2024-12-18T03:30:24.1827400Z remote: Compressing objects: 72% (283/393) 2024-12-18T03:30:24.1828295Z remote: Compressing objects: 73% (287/393) 2024-12-18T03:30:24.1828922Z remote: Compressing objects: 74% (291/393) 2024-12-18T03:30:24.1829512Z remote: Compressing objects: 75% (295/393) 2024-12-18T03:30:24.1830093Z remote: Compressing objects: 76% (299/393) 2024-12-18T03:30:24.1830678Z remote: Compressing objects: 77% (303/393) 2024-12-18T03:30:24.1831249Z remote: Compressing objects: 78% (307/393) 2024-12-18T03:30:24.1831832Z remote: Compressing objects: 79% (311/393) 2024-12-18T03:30:24.1832413Z remote: Compressing objects: 80% (315/393) 2024-12-18T03:30:24.1832985Z remote: Compressing objects: 81% (319/393) 2024-12-18T03:30:24.1833572Z remote: Compressing objects: 82% (323/393) 2024-12-18T03:30:24.1834147Z remote: Compressing objects: 83% (327/393) 2024-12-18T03:30:24.1834726Z remote: Compressing objects: 84% (331/393) 2024-12-18T03:30:24.1835600Z remote: Compressing objects: 85% (335/393) 2024-12-18T03:30:24.1836591Z remote: Compressing objects: 86% (338/393) 2024-12-18T03:30:24.1837244Z remote: Compressing objects: 87% (342/393) 2024-12-18T03:30:24.1837900Z remote: Compressing objects: 88% (346/393) 2024-12-18T03:30:24.1838583Z remote: Compressing objects: 89% (350/393) 2024-12-18T03:30:24.1839257Z remote: Compressing objects: 90% (354/393) 2024-12-18T03:30:24.1840087Z remote: Compressing objects: 91% (358/393) 2024-12-18T03:30:24.1840769Z remote: Compressing objects: 92% (362/393) 2024-12-18T03:30:24.1841551Z remote: Compressing objects: 93% (366/393) 2024-12-18T03:30:24.1842138Z remote: Compressing objects: 94% (370/393) 2024-12-18T03:30:24.1842890Z remote: Compressing objects: 95% (374/393) 2024-12-18T03:30:24.1843464Z remote: Compressing objects: 96% (378/393) 2024-12-18T03:30:24.1844101Z remote: Compressing objects: 97% (382/393) 2024-12-18T03:30:24.1844683Z remote: Compressing objects: 98% (386/393) 2024-12-18T03:30:24.1845260Z remote: Compressing objects: 99% (390/393) 2024-12-18T03:30:24.1845914Z remote: Compressing objects: 100% (393/393) 2024-12-18T03:30:24.1846567Z remote: Compressing objects: 100% (393/393), done. 2024-12-18T03:30:49.2380037Z remote: Total 1056587 (delta 894), reused 692 (delta 683), pack-reused 1055511 (from 5) 2024-12-18T03:30:59.7998950Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-18T03:30:59.8028398Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:30:59.8036131Z ##[endgroup] 2024-12-18T03:30:59.8036876Z ##[group]Determining the checkout info 2024-12-18T03:30:59.8038366Z ##[endgroup] 2024-12-18T03:30:59.8039160Z ##[group]Checking out the ref 2024-12-18T03:30:59.8044902Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:31:01.2233541Z ##[endgroup] 2024-12-18T03:31:01.2234350Z ##[group]Setting up auth for fetching submodules 2024-12-18T03:31:01.2240840Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T03:31:01.2312407Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-18T03:31:01.2350527Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-18T03:31:01.2387438Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-18T03:31:01.2430219Z ##[endgroup] 2024-12-18T03:31:01.2430621Z ##[group]Fetching submodules 2024-12-18T03:31:01.2435647Z [command]/usr/bin/git submodule sync --recursive 2024-12-18T03:31:01.2775409Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-18T03:31:01.3096375Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-18T03:31:01.3100544Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-18T03:31:01.3104534Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-18T03:31:01.3108725Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-18T03:31:01.3112888Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-18T03:31:01.3117272Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-18T03:31:01.3124423Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-18T03:31:01.3126045Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-18T03:31:01.3130933Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-18T03:31:01.3138474Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-18T03:31:01.3142825Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-18T03:31:01.3147739Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-18T03:31:01.3152394Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-18T03:31:01.3157150Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-18T03:31:01.3162526Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-18T03:31:01.3167491Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-18T03:31:01.3171486Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-18T03:31:01.3175770Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:31:01.3179925Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-18T03:31:01.3184521Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-18T03:31:01.3188824Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-18T03:31:01.3193299Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-18T03:31:01.3197721Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-18T03:31:01.3202754Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-18T03:31:01.3207304Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-18T03:31:01.3211930Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-18T03:31:01.3216686Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-18T03:31:01.3221502Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-18T03:31:01.3226627Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-18T03:31:01.3231349Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-18T03:31:01.3236234Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-18T03:31:01.3241602Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-18T03:31:01.3246771Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-18T03:31:01.3252005Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-18T03:31:01.3257423Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-18T03:31:01.3262316Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-18T03:31:01.3364878Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-18T03:31:04.6349319Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-18T03:31:07.7189022Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-18T03:31:10.3529346Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-18T03:31:12.9985650Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-18T03:31:16.6296570Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-18T03:31:21.5451771Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-18T03:31:31.1845360Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-18T03:31:34.7507651Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-18T03:31:39.3545989Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-18T03:31:41.9841153Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-18T03:31:44.4752761Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-18T03:31:49.8643953Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-18T03:31:53.9765370Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-18T03:31:58.0060428Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-18T03:32:01.2342043Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-18T03:32:04.4529453Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-18T03:32:08.9856401Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-18T03:32:11.7783530Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-18T03:32:14.5217396Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-18T03:32:17.7380876Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-18T03:32:19.9142906Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-18T03:32:22.1882582Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-18T03:32:25.9663532Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-18T03:32:28.9651993Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-18T03:32:31.7688410Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-18T03:32:45.6682135Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-18T03:32:49.6751051Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-18T03:32:54.6297330Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-18T03:32:57.6648674Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T03:33:07.7183782Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T03:33:10.6087600Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T03:33:13.9309187Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T03:33:17.2990240Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T03:33:19.5938314Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T03:33:22.1221148Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T03:33:24.8078550Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T03:33:24.8372987Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T03:33:24.8610159Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T03:33:24.8985946Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T03:33:24.9475632Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T03:33:24.9993555Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T03:33:25.7544874Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T03:33:25.7988821Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T03:33:26.0473226Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T03:33:26.1180497Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T03:33:26.2355192Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T03:33:26.2932128Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T03:33:26.8051495Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T03:33:27.0544268Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T03:33:27.1610255Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T03:33:27.1684332Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:33:27.1690923Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:33:27.1694696Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:33:27.1698418Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:33:27.1703013Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:33:27.1751286Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T03:33:29.9974924Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T03:33:32.8836372Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T03:33:36.7733976Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T03:33:40.0225370Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T03:33:43.3115617Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T03:33:43.4222511Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T03:33:43.8427041Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T03:33:43.9198529Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T03:33:43.9454147Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T03:33:44.0920650Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T03:33:44.1496084Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T03:33:44.2078293Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T03:33:44.2500001Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T03:33:44.3080234Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T03:33:44.3334840Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T03:33:44.3378213Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T03:33:44.3417416Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T03:33:58.2055733Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T03:33:58.2441710Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T03:33:58.3458470Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T03:33:58.3543965Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:33:58.3546326Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:33:58.3550802Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:33:58.3598635Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T03:34:00.8204773Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T03:34:04.1942272Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T03:34:07.9442444Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T03:34:07.9522929Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:34:07.9530948Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:34:07.9534059Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:34:07.9537244Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:34:07.9540229Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:34:07.9543990Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:34:07.9547528Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:34:07.9550780Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:34:07.9620654Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T03:34:11.5171320Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T03:34:14.3538862Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T03:34:17.3089687Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T03:34:19.9579294Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T03:34:23.9116713Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T03:34:27.5863177Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T03:34:36.0583935Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T03:34:39.8652126Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T03:34:39.9020536Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T03:34:39.9555572Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T03:34:39.9835677Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T03:34:39.9886233Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:34:39.9937178Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T03:34:43.2526906Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T03:34:43.2875771Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T03:34:43.3375192Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T03:34:43.4510861Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T03:34:43.4848374Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T03:34:43.5383193Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T03:34:43.6089058Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T03:34:43.6621600Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T03:34:43.7105926Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T03:34:43.8358478Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T03:34:44.2610467Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T03:34:44.2710134Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T03:34:44.2774631Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T03:34:49.0427993Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T03:34:49.1264789Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T03:34:49.1310361Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:34:49.1313484Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:34:49.1316657Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:34:49.1321439Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:34:49.1323834Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:34:49.1326945Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:34:49.1331254Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:34:49.1334605Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:34:49.1379617Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T03:34:52.1394055Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T03:34:55.9492179Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T03:34:59.4434638Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T03:35:07.8350681Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T03:35:11.0375361Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T03:35:14.1864389Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T03:35:16.9691098Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T03:35:28.9853292Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T03:35:29.0409935Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T03:35:29.0717662Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T03:35:29.1998925Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T03:35:29.2307726Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T03:35:29.2612448Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T03:35:29.2919571Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T03:35:29.2963025Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:35:29.2965771Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:35:29.3016930Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T03:35:33.9140073Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T03:35:37.8799121Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T03:35:37.9459909Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T03:35:38.4467076Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T03:35:38.4828961Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T03:35:38.7660453Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T03:35:38.7746033Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:35:38.7748410Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T03:35:38.7807959Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T03:35:42.8347350Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T03:35:46.7512925Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T03:35:46.8320743Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T03:35:46.8609215Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T03:35:46.8897924Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T03:35:46.9407718Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T03:35:46.9860534Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T03:35:47.0436735Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T03:35:47.0877569Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T03:35:47.0943393Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:35:47.0945874Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:35:47.0950254Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:35:47.0955107Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:35:47.0998046Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T03:35:51.1008413Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T03:35:54.3191987Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T03:35:59.9479258Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T03:36:07.0261067Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T03:36:07.0551486Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T03:36:07.1326075Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T03:36:07.1752739Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T03:36:07.1823686Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:07.1875691Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T03:36:10.7968312Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T03:36:10.8075635Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T03:36:10.8346621Z Entering 'android/libs/fbjni' 2024-12-18T03:36:10.8389368Z Entering 'third_party/FP16' 2024-12-18T03:36:10.8426215Z Entering 'third_party/FXdiv' 2024-12-18T03:36:10.8463468Z Entering 'third_party/NNPACK' 2024-12-18T03:36:10.8500187Z Entering 'third_party/NVTX' 2024-12-18T03:36:10.8543813Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:36:10.8581973Z Entering 'third_party/XNNPACK' 2024-12-18T03:36:10.8636876Z Entering 'third_party/benchmark' 2024-12-18T03:36:10.8676560Z Entering 'third_party/composable_kernel' 2024-12-18T03:36:10.8725664Z Entering 'third_party/cpp-httplib' 2024-12-18T03:36:10.8766949Z Entering 'third_party/cpuinfo' 2024-12-18T03:36:10.8803330Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:36:10.8841427Z Entering 'third_party/cutlass' 2024-12-18T03:36:10.8899962Z Entering 'third_party/eigen' 2024-12-18T03:36:10.8943338Z Entering 'third_party/fbgemm' 2024-12-18T03:36:10.8986611Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:10.9026260Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:10.9063577Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:10.9106758Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:10.9141248Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:10.9178547Z Entering 'third_party/flatbuffers' 2024-12-18T03:36:10.9222104Z Entering 'third_party/fmt' 2024-12-18T03:36:10.9261228Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:36:10.9301148Z Entering 'third_party/gloo' 2024-12-18T03:36:10.9343666Z Entering 'third_party/googletest' 2024-12-18T03:36:10.9386806Z Entering 'third_party/ideep' 2024-12-18T03:36:10.9423923Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:36:10.9469642Z Entering 'third_party/ittapi' 2024-12-18T03:36:10.9512262Z Entering 'third_party/kineto' 2024-12-18T03:36:10.9571119Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:36:10.9615766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:36:10.9657853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:36:10.9702647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:36:10.9738539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:36:10.9771957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:36:10.9819559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:36:10.9860670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:36:10.9900854Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:36:10.9936545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:36:10.9978114Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:36:11.0023318Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:36:11.0063921Z Entering 'third_party/mimalloc' 2024-12-18T03:36:11.0103242Z Entering 'third_party/nccl/nccl' 2024-12-18T03:36:11.0140283Z Entering 'third_party/nlohmann' 2024-12-18T03:36:11.0186696Z Entering 'third_party/onnx' 2024-12-18T03:36:11.0237125Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:36:11.0291331Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:36:11.0329172Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:36:11.0366966Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:36:11.0404872Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:36:11.0441990Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:36:11.0477274Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:36:11.0515095Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:36:11.0548019Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:36:11.0582560Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:36:11.0623468Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:36:11.0661364Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:36:11.0715395Z Entering 'third_party/pocketfft' 2024-12-18T03:36:11.0766901Z Entering 'third_party/protobuf' 2024-12-18T03:36:11.0808112Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:36:11.0843242Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:36:11.0881370Z Entering 'third_party/psimd' 2024-12-18T03:36:11.0918829Z Entering 'third_party/pthreadpool' 2024-12-18T03:36:11.0953058Z Entering 'third_party/pybind11' 2024-12-18T03:36:11.0994432Z Entering 'third_party/python-peachpy' 2024-12-18T03:36:11.1037279Z Entering 'third_party/sleef' 2024-12-18T03:36:11.1081486Z Entering 'third_party/tensorpipe' 2024-12-18T03:36:11.1121577Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:36:11.1155597Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:36:11.1198301Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:36:11.1235663Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:36:11.1277928Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:11.1341540Z ##[endgroup] 2024-12-18T03:36:11.1343853Z ##[group]Persisting credentials for submodules 2024-12-18T03:36:11.1347225Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T03:36:11.1604234Z Entering 'android/libs/fbjni' 2024-12-18T03:36:11.1647477Z Entering 'third_party/FP16' 2024-12-18T03:36:11.1687594Z Entering 'third_party/FXdiv' 2024-12-18T03:36:11.1727862Z Entering 'third_party/NNPACK' 2024-12-18T03:36:11.1767881Z Entering 'third_party/NVTX' 2024-12-18T03:36:11.1806060Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:36:11.1851062Z Entering 'third_party/XNNPACK' 2024-12-18T03:36:11.1904534Z Entering 'third_party/benchmark' 2024-12-18T03:36:11.1943404Z Entering 'third_party/composable_kernel' 2024-12-18T03:36:11.2000260Z Entering 'third_party/cpp-httplib' 2024-12-18T03:36:11.2048628Z Entering 'third_party/cpuinfo' 2024-12-18T03:36:11.2089889Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:36:11.2129969Z Entering 'third_party/cutlass' 2024-12-18T03:36:11.2185212Z Entering 'third_party/eigen' 2024-12-18T03:36:11.2234112Z Entering 'third_party/fbgemm' 2024-12-18T03:36:11.2278765Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:11.2325752Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:11.2365939Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:11.2415088Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:11.2458874Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:11.2504538Z Entering 'third_party/flatbuffers' 2024-12-18T03:36:11.2546767Z Entering 'third_party/fmt' 2024-12-18T03:36:11.2587445Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:36:11.2627354Z Entering 'third_party/gloo' 2024-12-18T03:36:11.2668219Z Entering 'third_party/googletest' 2024-12-18T03:36:11.2714565Z Entering 'third_party/ideep' 2024-12-18T03:36:11.2757179Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:36:11.2824063Z Entering 'third_party/ittapi' 2024-12-18T03:36:11.2876145Z Entering 'third_party/kineto' 2024-12-18T03:36:11.2921008Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:36:11.2964235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:36:11.3012939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:36:11.3051897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:36:11.3096657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:36:11.3139632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:36:11.3188868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:36:11.3237299Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:36:11.3281704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:36:11.3321613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:36:11.3365172Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:36:11.3412510Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:36:11.3454319Z Entering 'third_party/mimalloc' 2024-12-18T03:36:11.3499789Z Entering 'third_party/nccl/nccl' 2024-12-18T03:36:11.3543303Z Entering 'third_party/nlohmann' 2024-12-18T03:36:11.3584431Z Entering 'third_party/onnx' 2024-12-18T03:36:11.3637482Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:36:11.3688181Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:36:11.3730404Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:36:11.3769387Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:36:11.3809641Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:36:11.3846687Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:36:11.3889527Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:36:11.3926901Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:36:11.3969880Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:36:11.4007617Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:36:11.4061314Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:36:11.4108292Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:36:11.4170205Z Entering 'third_party/pocketfft' 2024-12-18T03:36:11.4211730Z Entering 'third_party/protobuf' 2024-12-18T03:36:11.4252691Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:36:11.4294198Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:36:11.4338139Z Entering 'third_party/psimd' 2024-12-18T03:36:11.4385303Z Entering 'third_party/pthreadpool' 2024-12-18T03:36:11.4427397Z Entering 'third_party/pybind11' 2024-12-18T03:36:11.4477853Z Entering 'third_party/python-peachpy' 2024-12-18T03:36:11.4518413Z Entering 'third_party/sleef' 2024-12-18T03:36:11.4561008Z Entering 'third_party/tensorpipe' 2024-12-18T03:36:11.4602356Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:36:11.4654659Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:36:11.4703340Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:36:11.4742150Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:36:11.4779061Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:11.4847139Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T03:36:11.5093917Z Entering 'android/libs/fbjni' 2024-12-18T03:36:11.5139982Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T03:36:11.5159160Z Entering 'third_party/FP16' 2024-12-18T03:36:11.5207074Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T03:36:11.5225827Z Entering 'third_party/FXdiv' 2024-12-18T03:36:11.5264114Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T03:36:11.5283462Z Entering 'third_party/NNPACK' 2024-12-18T03:36:11.5327045Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T03:36:11.5344443Z Entering 'third_party/NVTX' 2024-12-18T03:36:11.5378042Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T03:36:11.5405664Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:36:11.5445547Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T03:36:11.5462456Z Entering 'third_party/XNNPACK' 2024-12-18T03:36:11.5497072Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T03:36:11.5528958Z Entering 'third_party/benchmark' 2024-12-18T03:36:11.5566742Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:36:11.5583909Z Entering 'third_party/composable_kernel' 2024-12-18T03:36:11.5621428Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T03:36:11.5645084Z Entering 'third_party/cpp-httplib' 2024-12-18T03:36:11.5682918Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T03:36:11.5702162Z Entering 'third_party/cpuinfo' 2024-12-18T03:36:11.5739415Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T03:36:11.5758817Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:36:11.5796361Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T03:36:11.5820921Z Entering 'third_party/cutlass' 2024-12-18T03:36:11.5855955Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T03:36:11.5888000Z Entering 'third_party/eigen' 2024-12-18T03:36:11.5924828Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T03:36:11.5949104Z Entering 'third_party/fbgemm' 2024-12-18T03:36:11.6011158Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T03:36:11.6025511Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:11.6061568Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T03:36:11.6085116Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:11.6127726Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T03:36:11.6140477Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:11.6176679Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T03:36:11.6205606Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:11.6244948Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.6261954Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:11.6295096Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T03:36:11.6314981Z Entering 'third_party/flatbuffers' 2024-12-18T03:36:11.6354413Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T03:36:11.6374673Z Entering 'third_party/fmt' 2024-12-18T03:36:11.6417251Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T03:36:11.6438683Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:36:11.6471594Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T03:36:11.6489640Z Entering 'third_party/gloo' 2024-12-18T03:36:11.6522135Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T03:36:11.6541584Z Entering 'third_party/googletest' 2024-12-18T03:36:11.6579385Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.6602279Z Entering 'third_party/ideep' 2024-12-18T03:36:11.6637976Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T03:36:11.6659262Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:36:11.6697067Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T03:36:11.6728480Z Entering 'third_party/ittapi' 2024-12-18T03:36:11.6766192Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T03:36:11.6783564Z Entering 'third_party/kineto' 2024-12-18T03:36:11.6818633Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T03:36:11.6836399Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:36:11.6873921Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T03:36:11.6891777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:36:11.6933331Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T03:36:11.6957320Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:36:11.6998489Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T03:36:11.7020765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:36:11.7059764Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T03:36:11.7077984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:36:11.7120671Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T03:36:11.7137199Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:36:11.7176921Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T03:36:11.7202509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:36:11.7242874Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T03:36:11.7263269Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:36:11.7299780Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.7316892Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:36:11.7354443Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T03:36:11.7371934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:36:11.7409315Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T03:36:11.7430631Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:36:11.7465325Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T03:36:11.7482885Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:36:11.7525014Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.7545733Z Entering 'third_party/mimalloc' 2024-12-18T03:36:11.7582665Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T03:36:11.7602882Z Entering 'third_party/nccl/nccl' 2024-12-18T03:36:11.7648578Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T03:36:11.7668138Z Entering 'third_party/nlohmann' 2024-12-18T03:36:11.7705360Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T03:36:11.7724777Z Entering 'third_party/onnx' 2024-12-18T03:36:11.7762111Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T03:36:11.7795582Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:36:11.7834409Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:36:11.7855563Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:36:11.7895859Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T03:36:11.7917952Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:36:11.7955381Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:36:11.7972303Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:36:11.8008288Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.8025787Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:36:11.8059346Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T03:36:11.8076703Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:36:11.8117638Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T03:36:11.8137148Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:36:11.8178488Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T03:36:11.8203485Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:36:11.8237721Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T03:36:11.8259773Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:36:11.8294510Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T03:36:11.8325739Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:36:11.8364581Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T03:36:11.8382751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:36:11.8416971Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T03:36:11.8443364Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:36:11.8482390Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T03:36:11.8517545Z Entering 'third_party/pocketfft' 2024-12-18T03:36:11.8564228Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T03:36:11.8583099Z Entering 'third_party/protobuf' 2024-12-18T03:36:11.8631416Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T03:36:11.8659334Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:36:11.8702060Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:36:11.8724951Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:36:11.8767562Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.8790077Z Entering 'third_party/psimd' 2024-12-18T03:36:11.8826211Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T03:36:11.8845688Z Entering 'third_party/pthreadpool' 2024-12-18T03:36:11.8885666Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T03:36:11.8912097Z Entering 'third_party/pybind11' 2024-12-18T03:36:11.8952844Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:36:11.8971838Z Entering 'third_party/python-peachpy' 2024-12-18T03:36:11.9005402Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T03:36:11.9023195Z Entering 'third_party/sleef' 2024-12-18T03:36:11.9058930Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T03:36:11.9078312Z Entering 'third_party/tensorpipe' 2024-12-18T03:36:11.9121741Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T03:36:11.9142230Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:36:11.9177844Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:36:11.9194485Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:36:11.9227015Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T03:36:11.9246113Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:36:11.9285668Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T03:36:11.9304120Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:36:11.9342062Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:36:11.9365384Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:11.9405470Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T03:36:11.9670777Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T03:36:11.9933768Z Entering 'android/libs/fbjni' 2024-12-18T03:36:11.9970760Z Entering 'third_party/FP16' 2024-12-18T03:36:12.0006338Z Entering 'third_party/FXdiv' 2024-12-18T03:36:12.0048425Z Entering 'third_party/NNPACK' 2024-12-18T03:36:12.0084421Z Entering 'third_party/NVTX' 2024-12-18T03:36:12.0127694Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:36:12.0164424Z Entering 'third_party/XNNPACK' 2024-12-18T03:36:12.0217749Z Entering 'third_party/benchmark' 2024-12-18T03:36:12.0259899Z Entering 'third_party/composable_kernel' 2024-12-18T03:36:12.0303182Z Entering 'third_party/cpp-httplib' 2024-12-18T03:36:12.0337955Z Entering 'third_party/cpuinfo' 2024-12-18T03:36:12.0384902Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:36:12.0420213Z Entering 'third_party/cutlass' 2024-12-18T03:36:12.0463624Z Entering 'third_party/eigen' 2024-12-18T03:36:12.0505908Z Entering 'third_party/fbgemm' 2024-12-18T03:36:12.0547376Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:12.0584756Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:12.0618672Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:12.0668394Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:12.0704043Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:12.0744515Z Entering 'third_party/flatbuffers' 2024-12-18T03:36:12.0788125Z Entering 'third_party/fmt' 2024-12-18T03:36:12.0824653Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:36:12.0860710Z Entering 'third_party/gloo' 2024-12-18T03:36:12.0897109Z Entering 'third_party/googletest' 2024-12-18T03:36:12.0944618Z Entering 'third_party/ideep' 2024-12-18T03:36:12.0981792Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:36:12.1027085Z Entering 'third_party/ittapi' 2024-12-18T03:36:12.1063050Z Entering 'third_party/kineto' 2024-12-18T03:36:12.1101102Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:36:12.1141158Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:36:12.1182400Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:36:12.1216786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:36:12.1252090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:36:12.1287933Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:36:12.1328550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:36:12.1361813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:36:12.1396127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:36:12.1438312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:36:12.1481246Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:36:12.1517424Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:36:12.1563869Z Entering 'third_party/mimalloc' 2024-12-18T03:36:12.1603221Z Entering 'third_party/nccl/nccl' 2024-12-18T03:36:12.1637413Z Entering 'third_party/nlohmann' 2024-12-18T03:36:12.1678648Z Entering 'third_party/onnx' 2024-12-18T03:36:12.1734013Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:36:12.1782115Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:36:12.1829059Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:36:12.1871892Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:36:12.1911312Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:36:12.1943709Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:36:12.1983214Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:36:12.2021456Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:36:12.2053403Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:36:12.2084449Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:36:12.2118829Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:36:12.2164928Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:36:12.2214698Z Entering 'third_party/pocketfft' 2024-12-18T03:36:12.2258671Z Entering 'third_party/protobuf' 2024-12-18T03:36:12.2297393Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:36:12.2342255Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:36:12.2383133Z Entering 'third_party/psimd' 2024-12-18T03:36:12.2421455Z Entering 'third_party/pthreadpool' 2024-12-18T03:36:12.2461409Z Entering 'third_party/pybind11' 2024-12-18T03:36:12.2501529Z Entering 'third_party/python-peachpy' 2024-12-18T03:36:12.2542960Z Entering 'third_party/sleef' 2024-12-18T03:36:12.2582768Z Entering 'third_party/tensorpipe' 2024-12-18T03:36:12.2620112Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:36:12.2656112Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:36:12.2693299Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:36:12.2725382Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:36:12.2756579Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:12.2815794Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T03:36:12.3097485Z Entering 'android/libs/fbjni' 2024-12-18T03:36:12.3152434Z Entering 'third_party/FP16' 2024-12-18T03:36:12.3195822Z Entering 'third_party/FXdiv' 2024-12-18T03:36:12.3239085Z Entering 'third_party/NNPACK' 2024-12-18T03:36:12.3278855Z Entering 'third_party/NVTX' 2024-12-18T03:36:12.3324853Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:36:12.3359157Z Entering 'third_party/XNNPACK' 2024-12-18T03:36:12.3410445Z Entering 'third_party/benchmark' 2024-12-18T03:36:12.3450345Z Entering 'third_party/composable_kernel' 2024-12-18T03:36:12.3496308Z Entering 'third_party/cpp-httplib' 2024-12-18T03:36:12.3540085Z Entering 'third_party/cpuinfo' 2024-12-18T03:36:12.3577911Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:36:12.3621719Z Entering 'third_party/cutlass' 2024-12-18T03:36:12.3668016Z Entering 'third_party/eigen' 2024-12-18T03:36:12.3707073Z Entering 'third_party/fbgemm' 2024-12-18T03:36:12.3744732Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:12.3784941Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:12.3820361Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:12.3860848Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:12.3902096Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:12.3943229Z Entering 'third_party/flatbuffers' 2024-12-18T03:36:12.3981648Z Entering 'third_party/fmt' 2024-12-18T03:36:12.4023379Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:36:12.4061273Z Entering 'third_party/gloo' 2024-12-18T03:36:12.4100471Z Entering 'third_party/googletest' 2024-12-18T03:36:12.4138934Z Entering 'third_party/ideep' 2024-12-18T03:36:12.4179636Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:36:12.4223469Z Entering 'third_party/ittapi' 2024-12-18T03:36:12.4267763Z Entering 'third_party/kineto' 2024-12-18T03:36:12.4297130Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:36:12.4346065Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:36:12.4393274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:36:12.4435338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:36:12.4482063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:36:12.4531733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:36:12.4581878Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:36:12.4621533Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:36:12.4661404Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:36:12.4697575Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:36:12.4742000Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:36:12.4781973Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:36:12.4826075Z Entering 'third_party/mimalloc' 2024-12-18T03:36:12.4862865Z Entering 'third_party/nccl/nccl' 2024-12-18T03:36:12.4901712Z Entering 'third_party/nlohmann' 2024-12-18T03:36:12.4937671Z Entering 'third_party/onnx' 2024-12-18T03:36:12.4990000Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:36:12.5032819Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:36:12.5070180Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:36:12.5105008Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:36:12.5138627Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:36:12.5177868Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:36:12.5225404Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:36:12.5261730Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:36:12.5297897Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:36:12.5335289Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:36:12.5372669Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:36:12.5413153Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:36:12.5467303Z Entering 'third_party/pocketfft' 2024-12-18T03:36:12.5503694Z Entering 'third_party/protobuf' 2024-12-18T03:36:12.5542432Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:36:12.5585409Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:36:12.5625029Z Entering 'third_party/psimd' 2024-12-18T03:36:12.5667881Z Entering 'third_party/pthreadpool' 2024-12-18T03:36:12.5707159Z Entering 'third_party/pybind11' 2024-12-18T03:36:12.5747268Z Entering 'third_party/python-peachpy' 2024-12-18T03:36:12.5785310Z Entering 'third_party/sleef' 2024-12-18T03:36:12.5820093Z Entering 'third_party/tensorpipe' 2024-12-18T03:36:12.5859685Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:36:12.5894973Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:36:12.5932400Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:36:12.5969130Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:36:12.6005076Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:36:12.6059904Z ##[endgroup] 2024-12-18T03:36:12.6096971Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T03:36:12.6119091Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T03:36:12.6313411Z Prepare all required actions 2024-12-18T03:36:12.6314323Z Getting action download info 2024-12-18T03:36:12.6408352Z ##[group]Run ./.github/actions/setup-rocm 2024-12-18T03:36:12.6409076Z env: 2024-12-18T03:36:12.6409521Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.6410105Z ##[endgroup] 2024-12-18T03:36:12.6451013Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-12-18T03:36:12.6452056Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-12-18T03:36:12.6485073Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.6485785Z env: 2024-12-18T03:36:12.6486229Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.6486761Z ##[endgroup] 2024-12-18T03:36:12.6586603Z ##[group]Run set -ex 2024-12-18T03:36:12.6587139Z set -ex 2024-12-18T03:36:12.6587614Z  2024-12-18T03:36:12.6588056Z cat ~/.docker/config.json || true 2024-12-18T03:36:12.6589413Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-12-18T03:36:12.6590628Z rm -f ~/.docker/config.json 2024-12-18T03:36:12.6622929Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.6623428Z env: 2024-12-18T03:36:12.6623764Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.6624067Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.6624486Z ##[endgroup] 2024-12-18T03:36:12.6670799Z + cat /home/pytorchci/.docker/config.json 2024-12-18T03:36:12.6684624Z cat: /home/pytorchci/.docker/config.json: No such file or directory 2024-12-18T03:36:12.6685078Z + true 2024-12-18T03:36:12.6685426Z + rm -f /home/pytorchci/.docker/config.json 2024-12-18T03:36:12.6718582Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T03:36:12.6719240Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T03:36:12.6720152Z # shellcheck disable=SC2046 2024-12-18T03:36:12.6721322Z docker stop $(docker ps -q) || true 2024-12-18T03:36:12.6722064Z # Prune all stopped containers. 2024-12-18T03:36:12.6722634Z docker container prune -f 2024-12-18T03:36:12.6751547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.6752235Z env: 2024-12-18T03:36:12.6752771Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.6753289Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.6753863Z ##[endgroup] 2024-12-18T03:36:12.7096219Z "docker stop" requires at least 1 argument. 2024-12-18T03:36:12.7096684Z See 'docker stop --help'. 2024-12-18T03:36:12.7096873Z 2024-12-18T03:36:12.7097141Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T03:36:12.7097496Z 2024-12-18T03:36:12.7097632Z Stop one or more running containers 2024-12-18T03:36:12.7285682Z Total reclaimed space: 0B 2024-12-18T03:36:12.7335153Z ##[group]Run cat /etc/os-release || true 2024-12-18T03:36:12.7335553Z cat /etc/os-release || true 2024-12-18T03:36:12.7335900Z cat /etc/apt/sources.list.d/rocm.list || true 2024-12-18T03:36:12.7336265Z cat /opt/rocm/.info/version || true 2024-12-18T03:36:12.7472423Z whoami 2024-12-18T03:36:12.7496561Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.7496842Z env: 2024-12-18T03:36:12.7497002Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.7497227Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.7497470Z ##[endgroup] 2024-12-18T03:36:12.7542226Z PRETTY_NAME="Ubuntu 22.04.3 LTS" 2024-12-18T03:36:12.7542515Z NAME="Ubuntu" 2024-12-18T03:36:12.7542704Z VERSION_ID="22.04" 2024-12-18T03:36:12.7542923Z VERSION="22.04.3 LTS (Jammy Jellyfish)" 2024-12-18T03:36:12.7543195Z VERSION_CODENAME=jammy 2024-12-18T03:36:12.7543402Z ID=ubuntu 2024-12-18T03:36:12.7543576Z ID_LIKE=debian 2024-12-18T03:36:12.7543802Z HOME_URL="https://www.ubuntu.com/" 2024-12-18T03:36:12.7544485Z SUPPORT_URL="https://help.ubuntu.com/" 2024-12-18T03:36:12.7544859Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-12-18T03:36:12.7545320Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-12-18T03:36:12.7545748Z UBUNTU_CODENAME=jammy 2024-12-18T03:36:12.7552947Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.2.1 jammy main 2024-12-18T03:36:12.7569052Z 6.2.1-112 2024-12-18T03:36:12.7577973Z pytorchci 2024-12-18T03:36:12.7606135Z ##[group]Run rocm-smi 2024-12-18T03:36:12.7606485Z rocm-smi 2024-12-18T03:36:12.7641809Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.7642275Z env: 2024-12-18T03:36:12.7642539Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.7642947Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.7643373Z ##[endgroup] 2024-12-18T03:36:12.8472353Z 2024-12-18T03:36:12.8472424Z 2024-12-18T03:36:12.8473040Z ========================================= ROCm System Management Interface ========================================= 2024-12-18T03:36:12.8473869Z =================================================== Concise Info =================================================== 2024-12-18T03:36:12.8474755Z Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-12-18T03:36:12.8476124Z  (DID, GUID) (Edge) (Avg) (Mem, Compute, ID)  2024-12-18T03:36:12.8476824Z ==================================================================================================================== 2024-12-18T03:36:12.8477819Z 0 2 0x740f, 12261 44.0°C 42.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-12-18T03:36:12.8478795Z 1 3 0x740f, 36740 41.0°C 42.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-12-18T03:36:12.8481373Z ==================================================================================================================== 2024-12-18T03:36:12.8482022Z =============================================== End of ROCm SMI Log ================================================ 2024-12-18T03:36:12.8609931Z ##[group]Run rocminfo 2024-12-18T03:36:12.8610217Z rocminfo 2024-12-18T03:36:12.8641584Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.8642066Z env: 2024-12-18T03:36:12.8642331Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.8642705Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.8643115Z ##[endgroup] 2024-12-18T03:36:12.9376253Z ROCk module version 6.8.5 is loaded 2024-12-18T03:36:12.9376642Z ===================== 2024-12-18T03:36:12.9376866Z HSA System Attributes 2024-12-18T03:36:12.9377088Z ===================== 2024-12-18T03:36:12.9377306Z Runtime Version: 1.14 2024-12-18T03:36:12.9377540Z Runtime Ext Version: 1.6 2024-12-18T03:36:12.9377792Z System Timestamp Freq.: 1000.000000MHz 2024-12-18T03:36:12.9378236Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-12-18T03:36:12.9378695Z Machine Model: LARGE 2024-12-18T03:36:12.9379052Z System Endianness: LITTLE 2024-12-18T03:36:12.9379357Z Mwaitx: DISABLED 2024-12-18T03:36:12.9379600Z DMAbuf Support: YES 2024-12-18T03:36:12.9379755Z 2024-12-18T03:36:12.9379829Z ========== 2024-12-18T03:36:12.9380033Z HSA Agents 2024-12-18T03:36:12.9380229Z ========== 2024-12-18T03:36:12.9380440Z ******* 2024-12-18T03:36:12.9380641Z Agent 1 2024-12-18T03:36:12.9380840Z ******* 2024-12-18T03:36:12.9381090Z Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:36:12.9381445Z Uuid: CPU-XX 2024-12-18T03:36:12.9381805Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:36:12.9385583Z Vendor Name: CPU 2024-12-18T03:36:12.9385944Z Feature: None specified 2024-12-18T03:36:12.9386274Z Profile: FULL_PROFILE 2024-12-18T03:36:12.9386598Z Float Round Mode: NEAR 2024-12-18T03:36:12.9386937Z Max Queue Number: 0(0x0) 2024-12-18T03:36:12.9387269Z Queue Min Size: 0(0x0) 2024-12-18T03:36:12.9387584Z Queue Max Size: 0(0x0) 2024-12-18T03:36:12.9387905Z Queue Type: MULTI 2024-12-18T03:36:12.9388218Z Node: 0 2024-12-18T03:36:12.9388519Z Device Type: CPU 2024-12-18T03:36:12.9388808Z Cache Info: 2024-12-18T03:36:12.9389059Z L1: 32768(0x8000) KB 2024-12-18T03:36:12.9389365Z Chip ID: 0(0x0) 2024-12-18T03:36:12.9389694Z ASIC Revision: 0(0x0) 2024-12-18T03:36:12.9390034Z Cacheline Size: 64(0x40) 2024-12-18T03:36:12.9390376Z Max Clock Freq. (MHz): 2600 2024-12-18T03:36:12.9390702Z BDFID: 0 2024-12-18T03:36:12.9391026Z Internal Node ID: 0 2024-12-18T03:36:12.9391352Z Compute Unit: 32 2024-12-18T03:36:12.9391672Z SIMDs per CU: 0 2024-12-18T03:36:12.9391993Z Shader Engines: 0 2024-12-18T03:36:12.9392335Z Shader Arrs. per Eng.: 0 2024-12-18T03:36:12.9392685Z WatchPts on Addr. Ranges:1 2024-12-18T03:36:12.9393217Z Memory Properties: 2024-12-18T03:36:12.9393462Z Features: None 2024-12-18T03:36:12.9393693Z Pool Info: 2024-12-18T03:36:12.9393922Z Pool 1 2024-12-18T03:36:12.9394198Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:36:12.9394541Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:36:12.9394867Z Allocatable: TRUE 2024-12-18T03:36:12.9395210Z Alloc Granule: 4KB 2024-12-18T03:36:12.9395564Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9395927Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9396277Z Accessible by all: TRUE 2024-12-18T03:36:12.9396572Z Pool 2 2024-12-18T03:36:12.9396857Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:36:12.9397186Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:36:12.9397500Z Allocatable: TRUE 2024-12-18T03:36:12.9397834Z Alloc Granule: 4KB 2024-12-18T03:36:12.9398188Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9398539Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9398888Z Accessible by all: TRUE 2024-12-18T03:36:12.9399209Z Pool 3 2024-12-18T03:36:12.9399962Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:36:12.9400698Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:36:12.9401540Z Allocatable: TRUE 2024-12-18T03:36:12.9402623Z Alloc Granule: 4KB 2024-12-18T03:36:12.9403335Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9404025Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9404690Z Accessible by all: TRUE 2024-12-18T03:36:12.9405259Z ISA Info: 2024-12-18T03:36:12.9405719Z ******* 2024-12-18T03:36:12.9406104Z Agent 2 2024-12-18T03:36:12.9406478Z ******* 2024-12-18T03:36:12.9406951Z Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:36:12.9407557Z Uuid: CPU-XX 2024-12-18T03:36:12.9408196Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:36:12.9408858Z Vendor Name: CPU 2024-12-18T03:36:12.9409477Z Feature: None specified 2024-12-18T03:36:12.9410146Z Profile: FULL_PROFILE 2024-12-18T03:36:12.9410902Z Float Round Mode: NEAR 2024-12-18T03:36:12.9411665Z Max Queue Number: 0(0x0) 2024-12-18T03:36:12.9412401Z Queue Min Size: 0(0x0) 2024-12-18T03:36:12.9413136Z Queue Max Size: 0(0x0) 2024-12-18T03:36:12.9413850Z Queue Type: MULTI 2024-12-18T03:36:12.9414551Z Node: 1 2024-12-18T03:36:12.9415261Z Device Type: CPU 2024-12-18T03:36:12.9415909Z Cache Info: 2024-12-18T03:36:12.9416468Z L1: 32768(0x8000) KB 2024-12-18T03:36:12.9416922Z Chip ID: 0(0x0) 2024-12-18T03:36:12.9417309Z ASIC Revision: 0(0x0) 2024-12-18T03:36:12.9417805Z Cacheline Size: 64(0x40) 2024-12-18T03:36:12.9418139Z Max Clock Freq. (MHz): 2600 2024-12-18T03:36:12.9418474Z BDFID: 0 2024-12-18T03:36:12.9418781Z Internal Node ID: 1 2024-12-18T03:36:12.9419104Z Compute Unit: 32 2024-12-18T03:36:12.9419419Z SIMDs per CU: 0 2024-12-18T03:36:12.9419738Z Shader Engines: 0 2024-12-18T03:36:12.9420074Z Shader Arrs. per Eng.: 0 2024-12-18T03:36:12.9420443Z WatchPts on Addr. Ranges:1 2024-12-18T03:36:12.9420740Z Memory Properties: 2024-12-18T03:36:12.9420964Z Features: None 2024-12-18T03:36:12.9421201Z Pool Info: 2024-12-18T03:36:12.9421411Z Pool 1 2024-12-18T03:36:12.9421693Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:36:12.9422021Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:36:12.9422342Z Allocatable: TRUE 2024-12-18T03:36:12.9422733Z Alloc Granule: 4KB 2024-12-18T03:36:12.9423367Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9424178Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9424975Z Accessible by all: TRUE 2024-12-18T03:36:12.9425649Z Pool 2 2024-12-18T03:36:12.9426158Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:36:12.9426783Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:36:12.9427646Z Allocatable: TRUE 2024-12-18T03:36:12.9428312Z Alloc Granule: 4KB 2024-12-18T03:36:12.9429089Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9429771Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9430427Z Accessible by all: TRUE 2024-12-18T03:36:12.9430993Z Pool 3 2024-12-18T03:36:12.9431510Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:36:12.9432120Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:36:12.9432735Z Allocatable: TRUE 2024-12-18T03:36:12.9433382Z Alloc Granule: 4KB 2024-12-18T03:36:12.9434048Z Alloc Recommended Granule:4KB 2024-12-18T03:36:12.9434754Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9435424Z Accessible by all: TRUE 2024-12-18T03:36:12.9435979Z ISA Info: 2024-12-18T03:36:12.9436382Z ******* 2024-12-18T03:36:12.9436774Z Agent 3 2024-12-18T03:36:12.9437146Z ******* 2024-12-18T03:36:12.9437586Z Name: gfx90a 2024-12-18T03:36:12.9438180Z Uuid: GPU-57cd99ed4cd4643e 2024-12-18T03:36:12.9438821Z Marketing Name: AMD Instinct MI210 2024-12-18T03:36:12.9439588Z Vendor Name: AMD 2024-12-18T03:36:12.9440234Z Feature: KERNEL_DISPATCH 2024-12-18T03:36:12.9440856Z Profile: BASE_PROFILE 2024-12-18T03:36:12.9441499Z Float Round Mode: NEAR 2024-12-18T03:36:12.9442457Z Max Queue Number: 128(0x80) 2024-12-18T03:36:12.9443091Z Queue Min Size: 64(0x40) 2024-12-18T03:36:12.9443701Z Queue Max Size: 131072(0x20000) 2024-12-18T03:36:12.9444331Z Queue Type: MULTI 2024-12-18T03:36:12.9444922Z Node: 2 2024-12-18T03:36:12.9445504Z Device Type: GPU 2024-12-18T03:36:12.9446054Z Cache Info: 2024-12-18T03:36:12.9446509Z L1: 16(0x10) KB 2024-12-18T03:36:12.9447046Z L2: 8192(0x2000) KB 2024-12-18T03:36:12.9447602Z Chip ID: 29711(0x740f) 2024-12-18T03:36:12.9448232Z ASIC Revision: 1(0x1) 2024-12-18T03:36:12.9448931Z Cacheline Size: 64(0x40) 2024-12-18T03:36:12.9449701Z Max Clock Freq. (MHz): 1700 2024-12-18T03:36:12.9450422Z BDFID: 768 2024-12-18T03:36:12.9451127Z Internal Node ID: 2 2024-12-18T03:36:12.9451881Z Compute Unit: 104 2024-12-18T03:36:12.9452617Z SIMDs per CU: 4 2024-12-18T03:36:12.9453361Z Shader Engines: 8 2024-12-18T03:36:12.9454149Z Shader Arrs. per Eng.: 1 2024-12-18T03:36:12.9454952Z WatchPts on Addr. Ranges:4 2024-12-18T03:36:12.9455435Z Coherent Host Access: FALSE 2024-12-18T03:36:12.9455792Z Memory Properties: 2024-12-18T03:36:12.9456041Z Features: KERNEL_DISPATCH 2024-12-18T03:36:12.9456505Z Fast F16 Operation: TRUE 2024-12-18T03:36:12.9456861Z Wavefront Size: 64(0x40) 2024-12-18T03:36:12.9457209Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:36:12.9457519Z Workgroup Max Size per Dimension: 2024-12-18T03:36:12.9457784Z x 1024(0x400) 2024-12-18T03:36:12.9458055Z y 1024(0x400) 2024-12-18T03:36:12.9458319Z z 1024(0x400) 2024-12-18T03:36:12.9458620Z Max Waves Per CU: 32(0x20) 2024-12-18T03:36:12.9458950Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:36:12.9459274Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:36:12.9459571Z Grid Max Size per Dimension: 2024-12-18T03:36:12.9459823Z x 4294967295(0xffffffff) 2024-12-18T03:36:12.9460142Z y 4294967295(0xffffffff) 2024-12-18T03:36:12.9460415Z z 4294967295(0xffffffff) 2024-12-18T03:36:12.9460730Z Max fbarriers/Workgrp: 32 2024-12-18T03:36:12.9468309Z Packet Processor uCode:: 83 2024-12-18T03:36:12.9469107Z SDMA engine uCode:: 8 2024-12-18T03:36:12.9469826Z IOMMU Support:: None 2024-12-18T03:36:12.9470407Z Pool Info: 2024-12-18T03:36:12.9470836Z Pool 1 2024-12-18T03:36:12.9471379Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:36:12.9472034Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:36:12.9472692Z Allocatable: TRUE 2024-12-18T03:36:12.9473743Z Alloc Granule: 4KB 2024-12-18T03:36:12.9474431Z Alloc Recommended Granule:2048KB 2024-12-18T03:36:12.9475148Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9475845Z Accessible by all: FALSE 2024-12-18T03:36:12.9476424Z Pool 2 2024-12-18T03:36:12.9476973Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:36:12.9477624Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:36:12.9478265Z Allocatable: TRUE 2024-12-18T03:36:12.9478921Z Alloc Granule: 4KB 2024-12-18T03:36:12.9479734Z Alloc Recommended Granule:2048KB 2024-12-18T03:36:12.9480425Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9481100Z Accessible by all: FALSE 2024-12-18T03:36:12.9481681Z Pool 3 2024-12-18T03:36:12.9482182Z Segment: GROUP 2024-12-18T03:36:12.9482757Z Size: 64(0x40) KB 2024-12-18T03:36:12.9483370Z Allocatable: FALSE 2024-12-18T03:36:12.9484017Z Alloc Granule: 0KB 2024-12-18T03:36:12.9484678Z Alloc Recommended Granule:0KB 2024-12-18T03:36:12.9485375Z Alloc Alignment: 0KB 2024-12-18T03:36:12.9486074Z Accessible by all: FALSE 2024-12-18T03:36:12.9486656Z ISA Info: 2024-12-18T03:36:12.9487058Z ISA 1 2024-12-18T03:36:12.9487602Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:36:12.9488619Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:36:12.9489420Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:36:12.9490219Z Default Rounding Mode: NEAR 2024-12-18T03:36:12.9491050Z Default Rounding Mode: NEAR 2024-12-18T03:36:12.9491797Z Fast f16: TRUE 2024-12-18T03:36:12.9492552Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:36:12.9493278Z Workgroup Max Size per Dimension: 2024-12-18T03:36:12.9493903Z x 1024(0x400) 2024-12-18T03:36:12.9494548Z y 1024(0x400) 2024-12-18T03:36:12.9495177Z z 1024(0x400) 2024-12-18T03:36:12.9495624Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:36:12.9495999Z Grid Max Size per Dimension: 2024-12-18T03:36:12.9496263Z x 4294967295(0xffffffff) 2024-12-18T03:36:12.9496536Z y 4294967295(0xffffffff) 2024-12-18T03:36:12.9496819Z z 4294967295(0xffffffff) 2024-12-18T03:36:12.9497134Z FBarrier Max Size: 32 2024-12-18T03:36:12.9497419Z ******* 2024-12-18T03:36:12.9497632Z Agent 4 2024-12-18T03:36:12.9497832Z ******* 2024-12-18T03:36:12.9498054Z Name: gfx90a 2024-12-18T03:36:12.9498365Z Uuid: GPU-6b7c2bd8af06ff6a 2024-12-18T03:36:12.9498702Z Marketing Name: AMD Instinct MI210 2024-12-18T03:36:12.9499035Z Vendor Name: AMD 2024-12-18T03:36:12.9499373Z Feature: KERNEL_DISPATCH 2024-12-18T03:36:12.9499858Z Profile: BASE_PROFILE 2024-12-18T03:36:12.9500178Z Float Round Mode: NEAR 2024-12-18T03:36:12.9500513Z Max Queue Number: 128(0x80) 2024-12-18T03:36:12.9500837Z Queue Min Size: 64(0x40) 2024-12-18T03:36:12.9501155Z Queue Max Size: 131072(0x20000) 2024-12-18T03:36:12.9501534Z Queue Type: MULTI 2024-12-18T03:36:12.9502036Z Node: 3 2024-12-18T03:36:12.9502732Z Device Type: GPU 2024-12-18T03:36:12.9503381Z Cache Info: 2024-12-18T03:36:12.9503917Z L1: 16(0x10) KB 2024-12-18T03:36:12.9504523Z L2: 8192(0x2000) KB 2024-12-18T03:36:12.9505089Z Chip ID: 29711(0x740f) 2024-12-18T03:36:12.9505706Z ASIC Revision: 1(0x1) 2024-12-18T03:36:12.9506335Z Cacheline Size: 64(0x40) 2024-12-18T03:36:12.9506966Z Max Clock Freq. (MHz): 1700 2024-12-18T03:36:12.9507562Z BDFID: 33536 2024-12-18T03:36:12.9508157Z Internal Node ID: 3 2024-12-18T03:36:12.9508774Z Compute Unit: 104 2024-12-18T03:36:12.9509378Z SIMDs per CU: 4 2024-12-18T03:36:12.9510008Z Shader Engines: 8 2024-12-18T03:36:12.9510651Z Shader Arrs. per Eng.: 1 2024-12-18T03:36:12.9511321Z WatchPts on Addr. Ranges:4 2024-12-18T03:36:12.9512254Z Coherent Host Access: FALSE 2024-12-18T03:36:12.9512880Z Memory Properties: 2024-12-18T03:36:12.9513364Z Features: KERNEL_DISPATCH 2024-12-18T03:36:12.9513975Z Fast F16 Operation: TRUE 2024-12-18T03:36:12.9514623Z Wavefront Size: 64(0x40) 2024-12-18T03:36:12.9515288Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:36:12.9515888Z Workgroup Max Size per Dimension: 2024-12-18T03:36:12.9516387Z x 1024(0x400) 2024-12-18T03:36:12.9516920Z y 1024(0x400) 2024-12-18T03:36:12.9517434Z z 1024(0x400) 2024-12-18T03:36:12.9517992Z Max Waves Per CU: 32(0x20) 2024-12-18T03:36:12.9518644Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:36:12.9519308Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:36:12.9519978Z Grid Max Size per Dimension: 2024-12-18T03:36:12.9520433Z x 4294967295(0xffffffff) 2024-12-18T03:36:12.9520951Z y 4294967295(0xffffffff) 2024-12-18T03:36:12.9521463Z z 4294967295(0xffffffff) 2024-12-18T03:36:12.9522063Z Max fbarriers/Workgrp: 32 2024-12-18T03:36:12.9522760Z Packet Processor uCode:: 83 2024-12-18T03:36:12.9523431Z SDMA engine uCode:: 8 2024-12-18T03:36:12.9524081Z IOMMU Support:: None 2024-12-18T03:36:12.9524633Z Pool Info: 2024-12-18T03:36:12.9525031Z Pool 1 2024-12-18T03:36:12.9525559Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:36:12.9526513Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:36:12.9527117Z Allocatable: TRUE 2024-12-18T03:36:12.9527770Z Alloc Granule: 4KB 2024-12-18T03:36:12.9528457Z Alloc Recommended Granule:2048KB 2024-12-18T03:36:12.9529231Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9530003Z Accessible by all: FALSE 2024-12-18T03:36:12.9530674Z Pool 2 2024-12-18T03:36:12.9531338Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:36:12.9532074Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:36:12.9532790Z Allocatable: TRUE 2024-12-18T03:36:12.9533549Z Alloc Granule: 4KB 2024-12-18T03:36:12.9534359Z Alloc Recommended Granule:2048KB 2024-12-18T03:36:12.9535170Z Alloc Alignment: 4KB 2024-12-18T03:36:12.9535633Z Accessible by all: FALSE 2024-12-18T03:36:12.9535969Z Pool 3 2024-12-18T03:36:12.9536229Z Segment: GROUP 2024-12-18T03:36:12.9536527Z Size: 64(0x40) KB 2024-12-18T03:36:12.9536833Z Allocatable: FALSE 2024-12-18T03:36:12.9537180Z Alloc Granule: 0KB 2024-12-18T03:36:12.9537543Z Alloc Recommended Granule:0KB 2024-12-18T03:36:12.9537912Z Alloc Alignment: 0KB 2024-12-18T03:36:12.9538256Z Accessible by all: FALSE 2024-12-18T03:36:12.9538709Z ISA Info: 2024-12-18T03:36:12.9538931Z ISA 1 2024-12-18T03:36:12.9539205Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:36:12.9539576Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:36:12.9539926Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:36:12.9540267Z Default Rounding Mode: NEAR 2024-12-18T03:36:12.9540619Z Default Rounding Mode: NEAR 2024-12-18T03:36:12.9540946Z Fast f16: TRUE 2024-12-18T03:36:12.9541306Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:36:12.9541696Z Workgroup Max Size per Dimension: 2024-12-18T03:36:12.9542023Z x 1024(0x400) 2024-12-18T03:36:12.9542353Z y 1024(0x400) 2024-12-18T03:36:12.9542679Z z 1024(0x400) 2024-12-18T03:36:12.9543028Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:36:12.9543350Z Grid Max Size per Dimension: 2024-12-18T03:36:12.9543599Z x 4294967295(0xffffffff) 2024-12-18T03:36:12.9543881Z y 4294967295(0xffffffff) 2024-12-18T03:36:12.9544151Z z 4294967295(0xffffffff) 2024-12-18T03:36:12.9544480Z FBarrier Max Size: 32 2024-12-18T03:36:12.9544772Z *** Done *** 2024-12-18T03:36:12.9643662Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-12-18T03:36:12.9644421Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-12-18T03:36:12.9645682Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T03:36:12.9647214Z if [[ $ngpu -eq 0 ]]; then 2024-12-18T03:36:12.9647832Z  echo "Error: Failed to detect any GPUs on the runner" 2024-12-18T03:36:12.9648455Z  echo "$msg" 2024-12-18T03:36:12.9648857Z  exit 1 2024-12-18T03:36:12.9649276Z fi 2024-12-18T03:36:12.9649702Z if [[ $ngpu -eq 1 ]]; then 2024-12-18T03:36:12.9650585Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-12-18T03:36:12.9651490Z  echo "$msg" 2024-12-18T03:36:12.9651963Z  exit 1 2024-12-18T03:36:12.9652378Z fi 2024-12-18T03:36:12.9693138Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:12.9693887Z env: 2024-12-18T03:36:12.9694311Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:12.9694855Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:12.9695190Z ##[endgroup] 2024-12-18T03:36:13.0597756Z Prepare all required actions 2024-12-18T03:36:13.0621172Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-12-18T03:36:13.0621409Z with: 2024-12-18T03:36:13.0621563Z diskspace-cutoff: 70 2024-12-18T03:36:13.0621731Z env: 2024-12-18T03:36:13.0621879Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:13.0622090Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:13.0622322Z ##[endgroup] 2024-12-18T03:36:13.0652636Z ##[group]Run set -ex 2024-12-18T03:36:13.0652834Z set -ex 2024-12-18T03:36:13.0653007Z diskspace_cutoff=70 2024-12-18T03:36:13.0653272Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-12-18T03:36:13.0653680Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T03:36:13.0654298Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T03:36:13.0655037Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-12-18T03:36:13.0655312Z  docker system prune -af 2024-12-18T03:36:13.0655651Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T03:36:13.0656039Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-12-18T03:36:13.0656447Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-12-18T03:36:13.0656810Z  echo "$msg" 2024-12-18T03:36:13.0656990Z  exit 1 2024-12-18T03:36:13.0657158Z  else 2024-12-18T03:36:13.0657349Z  difference=$((diskspace - diskspace_new)) 2024-12-18T03:36:13.0657629Z  echo "Diskspace saved: $difference percent" 2024-12-18T03:36:13.0657866Z  fi 2024-12-18T03:36:13.0658011Z fi 2024-12-18T03:36:13.0680399Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:13.0680681Z env: 2024-12-18T03:36:13.0680826Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:13.0681048Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:13.0681278Z ##[endgroup] 2024-12-18T03:36:13.0716982Z + diskspace_cutoff=70 2024-12-18T03:36:13.0720994Z ++ docker info -f '{{.DockerRootDir}}' 2024-12-18T03:36:13.1233940Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-12-18T03:36:13.1241017Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-12-18T03:36:13.1241729Z ++ sed -n 2p 2024-12-18T03:36:13.1241946Z ++ sed s/%// 2024-12-18T03:36:13.1245603Z ++ sed 's/ //' 2024-12-18T03:36:13.1262948Z + diskspace=15 2024-12-18T03:36:13.1263455Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-12-18T03:36:13.1263950Z + [[ 15 -ge 70 ]] 2024-12-18T03:36:13.1314852Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:36:13.1315943Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:36:13.1316534Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:36:13.1357353Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:13.1358017Z env: 2024-12-18T03:36:13.1358374Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:13.1358899Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:13.1359577Z ##[endgroup] 2024-12-18T03:36:13.1469639Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-12-18T03:36:13.1470158Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-12-18T03:36:13.1470725Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-12-18T03:36:13.1491834Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:13.1492112Z env: 2024-12-18T03:36:13.1492309Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:13.1492532Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:13.1492773Z ##[endgroup] 2024-12-18T03:36:13.1604160Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-12-18T03:36:13.1604826Z with: 2024-12-18T03:36:13.1605432Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-12-18T03:36:13.1606176Z aws-region: us-east-1 2024-12-18T03:36:13.1606604Z role-duration-seconds: 18000 2024-12-18T03:36:13.1607092Z audience: sts.amazonaws.com 2024-12-18T03:36:13.1607530Z env: 2024-12-18T03:36:13.1607851Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:13.1608341Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:13.1609248Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:13.1610075Z ##[endgroup] 2024-12-18T03:36:13.7047746Z Assuming role with OIDC 2024-12-18T03:36:14.0466082Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-12-18T03:36:14.1604006Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-12-18T03:36:14.1604639Z with: 2024-12-18T03:36:14.1605030Z mask-password: true 2024-12-18T03:36:14.1605487Z registry-type: private 2024-12-18T03:36:14.1605938Z skip-logout: false 2024-12-18T03:36:14.1606329Z env: 2024-12-18T03:36:14.1606679Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:14.1607208Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:14.1608165Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:14.1609049Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:14.1609593Z AWS_REGION: us-east-1 2024-12-18T03:36:14.1610411Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:14.1611192Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:14.1621818Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:14.1622341Z ##[endgroup] 2024-12-18T03:36:14.6659944Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3263705Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T03:36:15.3264760Z with: 2024-12-18T03:36:15.3265900Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3267174Z docker-build-dir: .ci/docker 2024-12-18T03:36:15.3267670Z working-directory: . 2024-12-18T03:36:15.3268279Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3268955Z force-push: false 2024-12-18T03:36:15.3269339Z env: 2024-12-18T03:36:15.3269706Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:15.3270231Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:15.3271178Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:15.3272061Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:15.3272551Z AWS_REGION: us-east-1 2024-12-18T03:36:15.3273696Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:15.3274359Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:15.3284351Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:15.3284801Z ##[endgroup] 2024-12-18T03:36:15.3317299Z ##[group]Run set -ex 2024-12-18T03:36:15.3317816Z set -ex 2024-12-18T03:36:15.3318205Z  2024-12-18T03:36:15.3318917Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T03:36:15.3320184Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T03:36:15.3321135Z # job could then download the pre-built image as usual 2024-12-18T03:36:15.3321994Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T03:36:15.3322791Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3323550Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3324233Z  2024-12-18T03:36:15.3324865Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T03:36:15.3325608Z  exit 0 2024-12-18T03:36:15.3325989Z else 2024-12-18T03:36:15.3326440Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3326976Z fi 2024-12-18T03:36:15.3327330Z  2024-12-18T03:36:15.3327900Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T03:36:15.3328892Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T03:36:15.3329776Z  # use it as it is, but first let's extract the tag 2024-12-18T03:36:15.3330714Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T03:36:15.3331720Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3332681Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3333503Z else 2024-12-18T03:36:15.3334125Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T03:36:15.3335029Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3335890Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3336402Z fi 2024-12-18T03:36:15.3379993Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:15.3380335Z env: 2024-12-18T03:36:15.3380534Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:15.3380812Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:15.3381354Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:15.3382152Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:15.3382715Z AWS_REGION: us-east-1 2024-12-18T03:36:15.3383394Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:15.3384625Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:15.3394439Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:15.3394895Z REPO_NAME: pytorch 2024-12-18T03:36:15.3396032Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3397260Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T03:36:15.3397909Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3398568Z ##[endgroup] 2024-12-18T03:36:15.3474980Z + [[ ! -d .ci/docker ]] 2024-12-18T03:36:15.3475544Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T03:36:15.3476084Z + echo skip=false 2024-12-18T03:36:15.3477685Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T03:36:15.3489437Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3491775Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T03:36:15.3516165Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3517020Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3519859Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3563369Z ##[group]Run set +e 2024-12-18T03:36:15.3563773Z set +e 2024-12-18T03:36:15.3564074Z set -x 2024-12-18T03:36:15.3564363Z  2024-12-18T03:36:15.3564642Z login() { 2024-12-18T03:36:15.3565271Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T03:36:15.3565930Z } 2024-12-18T03:36:15.3566203Z  2024-12-18T03:36:15.3566480Z retry () { 2024-12-18T03:36:15.3566858Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T03:36:15.3567273Z } 2024-12-18T03:36:15.3567540Z  2024-12-18T03:36:15.3567837Z retry login "${DOCKER_REGISTRY}" 2024-12-18T03:36:15.3568230Z  2024-12-18T03:36:15.3568512Z START_TIME=$(date +%s) 2024-12-18T03:36:15.3568887Z # Wait up to 90 minutes 2024-12-18T03:36:15.3569355Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T03:36:15.3570083Z  # Check if image already exists, if it does then skip building it 2024-12-18T03:36:15.3570818Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T03:36:15.3571362Z  exit 0 2024-12-18T03:36:15.3571718Z  fi 2024-12-18T03:36:15.3572048Z  2024-12-18T03:36:15.3572616Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T03:36:15.3573595Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T03:36:15.3574580Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T03:36:15.3575364Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T03:36:15.3575956Z  # It's a Docker build job, let's build the image 2024-12-18T03:36:15.3576317Z  break 2024-12-18T03:36:15.3576560Z  else 2024-12-18T03:36:15.3576866Z  # It's a regular build job, wait for the image to become available 2024-12-18T03:36:15.3577227Z  sleep 300 2024-12-18T03:36:15.3577450Z  fi 2024-12-18T03:36:15.3577647Z done 2024-12-18T03:36:15.3577840Z  2024-12-18T03:36:15.3578155Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T03:36:15.3578643Z # be empty. The default action would be to continue rebuild the image 2024-12-18T03:36:15.3579277Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T03:36:15.3579693Z  # if we're on the base branch then use the parent commit 2024-12-18T03:36:15.3580039Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T03:36:15.3580312Z else 2024-12-18T03:36:15.3580597Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T03:36:15.3581007Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T03:36:15.3581349Z fi 2024-12-18T03:36:15.3581569Z  2024-12-18T03:36:15.3581872Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T03:36:15.3582426Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3582918Z  2024-12-18T03:36:15.3583601Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T03:36:15.3584411Z  exit 0 2024-12-18T03:36:15.3584728Z fi 2024-12-18T03:36:15.3585001Z  2024-12-18T03:36:15.3585394Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T03:36:15.3586509Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T03:36:15.3587232Z  exit 1 2024-12-18T03:36:15.3587512Z fi 2024-12-18T03:36:15.3587779Z  2024-12-18T03:36:15.3588236Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T03:36:15.3589053Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T03:36:15.3589786Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T03:36:15.3590621Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T03:36:15.3591553Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T03:36:15.3592119Z fi 2024-12-18T03:36:15.3592407Z  2024-12-18T03:36:15.3592734Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:36:15.3624722Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:15.3625223Z env: 2024-12-18T03:36:15.3625510Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:15.3625925Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:15.3626645Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:15.3627319Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:15.3627688Z AWS_REGION: us-east-1 2024-12-18T03:36:15.3628114Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:15.3628646Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:15.3635955Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:15.3636307Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T03:36:15.3636747Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:36:15.3637731Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3638685Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:15.3639242Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3639841Z DOCKER_PUSH: 2024-12-18T03:36:15.3640141Z ##[endgroup] 2024-12-18T03:36:15.3699587Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3700027Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:15.3703984Z + aws ecr get-login-password --region us-east-1 2024-12-18T03:36:15.3705751Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:16.8217281Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-12-18T03:36:16.8218472Z Configure a credential helper to remove this warning. See 2024-12-18T03:36:16.8219510Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T03:36:16.8220746Z 2024-12-18T03:36:16.8221063Z Login Succeeded 2024-12-18T03:36:16.8261972Z ++ date +%s 2024-12-18T03:36:16.8276911Z + START_TIME=1734492976 2024-12-18T03:36:16.8281503Z ++ date +%s 2024-12-18T03:36:16.8294128Z + [[ 1734487576 -lt 1734492976 ]] 2024-12-18T03:36:16.8294912Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:18.1614539Z { 2024-12-18T03:36:18.1615091Z "schemaVersion": 2, 2024-12-18T03:36:18.1615904Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T03:36:18.1616710Z "config": { 2024-12-18T03:36:18.1617364Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T03:36:18.1618059Z "size": 24144, 2024-12-18T03:36:18.1618769Z "digest": "sha256:07b91116eb1063321da7a63f31f1ef61d34f7dc8b7d56ec47c24f12af9f38edb" 2024-12-18T03:36:18.1619565Z }, 2024-12-18T03:36:18.1619908Z "layers": [ 2024-12-18T03:36:18.1620884Z { 2024-12-18T03:36:18.1621587Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1622517Z "size": 28583948, 2024-12-18T03:36:18.1623368Z "digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1" 2024-12-18T03:36:18.1624318Z }, 2024-12-18T03:36:18.1624703Z { 2024-12-18T03:36:18.1625403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1626242Z "size": 1825, 2024-12-18T03:36:18.1626748Z "digest": "sha256:7e5cdcc8d39cbfc3cbf5fe048da71a36c2c80d4a1e184cccc8c9f02e7e076f64" 2024-12-18T03:36:18.1627203Z }, 2024-12-18T03:36:18.1627372Z { 2024-12-18T03:36:18.1627664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1628020Z "size": 312642362, 2024-12-18T03:36:18.1628389Z "digest": "sha256:231ab5ed5e979cc01fda602a87a43fd305466b7cc01b98a76690ffa0b716f312" 2024-12-18T03:36:18.1628788Z }, 2024-12-18T03:36:18.1628957Z { 2024-12-18T03:36:18.1629245Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1629610Z "size": 863, 2024-12-18T03:36:18.1629974Z "digest": "sha256:fcbefe9ad79ece7f701532eae83e9de139fe1c0949a7813e55dd3a53003af9cd" 2024-12-18T03:36:18.1630385Z }, 2024-12-18T03:36:18.1630553Z { 2024-12-18T03:36:18.1630833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1631187Z "size": 106, 2024-12-18T03:36:18.1631561Z "digest": "sha256:af67ad6d287ebd8cbde55a749a5bbc06170aea40e4902c93cd34922e10c8c408" 2024-12-18T03:36:18.1631969Z }, 2024-12-18T03:36:18.1632140Z { 2024-12-18T03:36:18.1632417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1632771Z "size": 704, 2024-12-18T03:36:18.1633184Z "digest": "sha256:a87dd7b9e020ba8d0d077d74c105bbb7fe8cdd8f1e6ae0c98b84c3e2c06ace60" 2024-12-18T03:36:18.1633592Z }, 2024-12-18T03:36:18.1633763Z { 2024-12-18T03:36:18.1634043Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1634400Z "size": 1257, 2024-12-18T03:36:18.1634766Z "digest": "sha256:aa67f5368a1ddc10ff5b57e69f4ebcf2f83e0f32dfe4648e2a21aee041f107f8" 2024-12-18T03:36:18.1635167Z }, 2024-12-18T03:36:18.1635334Z { 2024-12-18T03:36:18.1635612Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1635968Z "size": 3708, 2024-12-18T03:36:18.1636327Z "digest": "sha256:e71aede6d46007bf2a44f7cd14800cf5486c895c0d90a0b18ea9c30e6fff4571" 2024-12-18T03:36:18.1636730Z }, 2024-12-18T03:36:18.1636899Z { 2024-12-18T03:36:18.1637174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1637534Z "size": 1860, 2024-12-18T03:36:18.1637891Z "digest": "sha256:ee65ff0efe5e4f68873b0389e60a87f13f1a1bb0df1178fd70e5e83e59753fab" 2024-12-18T03:36:18.1638296Z }, 2024-12-18T03:36:18.1638474Z { 2024-12-18T03:36:18.1638755Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1639271Z "size": 701, 2024-12-18T03:36:18.1639756Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:36:18.1640154Z }, 2024-12-18T03:36:18.1640351Z { 2024-12-18T03:36:18.1640628Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1640987Z "size": 2717169179, 2024-12-18T03:36:18.1641355Z "digest": "sha256:8362d69f07765b32e976c14dd6625372f9d81f263db1dc4ef06f46d99571f992" 2024-12-18T03:36:18.1641748Z }, 2024-12-18T03:36:18.1641918Z { 2024-12-18T03:36:18.1642198Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1642549Z "size": 380, 2024-12-18T03:36:18.1642885Z "digest": "sha256:0d29ca60b09574f6c0756a7025b88000a59b1009992fa1478637f5bdc5784ad2" 2024-12-18T03:36:18.1643270Z }, 2024-12-18T03:36:18.1643437Z { 2024-12-18T03:36:18.1643713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1644065Z "size": 12135, 2024-12-18T03:36:18.1644574Z "digest": "sha256:6697412e0463180656f0051e70d45098fb97bbdb91515623a3ca0d0785ed26c3" 2024-12-18T03:36:18.1644957Z }, 2024-12-18T03:36:18.1645120Z { 2024-12-18T03:36:18.1645395Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1645748Z "size": 504, 2024-12-18T03:36:18.1646095Z "digest": "sha256:3fe85e92871bbecf1a51782c8349d745adf952dabe41e4023f16496e13c89d6e" 2024-12-18T03:36:18.1646490Z }, 2024-12-18T03:36:18.1646657Z { 2024-12-18T03:36:18.1646934Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1647287Z "size": 121477504, 2024-12-18T03:36:18.1647641Z "digest": "sha256:db8079b9a247c260d493898268f96390bcef45198f5799ea28a82894417d3409" 2024-12-18T03:36:18.1648028Z }, 2024-12-18T03:36:18.1648196Z { 2024-12-18T03:36:18.1648474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1648825Z "size": 109, 2024-12-18T03:36:18.1649187Z "digest": "sha256:1e1cc236bfa694641ad0d2d7fb1a76caa46f0b5364084a30d576a593cdf09838" 2024-12-18T03:36:18.1649598Z }, 2024-12-18T03:36:18.1649763Z { 2024-12-18T03:36:18.1650040Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1650391Z "size": 489, 2024-12-18T03:36:18.1650738Z "digest": "sha256:5594c816d49f271fafd93fedc956a82586a1ddb33f6a81c6c47324b2bc675e6f" 2024-12-18T03:36:18.1651132Z }, 2024-12-18T03:36:18.1651301Z { 2024-12-18T03:36:18.1651582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1651932Z "size": 296, 2024-12-18T03:36:18.1652271Z "digest": "sha256:d0419af25312d587b2c3532b144859e2155c14d2ca73864f20041dd565b0c9c7" 2024-12-18T03:36:18.1652667Z }, 2024-12-18T03:36:18.1652840Z { 2024-12-18T03:36:18.1653124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1653479Z "size": 103, 2024-12-18T03:36:18.1653832Z "digest": "sha256:4a7e3c6b1985b14cd275d8480a86dabccb21244b38cca3771881b2f87a880a7c" 2024-12-18T03:36:18.1654235Z }, 2024-12-18T03:36:18.1654410Z { 2024-12-18T03:36:18.1654691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1655142Z "size": 1474, 2024-12-18T03:36:18.1655605Z "digest": "sha256:e7e49899a1507f1355d8dd6a16d5beb03f1ced22428dc3b796a8c453fba8b41d" 2024-12-18T03:36:18.1656015Z }, 2024-12-18T03:36:18.1656185Z { 2024-12-18T03:36:18.1656471Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1656827Z "size": 430331313, 2024-12-18T03:36:18.1657185Z "digest": "sha256:907b5d385c1450d33d646dcce152dec74855431d5aa6d427f39d74f87ab7a183" 2024-12-18T03:36:18.1657586Z }, 2024-12-18T03:36:18.1657751Z { 2024-12-18T03:36:18.1658028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1658378Z "size": 163, 2024-12-18T03:36:18.1658727Z "digest": "sha256:e09946f27e30eab5749280587f84a64accfe85ccfaabf422c1fae010dd492cbd" 2024-12-18T03:36:18.1659297Z }, 2024-12-18T03:36:18.1659471Z { 2024-12-18T03:36:18.1659747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1660097Z "size": 1640, 2024-12-18T03:36:18.1660447Z "digest": "sha256:7294a9f24454fce9dc046e77ab28a73149429dde57793009bb3223eb424713de" 2024-12-18T03:36:18.1660864Z }, 2024-12-18T03:36:18.1661058Z { 2024-12-18T03:36:18.1661513Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1662326Z "size": 7880578701, 2024-12-18T03:36:18.1663171Z "digest": "sha256:5cbdd1d7b36d7b6e2d6428e2857ec4f8614e3becbdb0ec6e3b64f3cf63584e59" 2024-12-18T03:36:18.1664044Z }, 2024-12-18T03:36:18.1664370Z { 2024-12-18T03:36:18.1664908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1665593Z "size": 105, 2024-12-18T03:36:18.1666257Z "digest": "sha256:f528471f2fcaa24ae54a666344201877b06d22561708fd21987ac10c0873d6a0" 2024-12-18T03:36:18.1667010Z }, 2024-12-18T03:36:18.1667594Z { 2024-12-18T03:36:18.1668154Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1668858Z "size": 1116, 2024-12-18T03:36:18.1669520Z "digest": "sha256:e1276511038b9b5b700d3a69400a0692e260be884871347905717fbec3009ced" 2024-12-18T03:36:18.1670270Z }, 2024-12-18T03:36:18.1670588Z { 2024-12-18T03:36:18.1671140Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1671846Z "size": 318568913, 2024-12-18T03:36:18.1672554Z "digest": "sha256:f3d817e0be62e96a38e3a6596705a35dfa864a2658a4d4e92890f192eaa1103e" 2024-12-18T03:36:18.1673318Z }, 2024-12-18T03:36:18.1673636Z { 2024-12-18T03:36:18.1674168Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1674848Z "size": 111, 2024-12-18T03:36:18.1675541Z "digest": "sha256:babfeddb429861eef35685a83ff84f1b120f8eda1b6a1c552453065dc0bd6cb3" 2024-12-18T03:36:18.1676322Z }, 2024-12-18T03:36:18.1676642Z { 2024-12-18T03:36:18.1677195Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1677875Z "size": 1555, 2024-12-18T03:36:18.1678564Z "digest": "sha256:e00cbec1ecdd51cc9b1b2471a6bf61f3e93cc03540114c13375e35cb48169f90" 2024-12-18T03:36:18.1679349Z }, 2024-12-18T03:36:18.1679794Z { 2024-12-18T03:36:18.1680322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1681030Z "size": 107, 2024-12-18T03:36:18.1681707Z "digest": "sha256:60175e320c6277d6be778429e32b3dd271c508af81bb52bdfec8c5c6edb55a66" 2024-12-18T03:36:18.1682477Z }, 2024-12-18T03:36:18.1682797Z { 2024-12-18T03:36:18.1683322Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1683996Z "size": 166, 2024-12-18T03:36:18.1684682Z "digest": "sha256:e4fd8cd3183ce9d5e635650cd867e61f68b6edc31b37e605f9c038a2064e42c0" 2024-12-18T03:36:18.1685456Z }, 2024-12-18T03:36:18.1685776Z { 2024-12-18T03:36:18.1686317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1687003Z "size": 2842293, 2024-12-18T03:36:18.1687696Z "digest": "sha256:f79cba221c4638d554fcf21bde3a813a5ddd7d4900f5928e43c92c9307e9d58d" 2024-12-18T03:36:18.1688472Z }, 2024-12-18T03:36:18.1688794Z { 2024-12-18T03:36:18.1689332Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1690013Z "size": 107, 2024-12-18T03:36:18.1690754Z "digest": "sha256:e425c457ab3e0a6637feed1c3ef95776f4c43fc59336a9da4e0a7f19858529c2" 2024-12-18T03:36:18.1691670Z }, 2024-12-18T03:36:18.1692049Z { 2024-12-18T03:36:18.1692674Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1693474Z "size": 566, 2024-12-18T03:36:18.1694273Z "digest": "sha256:6d2cbef9a0f53dff8893f40509e211ffe284a5d8fb4e03923b9c9646b06ad2e7" 2024-12-18T03:36:18.1695191Z }, 2024-12-18T03:36:18.1695575Z { 2024-12-18T03:36:18.1696167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1696758Z "size": 43195564, 2024-12-18T03:36:18.1697128Z "digest": "sha256:2c4472ad27f9258eee43c37d8fdd0ec0db693fc1d33d03902c90d85850a2fa74" 2024-12-18T03:36:18.1697527Z }, 2024-12-18T03:36:18.1697695Z { 2024-12-18T03:36:18.1697975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1698325Z "size": 106, 2024-12-18T03:36:18.1698675Z "digest": "sha256:48ffbd6d25729ef5a8cbfc2950686efcd5cc5854803572b079a40e7d59b81481" 2024-12-18T03:36:18.1699072Z }, 2024-12-18T03:36:18.1699242Z { 2024-12-18T03:36:18.1699517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1699871Z "size": 295, 2024-12-18T03:36:18.1700220Z "digest": "sha256:cf56c8fabbc0842e7f589f295894c2b6eb980e7f5ec8f2232b8c105f503a46e5" 2024-12-18T03:36:18.1700621Z }, 2024-12-18T03:36:18.1700792Z { 2024-12-18T03:36:18.1701073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1701425Z "size": 88293, 2024-12-18T03:36:18.1701959Z "digest": "sha256:b1c0b18891861b71d1c07dfb5e1063225b3ee635b5cf8d088bc62755ae2a3140" 2024-12-18T03:36:18.1702501Z }, 2024-12-18T03:36:18.1702871Z { 2024-12-18T03:36:18.1703511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1704312Z "size": 106, 2024-12-18T03:36:18.1705089Z "digest": "sha256:3021fd4a208aeeab27bda896abedaba113f2a3845b3b037c57f855a429bcd61a" 2024-12-18T03:36:18.1705862Z }, 2024-12-18T03:36:18.1706177Z { 2024-12-18T03:36:18.1706706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1707382Z "size": 1402, 2024-12-18T03:36:18.1708084Z "digest": "sha256:a6f5eec4a2df539ca39d87dfd6ba37ef1c60304eda99ed48b657823b8b5a83aa" 2024-12-18T03:36:18.1708870Z }, 2024-12-18T03:36:18.1709191Z { 2024-12-18T03:36:18.1709729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1710400Z "size": 701, 2024-12-18T03:36:18.1711079Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:36:18.1711858Z }, 2024-12-18T03:36:18.1712181Z { 2024-12-18T03:36:18.1712720Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1713399Z "size": 137, 2024-12-18T03:36:18.1714089Z "digest": "sha256:89e8bdba21ffada6ab13d8b625b5349b0cc0398741ffd1c5155fdea44633255a" 2024-12-18T03:36:18.1714875Z }, 2024-12-18T03:36:18.1715193Z { 2024-12-18T03:36:18.1715727Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1716399Z "size": 120, 2024-12-18T03:36:18.1717076Z "digest": "sha256:7239a56eb3344ab4d10ba3a41fc1e7af5559c747ee876bb8660f0237ed85a2f0" 2024-12-18T03:36:18.1717848Z }, 2024-12-18T03:36:18.1718169Z { 2024-12-18T03:36:18.1718708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1719494Z "size": 5014581981, 2024-12-18T03:36:18.1720229Z "digest": "sha256:22093b6a474afa19ed0f46a0491aa69688d1f0b3954fb7614a8f8e2538f90a43" 2024-12-18T03:36:18.1721014Z }, 2024-12-18T03:36:18.1721368Z { 2024-12-18T03:36:18.1721908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1722583Z "size": 174, 2024-12-18T03:36:18.1723266Z "digest": "sha256:48aa755f3a17db6703cc7bff22b93fe216ee7f1112fb2e8330c2e1f7737822c2" 2024-12-18T03:36:18.1724040Z }, 2024-12-18T03:36:18.1724363Z { 2024-12-18T03:36:18.1724904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1725578Z "size": 209, 2024-12-18T03:36:18.1726236Z "digest": "sha256:789ba2fe5a5187199259980262efd9e420b90ff489372e2cd60c98470a3db40d" 2024-12-18T03:36:18.1726988Z }, 2024-12-18T03:36:18.1727302Z { 2024-12-18T03:36:18.1727835Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1728509Z "size": 701, 2024-12-18T03:36:18.1729175Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:36:18.1730315Z }, 2024-12-18T03:36:18.1730703Z { 2024-12-18T03:36:18.1731362Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1732194Z "size": 633, 2024-12-18T03:36:18.1733000Z "digest": "sha256:e2e30e00626e69012828cfe72a0d31832f6ff31805e36d44356537d5f7fcb9ba" 2024-12-18T03:36:18.1733919Z }, 2024-12-18T03:36:18.1734306Z { 2024-12-18T03:36:18.1734949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1735774Z "size": 296742339, 2024-12-18T03:36:18.1736387Z "digest": "sha256:c631a2525143e662e72b9b97c2e87b09a43a1038995e76b403dad6ba88966e41" 2024-12-18T03:36:18.1736855Z }, 2024-12-18T03:36:18.1737053Z { 2024-12-18T03:36:18.1737347Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1737703Z "size": 1932, 2024-12-18T03:36:18.1738064Z "digest": "sha256:08d3deeeb0fbaac4992048b95790661de095ddd8925926cfa2f2ec217bf4e0f7" 2024-12-18T03:36:18.1738468Z }, 2024-12-18T03:36:18.1738829Z { 2024-12-18T03:36:18.1739105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1739463Z "size": 15983403, 2024-12-18T03:36:18.1739831Z "digest": "sha256:f098b0de0dbe9076c39d2aa97b15a08de76d7c33be71cc1069255184e8716618" 2024-12-18T03:36:18.1740233Z }, 2024-12-18T03:36:18.1740403Z { 2024-12-18T03:36:18.1740680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:36:18.1741033Z "size": 54145664, 2024-12-18T03:36:18.1741387Z "digest": "sha256:4e73d9ed9f9510302a97c5ce44701037bd8453e91d3dbe54738434120cf00300" 2024-12-18T03:36:18.1741777Z } 2024-12-18T03:36:18.1741950Z ] 2024-12-18T03:36:18.1742122Z } 2024-12-18T03:36:18.1742304Z + exit 0 2024-12-18T03:36:18.1784137Z ##[group]Run set -eux 2024-12-18T03:36:18.1784700Z set -eux 2024-12-18T03:36:18.1786213Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T03:36:18.1838175Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:18.1838847Z env: 2024-12-18T03:36:18.1839225Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:18.1839910Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:18.1840868Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:18.1841763Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:18.1842254Z AWS_REGION: us-east-1 2024-12-18T03:36:18.1842867Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:18.1843531Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:18.1853324Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:18.1853769Z ##[endgroup] 2024-12-18T03:36:18.1937694Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T03:36:18.1938577Z + jq --raw-output .SecretString 2024-12-18T03:36:18.1938990Z + docker login --username pytorchbot --password-stdin 2024-12-18T03:36:18.1939451Z + jq -r .docker_hub_readonly_token 2024-12-18T03:36:18.8912444Z 2024-12-18T03:36:18.8914989Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action 2024-12-18T03:36:18.9445335Z Error: Cannot perform an interactive login from a non TTY device 2024-12-18T03:36:18.9480383Z ##[error]Process completed with exit code 1. 2024-12-18T03:36:18.9545944Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T03:36:18.9546299Z with: 2024-12-18T03:36:18.9546760Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:18.9547338Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:18.9547645Z env: 2024-12-18T03:36:18.9547807Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:18.9548033Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:18.9548435Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:18.9548815Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:18.9549022Z AWS_REGION: us-east-1 2024-12-18T03:36:18.9549281Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:18.9549553Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:18.9553557Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:18.9553767Z ##[endgroup] 2024-12-18T03:36:18.9566116Z ##[group]Run set -x 2024-12-18T03:36:18.9566316Z set -x 2024-12-18T03:36:18.9566479Z set +e 2024-12-18T03:36:18.9566638Z  2024-12-18T03:36:18.9566791Z login() { 2024-12-18T03:36:18.9567136Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T03:36:18.9567715Z } 2024-12-18T03:36:18.9567865Z  2024-12-18T03:36:18.9568017Z retry () { 2024-12-18T03:36:18.9568214Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T03:36:18.9568437Z } 2024-12-18T03:36:18.9568586Z  2024-12-18T03:36:18.9568753Z retry login "${DOCKER_REGISTRY}" 2024-12-18T03:36:18.9568972Z  2024-12-18T03:36:18.9569127Z set -e 2024-12-18T03:36:18.9569369Z # ignore output since only exit code is used for conditional 2024-12-18T03:36:18.9569719Z # only pull docker image if it's not available locally 2024-12-18T03:36:18.9570096Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T03:36:18.9570441Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T03:36:18.9570689Z fi 2024-12-18T03:36:18.9591441Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:36:18.9591722Z env: 2024-12-18T03:36:18.9591893Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:36:18.9592118Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:36:18.9592528Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:36:18.9592902Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:36:18.9593104Z AWS_REGION: us-east-1 2024-12-18T03:36:18.9593337Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:36:18.9593623Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:36:18.9597596Z AWS_SESSION_TOKEN: *** 2024-12-18T03:36:18.9598068Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:18.9598632Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:18.9598910Z ##[endgroup] 2024-12-18T03:36:18.9643973Z + set +e 2024-12-18T03:36:18.9644741Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:18.9645665Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:18.9650158Z + aws ecr get-login-password --region us-east-1 2024-12-18T03:36:18.9652834Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:36:20.3683208Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-12-18T03:36:20.3684322Z Configure a credential helper to remove this warning. See 2024-12-18T03:36:20.3685328Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-12-18T03:36:20.3686012Z 2024-12-18T03:36:20.3687011Z Login Succeeded 2024-12-18T03:36:20.3716931Z + set -e 2024-12-18T03:36:20.3719076Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:20.3877656Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:20.3879891Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:36:21.5138679Z 45e1356b47a284893081276eff3000b7b534f3b1: Pulling from pytorch/pytorch-linux-focal-rocm-n-py3 2024-12-18T03:36:21.5146034Z 86e5016c2693: Already exists 2024-12-18T03:36:21.5173099Z 7e5cdcc8d39c: Pulling fs layer 2024-12-18T03:36:21.5173463Z 231ab5ed5e97: Pulling fs layer 2024-12-18T03:36:21.5173777Z fcbefe9ad79e: Pulling fs layer 2024-12-18T03:36:21.5174090Z af67ad6d287e: Pulling fs layer 2024-12-18T03:36:21.5174348Z a87dd7b9e020: Pulling fs layer 2024-12-18T03:36:21.5174622Z aa67f5368a1d: Pulling fs layer 2024-12-18T03:36:21.5174884Z e71aede6d460: Pulling fs layer 2024-12-18T03:36:21.5175157Z ee65ff0efe5e: Pulling fs layer 2024-12-18T03:36:21.5175422Z 82c83b062d94: Pulling fs layer 2024-12-18T03:36:21.5175663Z 8362d69f0776: Pulling fs layer 2024-12-18T03:36:21.5175904Z 0d29ca60b095: Pulling fs layer 2024-12-18T03:36:21.5176150Z 6697412e0463: Pulling fs layer 2024-12-18T03:36:21.5176868Z 3fe85e92871b: Pulling fs layer 2024-12-18T03:36:21.5177349Z db8079b9a247: Pulling fs layer 2024-12-18T03:36:21.5177876Z 1e1cc236bfa6: Pulling fs layer 2024-12-18T03:36:21.5178333Z 5594c816d49f: Pulling fs layer 2024-12-18T03:36:21.5178789Z af67ad6d287e: Waiting 2024-12-18T03:36:21.5179235Z d0419af25312: Pulling fs layer 2024-12-18T03:36:21.5179701Z 4a7e3c6b1985: Pulling fs layer 2024-12-18T03:36:21.5180156Z e7e49899a150: Pulling fs layer 2024-12-18T03:36:21.5180619Z 907b5d385c14: Pulling fs layer 2024-12-18T03:36:21.5181072Z ee65ff0efe5e: Waiting 2024-12-18T03:36:21.5181510Z a87dd7b9e020: Waiting 2024-12-18T03:36:21.5181932Z e09946f27e30: Pulling fs layer 2024-12-18T03:36:21.5182388Z 7294a9f24454: Pulling fs layer 2024-12-18T03:36:21.5182863Z 5cbdd1d7b36d: Pulling fs layer 2024-12-18T03:36:21.5183326Z f528471f2fca: Pulling fs layer 2024-12-18T03:36:21.5183780Z 82c83b062d94: Waiting 2024-12-18T03:36:21.5184177Z aa67f5368a1d: Waiting 2024-12-18T03:36:21.5184586Z e1276511038b: Pulling fs layer 2024-12-18T03:36:21.5185038Z 3fe85e92871b: Waiting 2024-12-18T03:36:21.5185450Z f3d817e0be62: Pulling fs layer 2024-12-18T03:36:21.5185892Z e7e49899a150: Waiting 2024-12-18T03:36:21.5186299Z babfeddb4298: Pulling fs layer 2024-12-18T03:36:21.5186766Z e00cbec1ecdd: Pulling fs layer 2024-12-18T03:36:21.5187214Z 8362d69f0776: Waiting 2024-12-18T03:36:21.5187666Z 907b5d385c14: Waiting 2024-12-18T03:36:21.5188066Z 60175e320c62: Pulling fs layer 2024-12-18T03:36:21.5188503Z e09946f27e30: Waiting 2024-12-18T03:36:21.5188892Z 0d29ca60b095: Waiting 2024-12-18T03:36:21.5189306Z e4fd8cd3183c: Pulling fs layer 2024-12-18T03:36:21.5189777Z f79cba221c46: Pulling fs layer 2024-12-18T03:36:21.5190215Z 6697412e0463: Waiting 2024-12-18T03:36:21.5190616Z e425c457ab3e: Pulling fs layer 2024-12-18T03:36:21.5191085Z 6d2cbef9a0f5: Pulling fs layer 2024-12-18T03:36:21.5191542Z 2c4472ad27f9: Pulling fs layer 2024-12-18T03:36:21.5192000Z 48ffbd6d2572: Pulling fs layer 2024-12-18T03:36:21.5192462Z cf56c8fabbc0: Pulling fs layer 2024-12-18T03:36:21.5192941Z b1c0b1889186: Pulling fs layer 2024-12-18T03:36:21.5193395Z 3021fd4a208a: Pulling fs layer 2024-12-18T03:36:21.5193860Z a6f5eec4a2df: Pulling fs layer 2024-12-18T03:36:21.5194326Z 89e8bdba21ff: Pulling fs layer 2024-12-18T03:36:21.5194794Z 7239a56eb334: Pulling fs layer 2024-12-18T03:36:21.5195250Z 22093b6a474a: Pulling fs layer 2024-12-18T03:36:21.5195687Z e71aede6d460: Waiting 2024-12-18T03:36:21.5196095Z 48aa755f3a17: Pulling fs layer 2024-12-18T03:36:21.5196556Z 789ba2fe5a51: Pulling fs layer 2024-12-18T03:36:21.5196997Z 7294a9f24454: Waiting 2024-12-18T03:36:21.5197744Z e2e30e00626e: Pulling fs layer 2024-12-18T03:36:21.5198225Z db8079b9a247: Waiting 2024-12-18T03:36:21.5198643Z c631a2525143: Pulling fs layer 2024-12-18T03:36:21.5199086Z 1e1cc236bfa6: Waiting 2024-12-18T03:36:21.5199860Z 08d3deeeb0fb: Pulling fs layer 2024-12-18T03:36:21.5200325Z 5cbdd1d7b36d: Waiting 2024-12-18T03:36:21.5200754Z f098b0de0dbe: Pulling fs layer 2024-12-18T03:36:21.5201202Z d0419af25312: Waiting 2024-12-18T03:36:21.5201629Z 4e73d9ed9f95: Pulling fs layer 2024-12-18T03:36:21.5202077Z 5594c816d49f: Waiting 2024-12-18T03:36:21.5202481Z f3d817e0be62: Waiting 2024-12-18T03:36:21.5202891Z f528471f2fca: Waiting 2024-12-18T03:36:21.5203302Z babfeddb4298: Waiting 2024-12-18T03:36:21.5203697Z 789ba2fe5a51: Waiting 2024-12-18T03:36:21.5204083Z e1276511038b: Waiting 2024-12-18T03:36:21.5204479Z 3021fd4a208a: Waiting 2024-12-18T03:36:21.5204857Z 7239a56eb334: Waiting 2024-12-18T03:36:21.5205161Z a6f5eec4a2df: Waiting 2024-12-18T03:36:21.5205387Z e00cbec1ecdd: Waiting 2024-12-18T03:36:21.5205600Z 60175e320c62: Waiting 2024-12-18T03:36:21.5205819Z 2c4472ad27f9: Waiting 2024-12-18T03:36:21.5206031Z 89e8bdba21ff: Waiting 2024-12-18T03:36:21.5206246Z e4fd8cd3183c: Waiting 2024-12-18T03:36:21.5206460Z e2e30e00626e: Waiting 2024-12-18T03:36:21.5206662Z c631a2525143: Waiting 2024-12-18T03:36:21.5206864Z 6d2cbef9a0f5: Waiting 2024-12-18T03:36:21.5207073Z f79cba221c46: Waiting 2024-12-18T03:36:21.5207281Z 08d3deeeb0fb: Waiting 2024-12-18T03:36:21.5207677Z e425c457ab3e: Waiting 2024-12-18T03:36:21.5207888Z f098b0de0dbe: Waiting 2024-12-18T03:36:21.5208097Z 4a7e3c6b1985: Waiting 2024-12-18T03:36:21.5208308Z 22093b6a474a: Waiting 2024-12-18T03:36:21.5208527Z 48aa755f3a17: Waiting 2024-12-18T03:36:21.5208742Z 48ffbd6d2572: Waiting 2024-12-18T03:36:21.5208956Z 4e73d9ed9f95: Waiting 2024-12-18T03:36:21.5209161Z b1c0b1889186: Waiting 2024-12-18T03:36:22.1112069Z fcbefe9ad79e: Download complete 2024-12-18T03:36:22.1339262Z 7e5cdcc8d39c: Download complete 2024-12-18T03:36:22.2013085Z 7e5cdcc8d39c: Pull complete 2024-12-18T03:36:22.7514458Z a87dd7b9e020: Verifying Checksum 2024-12-18T03:36:22.7515086Z a87dd7b9e020: Download complete 2024-12-18T03:36:22.7650096Z af67ad6d287e: Verifying Checksum 2024-12-18T03:36:22.7650730Z af67ad6d287e: Download complete 2024-12-18T03:36:23.3335136Z aa67f5368a1d: Download complete 2024-12-18T03:36:23.3567466Z e71aede6d460: Verifying Checksum 2024-12-18T03:36:23.3568085Z e71aede6d460: Download complete 2024-12-18T03:36:23.9446099Z ee65ff0efe5e: Verifying Checksum 2024-12-18T03:36:23.9446729Z ee65ff0efe5e: Download complete 2024-12-18T03:36:23.9519353Z 82c83b062d94: Download complete 2024-12-18T03:36:24.5532949Z 0d29ca60b095: Verifying Checksum 2024-12-18T03:36:24.5533605Z 0d29ca60b095: Download complete 2024-12-18T03:36:25.1627728Z 6697412e0463: Verifying Checksum 2024-12-18T03:36:25.1628343Z 6697412e0463: Download complete 2024-12-18T03:36:25.7699252Z 3fe85e92871b: Verifying Checksum 2024-12-18T03:36:25.7699909Z 3fe85e92871b: Download complete 2024-12-18T03:36:41.7057407Z db8079b9a247: Verifying Checksum 2024-12-18T03:36:41.7061740Z db8079b9a247: Download complete 2024-12-18T03:36:42.3827868Z 1e1cc236bfa6: Verifying Checksum 2024-12-18T03:36:42.3828502Z 1e1cc236bfa6: Download complete 2024-12-18T03:36:42.9758073Z 5594c816d49f: Verifying Checksum 2024-12-18T03:36:42.9758682Z 5594c816d49f: Download complete 2024-12-18T03:36:43.6145683Z d0419af25312: Verifying Checksum 2024-12-18T03:36:43.6146349Z d0419af25312: Download complete 2024-12-18T03:36:44.2349004Z 4a7e3c6b1985: Download complete 2024-12-18T03:36:44.8244542Z e7e49899a150: Download complete 2024-12-18T03:37:09.6094343Z 231ab5ed5e97: Verifying Checksum 2024-12-18T03:37:09.6098372Z 231ab5ed5e97: Download complete 2024-12-18T03:37:10.2424373Z e09946f27e30: Verifying Checksum 2024-12-18T03:37:10.2425165Z e09946f27e30: Download complete 2024-12-18T03:37:10.8519630Z 7294a9f24454: Verifying Checksum 2024-12-18T03:37:10.8520260Z 7294a9f24454: Download complete 2024-12-18T03:37:25.2377418Z 907b5d385c14: Verifying Checksum 2024-12-18T03:37:25.2379169Z 907b5d385c14: Download complete 2024-12-18T03:37:25.8817371Z f528471f2fca: Verifying Checksum 2024-12-18T03:37:25.8817983Z f528471f2fca: Download complete 2024-12-18T03:37:26.0174091Z 231ab5ed5e97: Pull complete 2024-12-18T03:37:26.0325436Z fcbefe9ad79e: Pull complete 2024-12-18T03:37:26.0446274Z af67ad6d287e: Pull complete 2024-12-18T03:37:26.0541607Z a87dd7b9e020: Pull complete 2024-12-18T03:37:26.0712779Z aa67f5368a1d: Pull complete 2024-12-18T03:37:26.0806078Z e71aede6d460: Pull complete 2024-12-18T03:37:26.0903443Z ee65ff0efe5e: Pull complete 2024-12-18T03:37:26.1007419Z 82c83b062d94: Pull complete 2024-12-18T03:37:26.5036875Z e1276511038b: Download complete 2024-12-18T03:38:03.3840533Z f3d817e0be62: Verifying Checksum 2024-12-18T03:38:03.3841184Z f3d817e0be62: Download complete 2024-12-18T03:38:04.0629697Z babfeddb4298: Verifying Checksum 2024-12-18T03:38:04.6369029Z e00cbec1ecdd: Download complete 2024-12-18T03:38:05.3109549Z 60175e320c62: Download complete 2024-12-18T03:38:05.9554123Z e4fd8cd3183c: Verifying Checksum 2024-12-18T03:38:05.9554782Z e4fd8cd3183c: Download complete 2024-12-18T03:38:08.0067322Z f79cba221c46: Verifying Checksum 2024-12-18T03:38:08.0068028Z f79cba221c46: Download complete 2024-12-18T03:38:08.6068989Z e425c457ab3e: Verifying Checksum 2024-12-18T03:38:08.6069623Z e425c457ab3e: Download complete 2024-12-18T03:38:09.2313248Z 6d2cbef9a0f5: Verifying Checksum 2024-12-18T03:38:09.2314557Z 6d2cbef9a0f5: Download complete 2024-12-18T03:38:37.6834759Z 2c4472ad27f9: Verifying Checksum 2024-12-18T03:38:37.6835435Z 2c4472ad27f9: Download complete 2024-12-18T03:38:38.2765998Z 48ffbd6d2572: Verifying Checksum 2024-12-18T03:38:38.2766643Z 48ffbd6d2572: Download complete 2024-12-18T03:38:38.8830145Z cf56c8fabbc0: Verifying Checksum 2024-12-18T03:38:38.8830772Z cf56c8fabbc0: Download complete 2024-12-18T03:38:39.6784787Z b1c0b1889186: Download complete 2024-12-18T03:38:40.3049068Z 3021fd4a208a: Verifying Checksum 2024-12-18T03:38:40.3049710Z 3021fd4a208a: Download complete 2024-12-18T03:38:40.9216983Z a6f5eec4a2df: Verifying Checksum 2024-12-18T03:38:40.9217638Z a6f5eec4a2df: Download complete 2024-12-18T03:38:41.5284161Z 89e8bdba21ff: Download complete 2024-12-18T03:38:42.1751743Z 7239a56eb334: Verifying Checksum 2024-12-18T03:38:42.1752401Z 7239a56eb334: Download complete 2024-12-18T03:39:54.0139805Z 8362d69f0776: Verifying Checksum 2024-12-18T03:39:54.0145382Z 8362d69f0776: Download complete 2024-12-18T03:39:54.6413024Z 48aa755f3a17: Verifying Checksum 2024-12-18T03:39:54.6413645Z 48aa755f3a17: Download complete 2024-12-18T03:39:55.2158003Z 789ba2fe5a51: Download complete 2024-12-18T03:39:55.8334789Z e2e30e00626e: Verifying Checksum 2024-12-18T03:39:55.8335407Z e2e30e00626e: Download complete 2024-12-18T03:40:40.7818234Z c631a2525143: Verifying Checksum 2024-12-18T03:40:40.7821716Z c631a2525143: Download complete 2024-12-18T03:40:41.4399891Z 08d3deeeb0fb: Download complete 2024-12-18T03:40:48.9863399Z f098b0de0dbe: Verifying Checksum 2024-12-18T03:40:48.9864154Z f098b0de0dbe: Download complete 2024-12-18T03:41:03.1085491Z 4e73d9ed9f95: Verifying Checksum 2024-12-18T03:41:03.1086243Z 4e73d9ed9f95: Download complete 2024-12-18T03:41:22.2666777Z 8362d69f0776: Pull complete 2024-12-18T03:41:22.2950447Z 0d29ca60b095: Pull complete 2024-12-18T03:41:22.3193549Z 6697412e0463: Pull complete 2024-12-18T03:41:22.3311677Z 3fe85e92871b: Pull complete 2024-12-18T03:41:28.6146659Z db8079b9a247: Pull complete 2024-12-18T03:41:28.6317912Z 1e1cc236bfa6: Pull complete 2024-12-18T03:41:28.6427782Z 5594c816d49f: Pull complete 2024-12-18T03:41:28.6584752Z d0419af25312: Pull complete 2024-12-18T03:41:28.6693036Z 4a7e3c6b1985: Pull complete 2024-12-18T03:41:28.6821189Z e7e49899a150: Pull complete 2024-12-18T03:41:42.9806448Z 907b5d385c14: Pull complete 2024-12-18T03:41:43.0186701Z e09946f27e30: Pull complete 2024-12-18T03:41:43.0312745Z 7294a9f24454: Pull complete 2024-12-18T03:44:48.5899819Z 22093b6a474a: Verifying Checksum 2024-12-18T03:44:48.5900565Z 22093b6a474a: Download complete 2024-12-18T03:45:50.4879526Z 5cbdd1d7b36d: Verifying Checksum 2024-12-18T03:45:50.4884963Z 5cbdd1d7b36d: Download complete 2024-12-18T03:50:44.4330706Z 5cbdd1d7b36d: Pull complete 2024-12-18T03:50:44.4442395Z f528471f2fca: Pull complete 2024-12-18T03:50:44.4556718Z e1276511038b: Pull complete 2024-12-18T03:50:59.2142767Z f3d817e0be62: Pull complete 2024-12-18T03:50:59.2260140Z babfeddb4298: Pull complete 2024-12-18T03:50:59.2379454Z e00cbec1ecdd: Pull complete 2024-12-18T03:50:59.2484882Z 60175e320c62: Pull complete 2024-12-18T03:50:59.2591516Z e4fd8cd3183c: Pull complete 2024-12-18T03:50:59.3442359Z f79cba221c46: Pull complete 2024-12-18T03:50:59.3909383Z e425c457ab3e: Pull complete 2024-12-18T03:50:59.4016171Z 6d2cbef9a0f5: Pull complete 2024-12-18T03:51:02.6721858Z 2c4472ad27f9: Pull complete 2024-12-18T03:51:02.6826829Z 48ffbd6d2572: Pull complete 2024-12-18T03:51:02.6926995Z cf56c8fabbc0: Pull complete 2024-12-18T03:51:02.7167685Z b1c0b1889186: Pull complete 2024-12-18T03:51:02.7278647Z 3021fd4a208a: Pull complete 2024-12-18T03:51:02.7396010Z a6f5eec4a2df: Pull complete 2024-12-18T03:51:02.7613532Z 89e8bdba21ff: Pull complete 2024-12-18T03:51:02.7754050Z 7239a56eb334: Pull complete 2024-12-18T03:53:19.9595734Z 22093b6a474a: Pull complete 2024-12-18T03:53:27.3046756Z 48aa755f3a17: Pull complete 2024-12-18T03:53:27.3163340Z 789ba2fe5a51: Pull complete 2024-12-18T03:53:27.3381429Z e2e30e00626e: Pull complete 2024-12-18T03:53:31.6193346Z c631a2525143: Pull complete 2024-12-18T03:53:31.6322425Z 08d3deeeb0fb: Pull complete 2024-12-18T03:53:32.2857637Z f098b0de0dbe: Pull complete 2024-12-18T03:53:35.4870171Z 4e73d9ed9f95: Pull complete 2024-12-18T03:53:35.4912222Z Digest: sha256:e9226d891a9f26f20ac70c9a92beae93e0afb8d4f29d55f1f60bb9f6365210dc 2024-12-18T03:53:35.4926739Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:53:35.4945701Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:53:35.5057620Z Prepare all required actions 2024-12-18T03:53:35.5058343Z Getting action download info 2024-12-18T03:53:36.0409045Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T03:53:36.6929826Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T03:53:37.3601243Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T03:53:37.3601527Z with: 2024-12-18T03:53:37.3601709Z name: linux-focal-rocm6.2-py3.10 2024-12-18T03:53:37.3601936Z s3-bucket: gha-artifacts 2024-12-18T03:53:37.3602130Z env: 2024-12-18T03:53:37.3602290Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:53:37.3602512Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:53:37.3602911Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:53:37.3603288Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:53:37.3603531Z AWS_REGION: us-east-1 2024-12-18T03:53:37.3603819Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:53:37.3604098Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:53:37.3608215Z AWS_SESSION_TOKEN: *** 2024-12-18T03:53:37.3608447Z ##[endgroup] 2024-12-18T03:53:37.3634824Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T03:53:37.3635085Z with: 2024-12-18T03:53:37.3635260Z name: linux-focal-rocm6.2-py3.10 2024-12-18T03:53:37.3635507Z s3-bucket: gha-artifacts 2024-12-18T03:53:37.3635697Z region: us-east-1 2024-12-18T03:53:37.3635860Z env: 2024-12-18T03:53:37.3636010Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:53:37.3636224Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:53:37.3636617Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:53:37.3636987Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:53:37.3637194Z AWS_REGION: us-east-1 2024-12-18T03:53:37.3637447Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:53:37.3637729Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:53:37.3642095Z AWS_SESSION_TOKEN: *** 2024-12-18T03:53:37.3642314Z ##[endgroup] 2024-12-18T03:53:37.7701655Z (node:2032374) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T03:53:37.7702552Z 2024-12-18T03:53:37.7702912Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T03:53:37.7703932Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T03:53:37.7704928Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T03:53:38.0494683Z Found 1 objects with prefix pytorch/pytorch/12383255596/linux-focal-rocm6.2-py3.10/ 2024-12-18T03:53:38.0495993Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T03:54:23.0924980Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T03:54:23.0943616Z Artifact download has finished successfully 2024-12-18T03:54:23.1465898Z ##[group]Run unzip -o artifacts.zip 2024-12-18T03:54:23.1466229Z unzip -o artifacts.zip 2024-12-18T03:54:23.1499578Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:23.1499926Z env: 2024-12-18T03:54:23.1500129Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:23.1500406Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:23.1501161Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:23.1501630Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:23.1501888Z AWS_REGION: us-east-1 2024-12-18T03:54:23.1502256Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:23.1502681Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:23.1510185Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:23.1510532Z ##[endgroup] 2024-12-18T03:54:23.1590674Z Archive: artifacts.zip 2024-12-18T03:54:23.1592779Z creating: dist/ 2024-12-18T03:54:25.7444154Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T03:54:25.7567086Z inflating: dist/.ninja_log 2024-12-18T03:54:25.7570816Z creating: build/custom_test_artifacts/ 2024-12-18T03:54:25.7571647Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T03:54:25.7572571Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T03:54:25.7573628Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:54:25.7574898Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:54:25.7576196Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T03:54:25.7577517Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:54:25.7578912Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:54:25.7580293Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:54:25.7581905Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:54:25.7582728Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:54:25.7583336Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:54:25.7583946Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:54:25.7584652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:54:25.7585384Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:54:25.7586084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:54:25.7586792Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:54:25.7587515Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:54:25.7588244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:54:25.7588853Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:54:25.7589390Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:54:25.7589950Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T03:54:25.7590537Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T03:54:25.7591175Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T03:54:25.7591902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T03:54:25.7592597Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T03:54:25.7593243Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T03:54:25.7593903Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T03:54:25.7594573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T03:54:25.7596638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T03:54:25.7597317Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T03:54:25.7597981Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T03:54:25.7611649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T03:54:25.7743415Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T03:54:25.7744798Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T03:54:25.7746206Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T03:54:25.7747773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T03:54:25.7749297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T03:54:25.7750658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T03:54:25.7752035Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T03:54:25.7753422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T03:54:25.7754821Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T03:54:25.7756362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T03:54:25.7757978Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T03:54:25.7767392Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T03:54:25.7843362Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T03:54:25.7844992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:54:25.7846559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:54:25.7848041Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T03:54:25.7849376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T03:54:25.7850674Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T03:54:25.7851817Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-12-18T03:54:25.7852398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T03:54:25.7852885Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T03:54:25.7853377Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T03:54:25.7956903Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T03:54:25.8012994Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T03:54:25.8014034Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T03:54:25.8014910Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T03:54:25.8015905Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:54:25.8017114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:54:25.8018275Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T03:54:25.8019366Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:54:25.8020998Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:54:25.8022135Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:54:25.8023479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:54:25.8024811Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:54:25.8026288Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:54:25.8027406Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:54:25.8028691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:54:25.8029979Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:54:25.8031250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:54:25.8032520Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:54:25.8033804Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:54:25.8035128Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:54:25.8036244Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:54:25.8037197Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:54:25.8038190Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T03:54:25.8039237Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T03:54:25.8040600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T03:54:25.8041932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T03:54:25.8043250Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T03:54:25.8044444Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T03:54:25.8045697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T03:54:25.8046962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T03:54:25.8048208Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T03:54:25.8049446Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T03:54:25.8050671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T03:54:25.8055199Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T03:54:25.8112876Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T03:54:25.8114448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:54:25.8115856Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:54:25.8117092Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T03:54:25.8118239Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T03:54:25.8119325Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T03:54:25.8121087Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-12-18T03:54:25.8122056Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T03:54:25.8123013Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T03:54:25.8123964Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T03:54:25.8162752Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T03:54:25.8163822Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T03:54:25.8165231Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T03:54:25.8166374Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:54:25.8167681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:54:25.8168873Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T03:54:25.8170078Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:54:25.8171383Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:54:25.8172689Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:54:25.8174099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:54:25.8175557Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:54:25.8176852Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:54:25.8178157Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:54:25.8179638Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:54:25.8181143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:54:25.8182598Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:54:25.8184057Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:54:25.8185534Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:54:25.8187032Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:54:25.8188371Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:54:25.8189480Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:54:25.8190632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T03:54:25.8191854Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T03:54:25.8193230Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T03:54:25.8194788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T03:54:25.8196267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T03:54:25.8197658Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T03:54:25.8199108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T03:54:25.8200780Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T03:54:25.8202634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T03:54:25.8204086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T03:54:25.8205507Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T03:54:25.8207052Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T03:54:25.8305200Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T03:54:25.8306694Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T03:54:25.8308159Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T03:54:25.8309791Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T03:54:25.8311362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T03:54:25.8312828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T03:54:25.8314349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T03:54:25.8315870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T03:54:25.8317385Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T03:54:25.8318908Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T03:54:25.8320505Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T03:54:25.8329830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T03:54:25.8380686Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T03:54:25.8382289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:54:25.8383723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:54:25.8384988Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T03:54:25.8386147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T03:54:25.8387295Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T03:54:25.8388413Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-12-18T03:54:25.8389455Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T03:54:25.8390443Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T03:54:25.8391458Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T03:54:25.8477104Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T03:54:25.8515626Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T03:54:25.8516453Z creating: build/lib/ 2024-12-18T03:54:25.8596396Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T03:54:25.8604922Z inflating: build/lib/libpthreadpool.a 2024-12-18T03:54:25.8612230Z inflating: build/lib/libcpuinfo.a 2024-12-18T03:54:25.8619306Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T03:54:25.9033728Z inflating: build/lib/libprotobuf.a 2024-12-18T03:54:25.9489731Z inflating: build/lib/libprotoc.a 2024-12-18T03:54:25.9490402Z inflating: build/lib/libclog.a 2024-12-18T03:54:25.9507829Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T03:54:25.9509245Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T03:54:25.9526096Z inflating: build/lib/libnnpack.a 2024-12-18T03:54:25.9694086Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T03:54:26.0466809Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T03:54:26.0527843Z inflating: build/lib/libgtest.a 2024-12-18T03:54:26.0544737Z inflating: build/lib/libgmock.a 2024-12-18T03:54:26.0545624Z inflating: build/lib/libgmock_main.a 2024-12-18T03:54:26.0546246Z inflating: build/lib/libgtest_main.a 2024-12-18T03:54:26.0611183Z inflating: build/lib/libbenchmark.a 2024-12-18T03:54:26.0611909Z inflating: build/lib/libbenchmark_main.a 2024-12-18T03:54:26.0690772Z inflating: build/lib/libXNNPACK.a 2024-12-18T03:54:26.0697540Z inflating: build/lib/libittnotify.a 2024-12-18T03:54:26.0753808Z inflating: build/lib/libasmjit.a 2024-12-18T03:54:26.1891322Z inflating: build/lib/libfbgemm.a 2024-12-18T03:54:26.1915820Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T03:54:26.2432302Z inflating: build/lib/libtensorpipe.a 2024-12-18T03:54:26.2470194Z inflating: build/lib/libonnx_proto.a 2024-12-18T03:54:26.2570629Z inflating: build/lib/libgloo.a 2024-12-18T03:54:26.2922397Z inflating: build/lib/libgloo_hip.a 2024-12-18T03:54:26.3578239Z inflating: build/lib/libonnx.a 2024-12-18T03:54:26.3596410Z inflating: build/lib/libfmt.a 2024-12-18T03:54:26.3886336Z inflating: build/lib/libkineto.a 2024-12-18T03:54:27.3160823Z inflating: build/lib/libdnnl.a 2024-12-18T03:54:27.3248001Z inflating: build/lib/libc10.so 2024-12-18T03:54:27.3248726Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T03:54:27.3291340Z inflating: build/lib/libc10_hip.so 2024-12-18T03:54:27.3292075Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T03:54:29.5613917Z inflating: build/lib/libtorch_cpu.so 2024-12-18T03:54:29.5617860Z inflating: build/lib/libunbox_lib.a 2024-12-18T03:54:29.5621439Z inflating: build/lib/libshm.so 2024-12-18T03:54:30.7329205Z inflating: build/lib/libtorch_hip.so 2024-12-18T03:54:30.7329918Z inflating: build/lib/libtorch.so 2024-12-18T03:54:30.7347861Z inflating: build/lib/libjitbackend_test.so 2024-12-18T03:54:30.7369567Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T03:54:30.7431331Z inflating: build/lib/libtorchbind_test.so 2024-12-18T03:54:30.7453502Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T03:54:30.9281349Z inflating: build/lib/libtorch_python.so 2024-12-18T03:54:30.9311373Z inflating: build/lib/libnnapi_backend.so 2024-12-18T03:54:30.9312060Z creating: build/bin/ 2024-12-18T03:54:30.9312578Z creating: build/bin/CMakeFiles/ 2024-12-18T03:54:30.9313168Z inflating: build/bin/cmake_install.cmake 2024-12-18T03:54:30.9313861Z inflating: build/bin/CTestTestfile.cmake 2024-12-18T03:54:30.9705619Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T03:54:31.0097401Z inflating: build/bin/protoc 2024-12-18T03:54:31.0144410Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T03:54:31.0192424Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T03:54:31.0240789Z inflating: build/bin/c10_Device_test 2024-12-18T03:54:31.0295902Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T03:54:31.0345694Z inflating: build/bin/c10_Scalar_test 2024-12-18T03:54:31.0391208Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T03:54:31.0438797Z inflating: build/bin/c10_SymInt_test 2024-12-18T03:54:31.0489218Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T03:54:31.0541099Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T03:54:31.0593103Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T03:54:31.0638880Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T03:54:31.0685303Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T03:54:31.0734446Z inflating: build/bin/c10_Bitset_test 2024-12-18T03:54:31.0799685Z inflating: build/bin/c10_cow_test 2024-12-18T03:54:31.0846147Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T03:54:31.0898582Z inflating: build/bin/c10_LeftRight_test 2024-12-18T03:54:31.0945825Z inflating: build/bin/c10_Half_test 2024-12-18T03:54:31.0997148Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T03:54:31.1047030Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T03:54:31.1093271Z inflating: build/bin/c10_Synchronized_test 2024-12-18T03:54:31.1145003Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T03:54:31.1192925Z inflating: build/bin/c10_TypeList_test 2024-12-18T03:54:31.1241309Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T03:54:31.1286940Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T03:54:31.1335374Z inflating: build/bin/c10_accumulate_test 2024-12-18T03:54:31.1387382Z inflating: build/bin/c10_bfloat16_test 2024-12-18T03:54:31.1434530Z inflating: build/bin/c10_bit_cast_test 2024-12-18T03:54:31.1481248Z inflating: build/bin/c10_error_test 2024-12-18T03:54:31.1534075Z inflating: build/bin/c10_complex_math_test 2024-12-18T03:54:31.1585364Z inflating: build/bin/c10_complex_test 2024-12-18T03:54:31.1634554Z inflating: build/bin/c10_exception_test 2024-12-18T03:54:31.1681882Z inflating: build/bin/c10_flags_test 2024-12-18T03:54:31.1729002Z inflating: build/bin/c10_generic_math_test 2024-12-18T03:54:31.1778894Z inflating: build/bin/c10_lazy_test 2024-12-18T03:54:31.1826546Z inflating: build/bin/c10_irange_test 2024-12-18T03:54:31.1879818Z inflating: build/bin/c10_logging_test 2024-12-18T03:54:31.2031772Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T03:54:31.2102112Z inflating: build/bin/c10_optional_test 2024-12-18T03:54:31.2160320Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T03:54:31.2209977Z inflating: build/bin/c10_registry_test 2024-12-18T03:54:31.2258287Z inflating: build/bin/c10_ssize_test 2024-12-18T03:54:31.2400340Z inflating: build/bin/c10_small_vector_test 2024-12-18T03:54:31.2448614Z inflating: build/bin/c10_string_util_test 2024-12-18T03:54:31.2495745Z inflating: build/bin/c10_tempfile_test 2024-12-18T03:54:31.2551269Z inflating: build/bin/c10_string_view_test 2024-12-18T03:54:31.2593868Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T03:54:31.2646345Z inflating: build/bin/c10_typeid_test 2024-12-18T03:54:31.2692083Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-12-18T03:54:31.2737709Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-12-18T03:54:31.2783490Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-12-18T03:54:31.2828895Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-12-18T03:54:31.2874650Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T03:54:31.2920335Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T03:54:31.2965803Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-12-18T03:54:31.3011271Z inflating: build/bin/c10_hip_HIPTest 2024-12-18T03:54:31.3367390Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T03:54:31.3732353Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T03:54:31.4110282Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T03:54:31.4160313Z inflating: build/bin/test_edge_op_registration 2024-12-18T03:54:31.4209526Z inflating: build/bin/BackoffTest 2024-12-18T03:54:31.4258822Z inflating: build/bin/FileStoreTest 2024-12-18T03:54:31.4310240Z inflating: build/bin/TCPStoreTest 2024-12-18T03:54:31.4359873Z inflating: build/bin/HashStoreTest 2024-12-18T03:54:31.4420616Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T03:54:31.4422265Z inflating: build/bin/example_allreduce 2024-12-18T03:54:31.4475045Z inflating: build/bin/static_runtime_bench 2024-12-18T03:54:31.4703239Z inflating: build/bin/static_runtime_test 2024-12-18T03:54:31.4772149Z inflating: build/bin/Dict_test 2024-12-18T03:54:31.4817802Z inflating: build/bin/hip_half_test 2024-12-18T03:54:31.4866241Z inflating: build/bin/Dimname_test 2024-12-18T03:54:31.4926432Z inflating: build/bin/MaybeOwned_test 2024-12-18T03:54:31.4980481Z inflating: build/bin/NamedTensor_test 2024-12-18T03:54:31.5034734Z inflating: build/bin/apply_utils_test 2024-12-18T03:54:31.5089830Z inflating: build/bin/atest 2024-12-18T03:54:31.5148184Z inflating: build/bin/basic 2024-12-18T03:54:31.5199361Z inflating: build/bin/broadcast_test 2024-12-18T03:54:31.5246988Z inflating: build/bin/cpu_allocator_test 2024-12-18T03:54:31.5301635Z inflating: build/bin/cpu_generator_test 2024-12-18T03:54:31.5351809Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T03:54:31.5436557Z inflating: build/bin/cpu_rng_test 2024-12-18T03:54:31.5483514Z inflating: build/bin/dispatch_key_set_test 2024-12-18T03:54:31.5530869Z inflating: build/bin/dlconvertor_test 2024-12-18T03:54:31.5584990Z inflating: build/bin/extension_backend_test 2024-12-18T03:54:31.5636388Z inflating: build/bin/half_test 2024-12-18T03:54:31.5724259Z inflating: build/bin/ivalue_test 2024-12-18T03:54:31.5770853Z inflating: build/bin/lazy_tensor_test 2024-12-18T03:54:31.5821161Z inflating: build/bin/math_kernel_test 2024-12-18T03:54:31.5871387Z inflating: build/bin/memory_format_test 2024-12-18T03:54:31.5920978Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T03:54:31.5970676Z inflating: build/bin/memory_overlapping_test 2024-12-18T03:54:31.6018152Z inflating: build/bin/operator_name_test 2024-12-18T03:54:31.6070469Z inflating: build/bin/native_test 2024-12-18T03:54:31.6118452Z inflating: build/bin/operators_test 2024-12-18T03:54:31.6167138Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T03:54:31.6220422Z inflating: build/bin/quantized_test 2024-12-18T03:54:31.6282513Z inflating: build/bin/pow_test 2024-12-18T03:54:31.6329200Z inflating: build/bin/reduce_ops_test 2024-12-18T03:54:31.6376899Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T03:54:31.6429966Z inflating: build/bin/scalar_tensor_test 2024-12-18T03:54:31.6478140Z inflating: build/bin/StorageUtils_test 2024-12-18T03:54:31.6532937Z inflating: build/bin/scalar_test 2024-12-18T03:54:31.6582229Z inflating: build/bin/stride_properties_test 2024-12-18T03:54:31.6632990Z inflating: build/bin/test_parallel 2024-12-18T03:54:31.6706685Z inflating: build/bin/tensor_iterator_test 2024-12-18T03:54:31.6758448Z inflating: build/bin/type_ptr_test 2024-12-18T03:54:31.6760040Z inflating: build/bin/thread_init_test 2024-12-18T03:54:31.6809937Z inflating: build/bin/undefined_tensor_test 2024-12-18T03:54:31.6865370Z inflating: build/bin/type_test 2024-12-18T03:54:31.6866088Z inflating: build/bin/verify_api_visibility 2024-12-18T03:54:31.6931944Z inflating: build/bin/legacy_vmap_test 2024-12-18T03:54:31.6980422Z inflating: build/bin/weakref_test 2024-12-18T03:54:31.7028648Z inflating: build/bin/wrapdim_test 2024-12-18T03:54:31.7077141Z inflating: build/bin/xla_tensor_test 2024-12-18T03:54:31.7132761Z inflating: build/bin/IListRef_test 2024-12-18T03:54:31.7231786Z inflating: build/bin/List_test 2024-12-18T03:54:31.7342676Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T03:54:31.7431061Z inflating: build/bin/kernel_function_test 2024-12-18T03:54:31.7548113Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T03:54:31.7610067Z inflating: build/bin/KernelFunction_test 2024-12-18T03:54:31.7705765Z inflating: build/bin/kernel_lambda_test 2024-12-18T03:54:31.7762812Z inflating: build/bin/kernel_stackbased_test 2024-12-18T03:54:31.7851086Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T03:54:31.7899013Z inflating: build/bin/CppSignature_test 2024-12-18T03:54:31.7944840Z inflating: build/bin/op_allowlist_test 2024-12-18T03:54:31.8218835Z inflating: build/bin/op_registration_test 2024-12-18T03:54:31.8270754Z inflating: build/bin/backend_fallback_test 2024-12-18T03:54:31.8316528Z inflating: build/bin/hip_complex_math_test 2024-12-18T03:54:31.8375804Z inflating: build/bin/inline_container_test 2024-12-18T03:54:31.8425712Z inflating: build/bin/hip_apply_test 2024-12-18T03:54:31.8471059Z inflating: build/bin/hip_complex_test 2024-12-18T03:54:31.8516770Z inflating: build/bin/hip_distributions_test 2024-12-18T03:54:31.8562586Z inflating: build/bin/hip_generator_test 2024-12-18T03:54:31.8610364Z inflating: build/bin/hip_dlconvertor_test 2024-12-18T03:54:31.8656029Z inflating: build/bin/hip_integer_divider_test 2024-12-18T03:54:31.8702355Z inflating: build/bin/hip_optional_test 2024-12-18T03:54:31.8748193Z inflating: build/bin/hip_packedtensoraccessor_test 2024-12-18T03:54:31.8793864Z inflating: build/bin/hip_vectorized_test 2024-12-18T03:54:31.9309894Z inflating: build/bin/test_jit 2024-12-18T03:54:32.0020359Z inflating: build/bin/test_tensorexpr 2024-12-18T03:54:32.0071636Z inflating: build/bin/test_dist_autograd 2024-12-18T03:54:32.0084391Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T03:54:32.0147947Z inflating: build/bin/test_cpp_rpc 2024-12-18T03:54:32.1195100Z inflating: build/bin/test_api 2024-12-18T03:54:32.1203070Z inflating: build/bin/aot_model_compiler_test 2024-12-18T03:54:32.1204450Z inflating: build/bin/parallel_benchmark 2024-12-18T03:54:32.1267064Z inflating: build/bin/test_mobile_nnc 2024-12-18T03:54:32.1582104Z inflating: build/bin/test_lazy 2024-12-18T03:54:32.1585822Z inflating: build/bin/torch_shm_manager 2024-12-18T03:54:32.1586558Z creating: .additional_ci_files/ 2024-12-18T03:54:32.1666173Z inflating: .additional_ci_files/test-times.json 2024-12-18T03:54:32.1980423Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T03:54:32.2034334Z ##[group]Run rm artifacts.zip 2024-12-18T03:54:32.2034901Z rm artifacts.zip 2024-12-18T03:54:32.2085061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:32.2085783Z env: 2024-12-18T03:54:32.2086244Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:32.2086937Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:32.2088107Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:32.2089182Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:32.2089761Z AWS_REGION: us-east-1 2024-12-18T03:54:32.2090436Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:32.2091218Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:32.2101796Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:32.2102234Z ##[endgroup] 2024-12-18T03:54:32.3621985Z ##[group]Run df -H 2024-12-18T03:54:32.3622464Z df -H 2024-12-18T03:54:32.3674115Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:32.3674806Z env: 2024-12-18T03:54:32.3675203Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:32.3675765Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:32.3676754Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:32.3677657Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:32.3678213Z AWS_REGION: us-east-1 2024-12-18T03:54:32.3678797Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:32.3679582Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:32.3689395Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:32.3689862Z ##[endgroup] 2024-12-18T03:54:32.3792530Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T03:54:32.3793333Z tmpfs 14G 18M 14G 1% /run 2024-12-18T03:54:32.3794548Z /dev/mapper/ubuntu--vg-ubuntu--lv 1.9T 323G 1.5T 19% / 2024-12-18T03:54:32.3795525Z tmpfs 68G 8.2k 68G 1% /dev/shm 2024-12-18T03:54:32.3796367Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-12-18T03:54:32.3797070Z /dev/sda2 2.1G 324M 1.6G 17% /boot 2024-12-18T03:54:32.3797834Z /dev/sda1 1.2G 6.4M 1.2G 1% /boot/efi 2024-12-18T03:54:32.3798550Z tmpfs 14G 17k 14G 1% /run/user/1001 2024-12-18T03:54:32.3799720Z tmpfs 14G 4.1k 14G 1% /run/user/1000 2024-12-18T03:54:32.3863709Z Prepare all required actions 2024-12-18T03:54:32.3864492Z Getting action download info 2024-12-18T03:54:32.6002466Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T03:54:32.6003044Z with: 2024-12-18T03:54:32.6003389Z env: 2024-12-18T03:54:32.6003758Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:32.6004298Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:32.6005202Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:32.6006039Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:32.6006500Z AWS_REGION: us-east-1 2024-12-18T03:54:32.6007010Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:32.6007628Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:32.6016838Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:32.6017264Z ##[endgroup] 2024-12-18T03:54:32.6064738Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T03:54:32.6065287Z with: 2024-12-18T03:54:32.6065635Z name: td_results 2024-12-18T03:54:32.6066072Z s3-bucket: gha-artifacts 2024-12-18T03:54:32.6066509Z region: us-east-1 2024-12-18T03:54:32.6066874Z env: 2024-12-18T03:54:32.6067224Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:32.6067728Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:32.6068613Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:32.6069457Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:32.6069945Z AWS_REGION: us-east-1 2024-12-18T03:54:32.6070447Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:32.6071073Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:32.6080369Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:32.6080786Z ##[endgroup] 2024-12-18T03:54:33.0108423Z (node:2032504) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T03:54:33.0109341Z 2024-12-18T03:54:33.0109685Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T03:54:33.0110681Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T03:54:33.0111699Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T03:54:33.3053234Z Found 1 objects with prefix pytorch/pytorch/12383255596/td_results/ 2024-12-18T03:54:33.3054436Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T03:54:33.7164770Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T03:54:33.7176204Z Artifact download has finished successfully 2024-12-18T03:54:33.7702048Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T03:54:33.7702801Z mkdir -p .additional_ci_files 2024-12-18T03:54:33.7703554Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T03:54:33.7755332Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:33.7756017Z env: 2024-12-18T03:54:33.7756408Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:33.7756991Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:33.7757967Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:33.7758866Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:33.7759363Z AWS_REGION: us-east-1 2024-12-18T03:54:33.7760511Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:33.7761187Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:33.7771664Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:33.7772211Z ##[endgroup] 2024-12-18T03:54:33.7946079Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T03:54:33.7946784Z .github/scripts/parse_ref.py 2024-12-18T03:54:33.7993303Z shell: /usr/bin/bash -e {0} 2024-12-18T03:54:33.7993815Z env: 2024-12-18T03:54:33.7994207Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:33.7994768Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:33.7995753Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:33.7996676Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:33.7997176Z AWS_REGION: us-east-1 2024-12-18T03:54:33.7997728Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:33.7998403Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:33.8008345Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:33.8008814Z ##[endgroup] 2024-12-18T03:54:33.8307871Z Prepare all required actions 2024-12-18T03:54:33.8364140Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T03:54:33.8364834Z with: 2024-12-18T03:54:33.8365519Z github-token: *** 2024-12-18T03:54:33.8365943Z env: 2024-12-18T03:54:33.8366325Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:33.8366856Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:33.8367814Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:33.8368709Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:33.8369200Z AWS_REGION: us-east-1 2024-12-18T03:54:33.8369701Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:33.8370357Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:33.8380070Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:33.8380607Z ##[endgroup] 2024-12-18T03:54:33.8408879Z ##[group]Run set -eux 2024-12-18T03:54:33.8409432Z set -eux 2024-12-18T03:54:33.8410382Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T03:54:33.8456557Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:33.8457227Z env: 2024-12-18T03:54:33.8457617Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:33.8458162Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:33.8459129Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:33.8460142Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:33.8460739Z AWS_REGION: us-east-1 2024-12-18T03:54:33.8461393Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:33.8462173Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:33.8472099Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:33.8472770Z GITHUB_TOKEN: *** 2024-12-18T03:54:33.8473196Z ##[endgroup] 2024-12-18T03:54:33.8541861Z + python3 .github/scripts/get_workflow_job_id.py 12383255596 pytorch-rocm-hw-43 2024-12-18T03:54:34.9523228Z setting job-id=34566663857 2024-12-18T03:54:34.9524238Z setting job-name=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:54:34.9875480Z Prepare all required actions 2024-12-18T03:54:34.9876236Z Getting action download info 2024-12-18T03:54:35.1928605Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T03:54:35.1929293Z with: 2024-12-18T03:54:35.1930059Z github-token: *** 2024-12-18T03:54:35.1931667Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-12-18T03:54:35.1933612Z job-name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:54:35.1934435Z env: 2024-12-18T03:54:35.1934863Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:35.1935448Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:35.1936454Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:35.1937887Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:35.1938423Z AWS_REGION: us-east-1 2024-12-18T03:54:35.1938965Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:35.1939643Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:35.1949481Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:35.1949942Z ##[endgroup] 2024-12-18T03:54:35.2016678Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T03:54:35.2017263Z with: 2024-12-18T03:54:35.2017651Z shell: bash 2024-12-18T03:54:35.2018103Z timeout_minutes: 10 2024-12-18T03:54:35.2018555Z max_attempts: 5 2024-12-18T03:54:35.2018973Z retry_wait_seconds: 30 2024-12-18T03:54:35.2020329Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T03:54:35.2021725Z polling_interval_seconds: 1 2024-12-18T03:54:35.2022225Z warning_on_retry: true 2024-12-18T03:54:35.2022689Z continue_on_error: false 2024-12-18T03:54:35.2023171Z env: 2024-12-18T03:54:35.2023556Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:35.2024100Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:35.2025077Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:35.2025983Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:35.2026491Z AWS_REGION: us-east-1 2024-12-18T03:54:35.2027039Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:35.2027724Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:35.2037535Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:35.2038209Z GITHUB_TOKEN: *** 2024-12-18T03:54:35.2038655Z ##[endgroup] 2024-12-18T03:54:35.2774912Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T03:54:35.5276585Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T03:54:35.6056430Z Requirement already satisfied: requests==2.27.1 in /home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-12-18T03:54:35.6061406Z Requirement already satisfied: pyyaml==6.0.1 in /home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-12-18T03:54:35.6156735Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-12-18T03:54:35.6166858Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-12-18T03:54:35.6171195Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-12-18T03:54:35.6175768Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-12-18T03:54:36.2775227Z Command completed after 1 attempt(s). 2024-12-18T03:54:36.2910681Z ##[group]Run set -x 2024-12-18T03:54:36.2911186Z set -x 2024-12-18T03:54:36.2911591Z  2024-12-18T03:54:36.2912267Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T03:54:36.2913586Z # in runner workspace 2024-12-18T03:54:36.2914294Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T03:54:36.2965719Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:36.2966404Z env: 2024-12-18T03:54:36.2966797Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:36.2967351Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:36.2968362Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:36.2969277Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:36.2969795Z AWS_REGION: us-east-1 2024-12-18T03:54:36.2970376Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:36.2971005Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:36.2980179Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:36.2980629Z ##[endgroup] 2024-12-18T03:54:36.3061712Z + python3 /home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T03:54:36.3290509Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T03:54:36.3291389Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T03:54:36.3292082Z echo "Job name: ${JOB_NAME}" 2024-12-18T03:54:36.3292678Z  2024-12-18T03:54:36.3293464Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T03:54:36.3294452Z # in runner workspace 2024-12-18T03:54:36.3295188Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T03:54:36.3295696Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T03:54:36.3296019Z  --job-name "${JOB_NAME}" \ 2024-12-18T03:54:36.3296904Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-12-18T03:54:36.3297842Z  --selected-test-configs "" \ 2024-12-18T03:54:36.3298150Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T03:54:36.3298433Z  --tag "${TAG}" \ 2024-12-18T03:54:36.3298710Z  --event-name "${EVENT_NAME}" \ 2024-12-18T03:54:36.3299005Z  --schedule "${SCHEDULE}" \ 2024-12-18T03:54:36.3299285Z  --branch "${HEAD_BRANCH}" 2024-12-18T03:54:36.3346772Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:36.3347455Z env: 2024-12-18T03:54:36.3347853Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:36.3348392Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:36.3349342Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:36.3350231Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:36.3350721Z AWS_REGION: us-east-1 2024-12-18T03:54:36.3351283Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:36.3351988Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:36.3361953Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:36.3362679Z GITHUB_TOKEN: *** 2024-12-18T03:54:36.3363402Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:54:36.3364196Z PR_NUMBER: 2024-12-18T03:54:36.3364591Z TAG: 2024-12-18T03:54:36.3364954Z EVENT_NAME: push 2024-12-18T03:54:36.3365360Z SCHEDULE: 2024-12-18T03:54:36.3365744Z HEAD_BRANCH: 2024-12-18T03:54:36.3366144Z ##[endgroup] 2024-12-18T03:54:36.3444335Z Workflow: trunk 2024-12-18T03:54:36.3445108Z Job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:54:36.9992390Z ##[group]Run echo "Filtered matrix:" 2024-12-18T03:54:36.9993021Z echo "Filtered matrix:" 2024-12-18T03:54:36.9995041Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-12-18T03:54:36.9996707Z  2024-12-18T03:54:36.9997080Z echo 2024-12-18T03:54:36.9997565Z echo "Is the current job unstable? False" 2024-12-18T03:54:36.9998155Z  2024-12-18T03:54:36.9998516Z echo 2024-12-18T03:54:36.9998975Z echo "Is keep-going label set? False" 2024-12-18T03:54:36.9999687Z  2024-12-18T03:54:37.0000037Z echo 2024-12-18T03:54:37.0000465Z echo "Renabled issues? " 2024-12-18T03:54:37.0046535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:37.0047226Z env: 2024-12-18T03:54:37.0047632Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:37.0048203Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:37.0049225Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:37.0050302Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:37.0051397Z AWS_REGION: us-east-1 2024-12-18T03:54:37.0052198Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:37.0053008Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:37.0059344Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:37.0059586Z ##[endgroup] 2024-12-18T03:54:37.0134726Z Filtered matrix: 2024-12-18T03:54:37.0136495Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-12-18T03:54:37.0137844Z 2024-12-18T03:54:37.0138097Z Is the current job unstable? False 2024-12-18T03:54:37.0138784Z 2024-12-18T03:54:37.0139043Z Is keep-going label set? False 2024-12-18T03:54:37.0139387Z 2024-12-18T03:54:37.0139557Z Renabled issues? 2024-12-18T03:54:37.0214642Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T03:54:37.0215759Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T03:54:37.0263254Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:54:37.0264054Z env: 2024-12-18T03:54:37.0264494Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:37.0265051Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:37.0266039Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:37.0266967Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:37.0267465Z AWS_REGION: us-east-1 2024-12-18T03:54:37.0268006Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:37.0268652Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:37.0278398Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:37.0278857Z JOB_TIMEOUT: 300 2024-12-18T03:54:37.0279270Z ##[endgroup] 2024-12-18T03:54:37.0435023Z ##[group]Run set -x 2024-12-18T03:54:37.0435617Z set -x 2024-12-18T03:54:37.0436019Z  2024-12-18T03:54:37.0436480Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T03:54:37.0437185Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T03:54:37.0437893Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T03:54:37.0438532Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-12-18T03:54:37.0439068Z else 2024-12-18T03:54:37.0439682Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T03:54:37.0440235Z fi 2024-12-18T03:54:37.0440596Z  2024-12-18T03:54:37.0441187Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T03:54:37.0442141Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T03:54:37.0442957Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T03:54:37.0443677Z # shellcheck disable=SC2086,SC2090 2024-12-18T03:54:37.0444264Z container_name=$(docker run \ 2024-12-18T03:54:37.0444808Z  ${GPU_FLAG:-} \ 2024-12-18T03:54:37.0445301Z  -e BUILD_ENVIRONMENT \ 2024-12-18T03:54:37.0445831Z  -e PR_NUMBER \ 2024-12-18T03:54:37.0446337Z  -e GITHUB_ACTIONS \ 2024-12-18T03:54:37.0446847Z  -e GITHUB_REPOSITORY \ 2024-12-18T03:54:37.0447375Z  -e GITHUB_WORKFLOW \ 2024-12-18T03:54:37.0447874Z  -e GITHUB_JOB \ 2024-12-18T03:54:37.0448339Z  -e GITHUB_RUN_ID \ 2024-12-18T03:54:37.0448852Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T03:54:37.0449462Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T03:54:37.0450066Z  -e JOB_ID \ 2024-12-18T03:54:37.0450591Z  -e JOB_NAME \ 2024-12-18T03:54:37.0451116Z  -e BRANCH \ 2024-12-18T03:54:37.0451621Z  -e SHA1 \ 2024-12-18T03:54:37.0452147Z  -e AWS_DEFAULT_REGION \ 2024-12-18T03:54:37.0452763Z  -e IN_WHEEL_TEST \ 2024-12-18T03:54:37.0453343Z  -e SHARD_NUMBER \ 2024-12-18T03:54:37.0453918Z  -e TEST_CONFIG \ 2024-12-18T03:54:37.0454499Z  -e NUM_TEST_SHARDS \ 2024-12-18T03:54:37.0455113Z  -e REENABLED_ISSUES \ 2024-12-18T03:54:37.0456398Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T03:54:37.0457050Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T03:54:37.0457656Z  -e TEST_SHOWLOCALS \ 2024-12-18T03:54:37.0458296Z  -e NO_TEST_TIMEOUT \ 2024-12-18T03:54:37.0458788Z  -e NO_TD \ 2024-12-18T03:54:37.0459313Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T03:54:37.0459902Z  -e SCCACHE_BUCKET \ 2024-12-18T03:54:37.0460514Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T03:54:37.0461164Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T03:54:37.0461926Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T03:54:37.0462645Z  -e TESTS_TO_INCLUDE \ 2024-12-18T03:54:37.0463370Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T03:54:37.0464136Z  --ulimit stack=10485760:83886080 \ 2024-12-18T03:54:37.0464766Z  --ulimit core=0 \ 2024-12-18T03:54:37.0465306Z  --security-opt seccomp=unconfined \ 2024-12-18T03:54:37.0465905Z  --cap-add=SYS_PTRACE \ 2024-12-18T03:54:37.0466412Z  --shm-size="8g" \ 2024-12-18T03:54:37.0466880Z  --tty \ 2024-12-18T03:54:37.0467294Z  --detach \ 2024-12-18T03:54:37.0467757Z  --name="${container_name}" \ 2024-12-18T03:54:37.0468295Z  --user jenkins \ 2024-12-18T03:54:37.0468899Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T03:54:37.0469596Z  -w /var/lib/jenkins/workspace \ 2024-12-18T03:54:37.0470151Z  "${DOCKER_IMAGE}" 2024-12-18T03:54:37.0470606Z ) 2024-12-18T03:54:37.0471042Z # save container name for later step 2024-12-18T03:54:37.0472133Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-12-18T03:54:37.0473399Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-12-18T03:54:37.0474987Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-12-18T03:54:37.0520495Z shell: /usr/bin/bash -e {0} 2024-12-18T03:54:37.0520998Z env: 2024-12-18T03:54:37.0521397Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:54:37.0521953Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T03:54:37.0522930Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:54:37.0523858Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:54:37.0524363Z AWS_REGION: us-east-1 2024-12-18T03:54:37.0524913Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:54:37.0525590Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:54:37.0535397Z AWS_SESSION_TOKEN: *** 2024-12-18T03:54:37.0536055Z BUILD_ENVIRONMENT: linux-focal-rocm6.2-py3.10 2024-12-18T03:54:37.0536763Z PR_NUMBER: 2024-12-18T03:54:37.0537274Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T03:54:37.0537912Z GITHUB_WORKFLOW: trunk 2024-12-18T03:54:37.0538467Z GITHUB_JOB: test 2024-12-18T03:54:37.0538964Z GITHUB_RUN_ID: 12383255596 2024-12-18T03:54:37.0539520Z GITHUB_RUN_NUMBER: 109701 2024-12-18T03:54:37.0540075Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T03:54:37.0540586Z JOB_ID: 34566663857 2024-12-18T03:54:37.0541411Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:54:37.0542310Z BRANCH: release/2.6 2024-12-18T03:54:37.0542812Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:54:37.0543406Z CONTINUE_THROUGH_ERROR: False 2024-12-18T03:54:37.0543898Z VERBOSE_TEST_LOGS: False 2024-12-18T03:54:37.0544366Z TEST_SHOWLOCALS: False 2024-12-18T03:54:37.0544807Z NO_TEST_TIMEOUT: False 2024-12-18T03:54:37.0545243Z NO_TD: False 2024-12-18T03:54:37.0545656Z TEST_CONFIG: distributed 2024-12-18T03:54:37.0546098Z SHARD_NUMBER: 1 2024-12-18T03:54:37.0546510Z NUM_TEST_SHARDS: 1 2024-12-18T03:54:37.0546924Z REENABLED_ISSUES: 2024-12-18T03:54:37.0547478Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T03:54:37.0549479Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:54:37.0551079Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:54:37.0551865Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T03:54:37.0552426Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T03:54:37.0584493Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-12-18T03:54:37.0585993Z ##[endgroup] 2024-12-18T03:54:37.0659078Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T03:54:37.0659729Z + [[ linux-focal-rocm6.2-py3.10 == *onnx* ]] 2024-12-18T03:54:37.0660310Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T03:54:37.0671276Z +++ nproc --ignore=2 2024-12-18T03:54:37.0700112Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_12383255596 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:54:40.4429747Z + container_name=548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T03:54:40.4430779Z + echo CONTAINER_NAME=548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T03:54:40.4432220Z + docker exec -t 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-12-18T03:55:04.5765194Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T03:55:04.9923115Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T03:55:04.9926184Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T03:55:04.9928115Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T03:55:04.9929975Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T03:55:04.9931833Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T03:55:04.9937079Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T03:55:04.9950095Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T03:55:05.0253911Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T03:55:05.3329567Z Installing collected packages: torch 2024-12-18T03:55:13.8020856Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T03:55:13.8544084Z + export TERM=vt100 2024-12-18T03:55:13.8544638Z + TERM=vt100 2024-12-18T03:55:13.8549760Z ++ dirname .ci/pytorch/test.sh 2024-12-18T03:55:13.8571051Z + source .ci/pytorch/common.sh 2024-12-18T03:55:13.8574468Z +++ dirname .ci/pytorch/common.sh 2024-12-18T03:55:13.8583281Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T03:55:13.8583951Z +++ declare -f -t trap_add 2024-12-18T03:55:13.8591144Z ++ set -ex 2024-12-18T03:55:13.8591839Z ++ [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:55:13.8592542Z ++ unset HIP_PLATFORM 2024-12-18T03:55:13.8593058Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:55:13.8593667Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:55:13.8594164Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:55:13.8594665Z ++ HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:55:13.8595139Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:55:13.8595687Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:55:13.8596185Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T03:55:13.8596722Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-12-18T03:55:13.8597334Z + echo 'Environment variables:' 2024-12-18T03:55:13.8597830Z Environment variables: 2024-12-18T03:55:13.8598255Z + env 2024-12-18T03:55:13.8607762Z INSTALLED_DB=yes 2024-12-18T03:55:13.8608205Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:55:13.8608676Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-12-18T03:55:13.8608998Z CONTINUE_THROUGH_ERROR=False 2024-12-18T03:55:13.8609307Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-12-18T03:55:13.8609626Z HOSTNAME=548d3077c0cd 2024-12-18T03:55:13.8610693Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8611273Z GITHUB_ACTION=__self 2024-12-18T03:55:13.8611538Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T03:55:13.8611825Z GITHUB_RUN_NUMBER=109701 2024-12-18T03:55:13.8612072Z TEST_CONFIG=distributed 2024-12-18T03:55:13.8612331Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T03:55:13.8612618Z AWS_DEFAULT_REGION=us-east-1 2024-12-18T03:55:13.8612876Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T03:55:13.8613130Z GITHUB_REF_TYPE=branch 2024-12-18T03:55:13.8615696Z *** 2024-12-18T03:55:13.8615929Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T03:55:13.8616200Z GITHUB_ACTIONS=true 2024-12-18T03:55:13.8616459Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8616800Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8617295Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/heads/release/2.6 2024-12-18T03:55:13.8617743Z VERBOSE_TEST_LOGS=False 2024-12-18T03:55:13.8618004Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T03:55:13.8618270Z SHARD_NUMBER=1 2024-12-18T03:55:13.8618493Z GITHUB_REF_PROTECTED=true 2024-12-18T03:55:13.8618737Z HOME=/var/lib/jenkins 2024-12-18T03:55:13.8618993Z GITHUB_API_URL=https://api.github.com 2024-12-18T03:55:13.8619296Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T03:55:13.8619582Z LANG=C.UTF-8 2024-12-18T03:55:13.8619796Z PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:55:13.8620030Z NUM_TEST_SHARDS=1 2024-12-18T03:55:13.8620563Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8621293Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:55:13.8621712Z MAGMA_HOME=/opt/rocm/magma 2024-12-18T03:55:13.8622257Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8622801Z HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:55:13.8623258Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T03:55:13.8623697Z GITHUB_EVENT_NAME=push 2024-12-18T03:55:13.8623931Z GITHUB_RUN_ID=12383255596 2024-12-18T03:55:13.8624497Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8625418Z GITHUB_ACTOR=malfet 2024-12-18T03:55:13.8625629Z PR_NUMBER= 2024-12-18T03:55:13.8625830Z GITHUB_RUN_ATTEMPT=1 2024-12-18T03:55:13.8626059Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T03:55:13.8626351Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T03:55:13.8626653Z TERM=vt100 2024-12-18T03:55:13.8626851Z INSTALLED_VISION=yes 2024-12-18T03:55:13.8627063Z BRANCH=release/2.6 2024-12-18T03:55:13.8627765Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-12-18T03:55:13.8628735Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-12-18T03:55:13.8629240Z GITHUB_SERVER_URL=https://github.com 2024-12-18T03:55:13.8629519Z PYTORCH_ROCM_ARCH=gfx90a 2024-12-18T03:55:13.8629753Z REENABLED_ISSUES= 2024-12-18T03:55:13.8629954Z SHLVL=1 2024-12-18T03:55:13.8630144Z MAX_JOBS=62 2024-12-18T03:55:13.8630344Z GITHUB_ACTOR_ID=2453524 2024-12-18T03:55:13.8630697Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8631350Z GITHUB_REF_NAME=release/2.6 2024-12-18T03:55:13.8631853Z ROCM_PATH=/opt/rocm 2024-12-18T03:55:13.8632558Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:55:13.8633355Z GITHUB_JOB=test 2024-12-18T03:55:13.8633759Z NO_TEST_TIMEOUT=False 2024-12-18T03:55:13.8634192Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T03:55:13.8634653Z LC_ALL=C.UTF-8 2024-12-18T03:55:13.8635027Z GITHUB_RETENTION_DAYS=90 2024-12-18T03:55:13.8635455Z GITHUB_ACTION_REPOSITORY= 2024-12-18T03:55:13.8637369Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:55:13.8638960Z GITHUB_BASE_REF= 2024-12-18T03:55:13.8639325Z CI=true 2024-12-18T03:55:13.8639831Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:55:13.8640285Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T03:55:13.8640732Z JOB_ID=34566663857 2024-12-18T03:55:13.8641114Z INSTALLED_PROTOBUF=yes 2024-12-18T03:55:13.8641515Z GITHUB_HEAD_REF= 2024-12-18T03:55:13.8641892Z GITHUB_ACTION_REF= 2024-12-18T03:55:13.8642362Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T03:55:13.8642935Z TEST_SHOWLOCALS=False 2024-12-18T03:55:13.8643344Z GITHUB_WORKFLOW=trunk 2024-12-18T03:55:13.8643763Z DEBIAN_FRONTEND=noninteractive 2024-12-18T03:55:13.8644773Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8645794Z NO_TD=False 2024-12-18T03:55:13.8646150Z OLDPWD=/var/lib/jenkins 2024-12-18T03:55:13.8646547Z _=/usr/bin/env 2024-12-18T03:55:13.8647055Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T03:55:13.8837258Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T03:55:13.8838630Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T03:55:13.8839811Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T03:55:13.8840824Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T03:55:13.8841591Z + BUILD_DIR=build 2024-12-18T03:55:13.8842033Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T03:55:13.8842564Z + BUILD_BIN_DIR=build/bin 2024-12-18T03:55:13.8843013Z + SHARD_NUMBER=1 2024-12-18T03:55:13.8843407Z + NUM_TEST_SHARDS=1 2024-12-18T03:55:13.8843853Z + export VALGRIND=ON 2024-12-18T03:55:13.8844281Z + VALGRIND=ON 2024-12-18T03:55:13.8844747Z + [[ linux-focal-rocm6.2-py3.10 == *clang9* ]] 2024-12-18T03:55:13.8845383Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-12-18T03:55:13.8845903Z + [[ 0 == \1 ]] 2024-12-18T03:55:13.8846283Z + [[ False == \1 ]] 2024-12-18T03:55:13.8846733Z + [[ linux-focal-rocm6.2-py3.10 != *bazel* ]] 2024-12-18T03:55:13.8847776Z ++ realpath build/custom_test_artifacts 2024-12-18T03:55:13.8868756Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-12-18T03:55:13.8870623Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-12-18T03:55:13.8872053Z + echo 'Setting INCLUDE_CLAUSE' 2024-12-18T03:55:13.8872587Z Setting INCLUDE_CLAUSE 2024-12-18T03:55:13.8874076Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-12-18T03:55:13.8875647Z + echo 'Environment variables' 2024-12-18T03:55:13.8876141Z Environment variables 2024-12-18T03:55:13.8876571Z + env 2024-12-18T03:55:13.8877249Z INSTALLED_DB=yes 2024-12-18T03:55:13.8877949Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:55:13.8878779Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-12-18T03:55:13.8879526Z CONTINUE_THROUGH_ERROR=False 2024-12-18T03:55:13.8880091Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-12-18T03:55:13.8880678Z HOSTNAME=548d3077c0cd 2024-12-18T03:55:13.8881752Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8882819Z GITHUB_ACTION=__self 2024-12-18T03:55:13.8883285Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T03:55:13.8883812Z GITHUB_RUN_NUMBER=109701 2024-12-18T03:55:13.8884268Z TEST_CONFIG=distributed 2024-12-18T03:55:13.8884742Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T03:55:13.8885712Z AWS_DEFAULT_REGION=us-east-1 2024-12-18T03:55:13.8886258Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T03:55:13.8886748Z GITHUB_REF_TYPE=branch 2024-12-18T03:55:13.8887285Z *** 2024-12-18T03:55:13.8887687Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T03:55:13.8888210Z GITHUB_ACTIONS=true 2024-12-18T03:55:13.8888712Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8889365Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8890291Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/heads/release/2.6 2024-12-18T03:55:13.8891147Z VERBOSE_TEST_LOGS=False 2024-12-18T03:55:13.8891667Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T03:55:13.8892183Z SHARD_NUMBER=1 2024-12-18T03:55:13.8892434Z GITHUB_REF_PROTECTED=true 2024-12-18T03:55:13.8892702Z HOME=/var/lib/jenkins 2024-12-18T03:55:13.8892984Z GITHUB_API_URL=https://api.github.com 2024-12-18T03:55:13.8893324Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T03:55:13.8893624Z LANG=C.UTF-8 2024-12-18T03:55:13.8893866Z PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:55:13.8894134Z NUM_TEST_SHARDS=1 2024-12-18T03:55:13.8894727Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8895529Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-12-18T03:55:13.8895997Z MAGMA_HOME=/opt/rocm/magma 2024-12-18T03:55:13.8896582Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8897206Z HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:55:13.8897656Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T03:55:13.8898167Z GITHUB_EVENT_NAME=push 2024-12-18T03:55:13.8898430Z GITHUB_RUN_ID=12383255596 2024-12-18T03:55:13.8899075Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8899758Z GITHUB_ACTOR=malfet 2024-12-18T03:55:13.8900013Z PR_NUMBER= 2024-12-18T03:55:13.8900283Z GITHUB_RUN_ATTEMPT=1 2024-12-18T03:55:13.8900619Z VALGRIND=ON 2024-12-18T03:55:13.8901131Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T03:55:13.8901906Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T03:55:13.8902569Z TERM=vt100 2024-12-18T03:55:13.8903266Z INSTALLED_VISION=yes 2024-12-18T03:55:13.8903686Z BRANCH=release/2.6 2024-12-18T03:55:13.8905045Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-12-18T03:55:13.8906928Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-12-18T03:55:13.8907915Z GITHUB_SERVER_URL=https://github.com 2024-12-18T03:55:13.8908460Z PYTORCH_ROCM_ARCH=gfx90a 2024-12-18T03:55:13.8908908Z REENABLED_ISSUES= 2024-12-18T03:55:13.8909306Z SHLVL=1 2024-12-18T03:55:13.8909657Z MAX_JOBS=62 2024-12-18T03:55:13.8910053Z GITHUB_ACTOR_ID=2453524 2024-12-18T03:55:13.8910642Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:55:13.8911313Z GITHUB_REF_NAME=release/2.6 2024-12-18T03:55:13.8911780Z ROCM_PATH=/opt/rocm 2024-12-18T03:55:13.8912430Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:55:13.8913165Z GITHUB_JOB=test 2024-12-18T03:55:13.8913576Z NO_TEST_TIMEOUT=False 2024-12-18T03:55:13.8914044Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T03:55:13.8914538Z LC_ALL=C.UTF-8 2024-12-18T03:55:13.8914938Z GITHUB_RETENTION_DAYS=90 2024-12-18T03:55:13.8915409Z GITHUB_ACTION_REPOSITORY= 2024-12-18T03:55:13.8917023Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:55:13.8918659Z GITHUB_BASE_REF= 2024-12-18T03:55:13.8919043Z CI=true 2024-12-18T03:55:13.8919546Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:55:13.8920380Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T03:55:13.8920885Z JOB_ID=34566663857 2024-12-18T03:55:13.8921301Z INSTALLED_PROTOBUF=yes 2024-12-18T03:55:13.8921729Z GITHUB_HEAD_REF= 2024-12-18T03:55:13.8922131Z GITHUB_ACTION_REF= 2024-12-18T03:55:13.8922632Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T03:55:13.8923249Z TEST_SHOWLOCALS=False 2024-12-18T03:55:13.8923677Z GITHUB_WORKFLOW=trunk 2024-12-18T03:55:13.8924123Z DEBIAN_FRONTEND=noninteractive 2024-12-18T03:55:13.8925210Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_43e50a74-22f9-4fc9-8f79-519dd7e1f87a 2024-12-18T03:55:13.8926291Z NO_TD=False 2024-12-18T03:55:13.8926666Z OLDPWD=/var/lib/jenkins 2024-12-18T03:55:13.8927088Z _=/usr/bin/env 2024-12-18T03:55:13.8927512Z + echo 'Testing pytorch' 2024-12-18T03:55:13.8927949Z Testing pytorch 2024-12-18T03:55:13.8928349Z + export LANG=C.UTF-8 2024-12-18T03:55:13.8928752Z + LANG=C.UTF-8 2024-12-18T03:55:13.8929125Z + PR_NUMBER= 2024-12-18T03:55:13.8929538Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-12-18T03:55:13.8930085Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T03:55:13.8930668Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:55:13.8931236Z + export HIP_VISIBLE_DEVICES=0,1 2024-12-18T03:55:13.8931866Z + HIP_VISIBLE_DEVICES=0,1 2024-12-18T03:55:13.8932413Z + [[ distributed == \s\l\o\w ]] 2024-12-18T03:55:13.8933064Z + [[ linux-focal-rocm6.2-py3.10 == *slow-gradcheck* ]] 2024-12-18T03:55:13.8933477Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T03:55:13.8933821Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:55:13.8934140Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T03:55:13.8934456Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T03:55:13.8934754Z + [[ distributed == *crossref* ]] 2024-12-18T03:55:13.8935046Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:55:13.8935336Z + export VALGRIND=OFF 2024-12-18T03:55:13.8935561Z + VALGRIND=OFF 2024-12-18T03:55:13.8935772Z + rocminfo 2024-12-18T03:55:13.9044456Z ROCk module version 6.8.5 is loaded 2024-12-18T03:55:13.9647722Z ===================== 2024-12-18T03:55:13.9648344Z HSA System Attributes 2024-12-18T03:55:13.9648840Z ===================== 2024-12-18T03:55:13.9649327Z Runtime Version: 1.14 2024-12-18T03:55:13.9650323Z Runtime Ext Version: 1.6 2024-12-18T03:55:13.9650947Z System Timestamp Freq.: 1000.000000MHz 2024-12-18T03:55:13.9651995Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-12-18T03:55:13.9653050Z Machine Model: LARGE 2024-12-18T03:55:13.9653794Z System Endianness: LITTLE 2024-12-18T03:55:13.9654450Z Mwaitx: DISABLED 2024-12-18T03:55:13.9654998Z DMAbuf Support: YES 2024-12-18T03:55:13.9655307Z 2024-12-18T03:55:13.9655481Z ========== 2024-12-18T03:55:13.9655924Z HSA Agents 2024-12-18T03:55:13.9656362Z ========== 2024-12-18T03:55:13.9656767Z ******* 2024-12-18T03:55:13.9657221Z Agent 1 2024-12-18T03:55:13.9657637Z ******* 2024-12-18T03:55:13.9658154Z Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:13.9658822Z Uuid: CPU-XX 2024-12-18T03:55:13.9659496Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:13.9660206Z Vendor Name: CPU 2024-12-18T03:55:13.9660900Z Feature: None specified 2024-12-18T03:55:13.9661315Z Profile: FULL_PROFILE 2024-12-18T03:55:13.9661686Z Float Round Mode: NEAR 2024-12-18T03:55:13.9662077Z Max Queue Number: 0(0x0) 2024-12-18T03:55:13.9662453Z Queue Min Size: 0(0x0) 2024-12-18T03:55:13.9662981Z Queue Max Size: 0(0x0) 2024-12-18T03:55:13.9663370Z Queue Type: MULTI 2024-12-18T03:55:13.9663724Z Node: 0 2024-12-18T03:55:13.9664071Z Device Type: CPU 2024-12-18T03:55:13.9664408Z Cache Info: 2024-12-18T03:55:13.9664774Z L1: 32768(0x8000) KB 2024-12-18T03:55:13.9665121Z Chip ID: 0(0x0) 2024-12-18T03:55:13.9665490Z ASIC Revision: 0(0x0) 2024-12-18T03:55:13.9665877Z Cacheline Size: 64(0x40) 2024-12-18T03:55:13.9666259Z Max Clock Freq. (MHz): 2600 2024-12-18T03:55:13.9666619Z BDFID: 0 2024-12-18T03:55:13.9666980Z Internal Node ID: 0 2024-12-18T03:55:13.9667354Z Compute Unit: 32 2024-12-18T03:55:13.9667714Z SIMDs per CU: 0 2024-12-18T03:55:13.9668091Z Shader Engines: 0 2024-12-18T03:55:13.9668479Z Shader Arrs. per Eng.: 0 2024-12-18T03:55:13.9668873Z WatchPts on Addr. Ranges:1 2024-12-18T03:55:13.9669226Z Memory Properties: 2024-12-18T03:55:13.9669492Z Features: None 2024-12-18T03:55:13.9669753Z Pool Info: 2024-12-18T03:55:13.9670002Z Pool 1 2024-12-18T03:55:13.9670368Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:55:13.9671047Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:55:13.9671850Z Allocatable: TRUE 2024-12-18T03:55:13.9672587Z Alloc Granule: 4KB 2024-12-18T03:55:13.9673276Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9673962Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9674885Z Accessible by all: TRUE 2024-12-18T03:55:13.9675457Z Pool 2 2024-12-18T03:55:13.9675990Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:55:13.9676615Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:55:13.9677218Z Allocatable: TRUE 2024-12-18T03:55:13.9677854Z Alloc Granule: 4KB 2024-12-18T03:55:13.9678523Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9679204Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9680007Z Accessible by all: TRUE 2024-12-18T03:55:13.9680619Z Pool 3 2024-12-18T03:55:13.9681243Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:55:13.9681961Z Size: 65790776(0x3ebe338) KB 2024-12-18T03:55:13.9682690Z Allocatable: TRUE 2024-12-18T03:55:13.9683330Z Alloc Granule: 4KB 2024-12-18T03:55:13.9683994Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9684679Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9685339Z Accessible by all: TRUE 2024-12-18T03:55:13.9685926Z ISA Info: 2024-12-18T03:55:13.9686347Z ******* 2024-12-18T03:55:13.9686766Z Agent 2 2024-12-18T03:55:13.9687438Z ******* 2024-12-18T03:55:13.9687926Z Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:13.9688529Z Uuid: CPU-XX 2024-12-18T03:55:13.9689159Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:13.9689825Z Vendor Name: CPU 2024-12-18T03:55:13.9690473Z Feature: None specified 2024-12-18T03:55:13.9691213Z Profile: FULL_PROFILE 2024-12-18T03:55:13.9691954Z Float Round Mode: NEAR 2024-12-18T03:55:13.9692706Z Max Queue Number: 0(0x0) 2024-12-18T03:55:13.9693450Z Queue Min Size: 0(0x0) 2024-12-18T03:55:13.9694171Z Queue Max Size: 0(0x0) 2024-12-18T03:55:13.9694890Z Queue Type: MULTI 2024-12-18T03:55:13.9695589Z Node: 1 2024-12-18T03:55:13.9695990Z Device Type: CPU 2024-12-18T03:55:13.9696340Z Cache Info: 2024-12-18T03:55:13.9696594Z L1: 32768(0x8000) KB 2024-12-18T03:55:13.9696913Z Chip ID: 0(0x0) 2024-12-18T03:55:13.9697246Z ASIC Revision: 0(0x0) 2024-12-18T03:55:13.9697599Z Cacheline Size: 64(0x40) 2024-12-18T03:55:13.9697950Z Max Clock Freq. (MHz): 2600 2024-12-18T03:55:13.9698278Z BDFID: 0 2024-12-18T03:55:13.9698632Z Internal Node ID: 1 2024-12-18T03:55:13.9698989Z Compute Unit: 32 2024-12-18T03:55:13.9699339Z SIMDs per CU: 0 2024-12-18T03:55:13.9699680Z Shader Engines: 0 2024-12-18T03:55:13.9700039Z Shader Arrs. per Eng.: 0 2024-12-18T03:55:13.9700408Z WatchPts on Addr. Ranges:1 2024-12-18T03:55:13.9700959Z Memory Properties: 2024-12-18T03:55:13.9701322Z Features: None 2024-12-18T03:55:13.9701886Z Pool Info: 2024-12-18T03:55:13.9702417Z Pool 1 2024-12-18T03:55:13.9703096Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:55:13.9703808Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:55:13.9704477Z Allocatable: TRUE 2024-12-18T03:55:13.9705170Z Alloc Granule: 4KB 2024-12-18T03:55:13.9705902Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9706643Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9707345Z Accessible by all: TRUE 2024-12-18T03:55:13.9707945Z Pool 2 2024-12-18T03:55:13.9708524Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:55:13.9709195Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:55:13.9709851Z Allocatable: TRUE 2024-12-18T03:55:13.9710541Z Alloc Granule: 4KB 2024-12-18T03:55:13.9711254Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9711978Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9712679Z Accessible by all: TRUE 2024-12-18T03:55:13.9713279Z Pool 3 2024-12-18T03:55:13.9714072Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:55:13.9714738Z Size: 66046476(0x3efca0c) KB 2024-12-18T03:55:13.9715377Z Allocatable: TRUE 2024-12-18T03:55:13.9716053Z Alloc Granule: 4KB 2024-12-18T03:55:13.9716774Z Alloc Recommended Granule:4KB 2024-12-18T03:55:13.9717487Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9718179Z Accessible by all: TRUE 2024-12-18T03:55:13.9718781Z ISA Info: 2024-12-18T03:55:13.9719214Z ******* 2024-12-18T03:55:13.9719736Z Agent 3 2024-12-18T03:55:13.9720150Z ******* 2024-12-18T03:55:13.9720627Z Name: gfx90a 2024-12-18T03:55:13.9721277Z Uuid: GPU-57cd99ed4cd4643e 2024-12-18T03:55:13.9721952Z Marketing Name: AMD Instinct MI210 2024-12-18T03:55:13.9722634Z Vendor Name: AMD 2024-12-18T03:55:13.9723295Z Feature: KERNEL_DISPATCH 2024-12-18T03:55:13.9723966Z Profile: BASE_PROFILE 2024-12-18T03:55:13.9724637Z Float Round Mode: NEAR 2024-12-18T03:55:13.9725306Z Max Queue Number: 128(0x80) 2024-12-18T03:55:13.9725968Z Queue Min Size: 64(0x40) 2024-12-18T03:55:13.9726619Z Queue Max Size: 131072(0x20000) 2024-12-18T03:55:13.9727265Z Queue Type: MULTI 2024-12-18T03:55:13.9727879Z Node: 2 2024-12-18T03:55:13.9728505Z Device Type: GPU 2024-12-18T03:55:13.9729097Z Cache Info: 2024-12-18T03:55:13.9729592Z L1: 16(0x10) KB 2024-12-18T03:55:13.9730171Z L2: 8192(0x2000) KB 2024-12-18T03:55:13.9730836Z Chip ID: 29711(0x740f) 2024-12-18T03:55:13.9731968Z ASIC Revision: 1(0x1) 2024-12-18T03:55:13.9732763Z Cacheline Size: 64(0x40) 2024-12-18T03:55:13.9733575Z Max Clock Freq. (MHz): 1700 2024-12-18T03:55:13.9734342Z BDFID: 768 2024-12-18T03:55:13.9735103Z Internal Node ID: 2 2024-12-18T03:55:13.9735777Z Compute Unit: 104 2024-12-18T03:55:13.9736176Z SIMDs per CU: 4 2024-12-18T03:55:13.9736545Z Shader Engines: 8 2024-12-18T03:55:13.9736906Z Shader Arrs. per Eng.: 1 2024-12-18T03:55:13.9737275Z WatchPts on Addr. Ranges:4 2024-12-18T03:55:13.9737649Z Coherent Host Access: FALSE 2024-12-18T03:55:13.9737990Z Memory Properties: 2024-12-18T03:55:13.9738259Z Features: KERNEL_DISPATCH 2024-12-18T03:55:13.9738602Z Fast F16 Operation: TRUE 2024-12-18T03:55:13.9738961Z Wavefront Size: 64(0x40) 2024-12-18T03:55:13.9739318Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:55:13.9739650Z Workgroup Max Size per Dimension: 2024-12-18T03:55:13.9739935Z x 1024(0x400) 2024-12-18T03:55:13.9740228Z y 1024(0x400) 2024-12-18T03:55:13.9740704Z z 1024(0x400) 2024-12-18T03:55:13.9741117Z Max Waves Per CU: 32(0x20) 2024-12-18T03:55:13.9741896Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:55:13.9742705Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:55:13.9743398Z Grid Max Size per Dimension: 2024-12-18T03:55:13.9743900Z x 4294967295(0xffffffff) 2024-12-18T03:55:13.9744465Z y 4294967295(0xffffffff) 2024-12-18T03:55:13.9745029Z z 4294967295(0xffffffff) 2024-12-18T03:55:13.9745676Z Max fbarriers/Workgrp: 32 2024-12-18T03:55:13.9746458Z Packet Processor uCode:: 83 2024-12-18T03:55:13.9747171Z SDMA engine uCode:: 8 2024-12-18T03:55:13.9747863Z IOMMU Support:: None 2024-12-18T03:55:13.9748458Z Pool Info: 2024-12-18T03:55:13.9748902Z Pool 1 2024-12-18T03:55:13.9749471Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:55:13.9750143Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9750811Z Allocatable: TRUE 2024-12-18T03:55:13.9751499Z Alloc Granule: 4KB 2024-12-18T03:55:13.9752216Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9752949Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9753661Z Accessible by all: FALSE 2024-12-18T03:55:13.9754271Z Pool 2 2024-12-18T03:55:13.9754853Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:55:13.9755546Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9756224Z Allocatable: TRUE 2024-12-18T03:55:13.9756919Z Alloc Granule: 4KB 2024-12-18T03:55:13.9757643Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9758632Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9759341Z Accessible by all: FALSE 2024-12-18T03:55:13.9760098Z Pool 3 2024-12-18T03:55:13.9760652Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:55:13.9761296Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9761942Z Allocatable: TRUE 2024-12-18T03:55:13.9762619Z Alloc Granule: 4KB 2024-12-18T03:55:13.9763356Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9764092Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9764794Z Accessible by all: FALSE 2024-12-18T03:55:13.9765393Z Pool 4 2024-12-18T03:55:13.9765938Z Segment: GROUP 2024-12-18T03:55:13.9766551Z Size: 64(0x40) KB 2024-12-18T03:55:13.9767183Z Allocatable: FALSE 2024-12-18T03:55:13.9767863Z Alloc Granule: 0KB 2024-12-18T03:55:13.9768573Z Alloc Recommended Granule:0KB 2024-12-18T03:55:13.9769292Z Alloc Alignment: 0KB 2024-12-18T03:55:13.9769988Z Accessible by all: FALSE 2024-12-18T03:55:13.9770986Z ISA Info: 2024-12-18T03:55:13.9771528Z ISA 1 2024-12-18T03:55:13.9772216Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:55:13.9773105Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:55:13.9773967Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:55:13.9774849Z Default Rounding Mode: NEAR 2024-12-18T03:55:13.9775701Z Default Rounding Mode: NEAR 2024-12-18T03:55:13.9776141Z Fast f16: TRUE 2024-12-18T03:55:13.9776537Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:55:13.9776881Z Workgroup Max Size per Dimension: 2024-12-18T03:55:13.9777183Z x 1024(0x400) 2024-12-18T03:55:13.9777494Z y 1024(0x400) 2024-12-18T03:55:13.9777795Z z 1024(0x400) 2024-12-18T03:55:13.9778125Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:55:13.9778458Z Grid Max Size per Dimension: 2024-12-18T03:55:13.9778739Z x 4294967295(0xffffffff) 2024-12-18T03:55:13.9779050Z y 4294967295(0xffffffff) 2024-12-18T03:55:13.9779357Z z 4294967295(0xffffffff) 2024-12-18T03:55:13.9779695Z FBarrier Max Size: 32 2024-12-18T03:55:13.9780011Z ******* 2024-12-18T03:55:13.9780239Z Agent 4 2024-12-18T03:55:13.9780461Z ******* 2024-12-18T03:55:13.9780722Z Name: gfx90a 2024-12-18T03:55:13.9781122Z Uuid: GPU-6b7c2bd8af06ff6a 2024-12-18T03:55:13.9781842Z Marketing Name: AMD Instinct MI210 2024-12-18T03:55:13.9782668Z Vendor Name: AMD 2024-12-18T03:55:13.9783343Z Feature: KERNEL_DISPATCH 2024-12-18T03:55:13.9784009Z Profile: BASE_PROFILE 2024-12-18T03:55:13.9784984Z Float Round Mode: NEAR 2024-12-18T03:55:13.9785663Z Max Queue Number: 128(0x80) 2024-12-18T03:55:13.9786328Z Queue Min Size: 64(0x40) 2024-12-18T03:55:13.9786982Z Queue Max Size: 131072(0x20000) 2024-12-18T03:55:13.9787639Z Queue Type: MULTI 2024-12-18T03:55:13.9788251Z Node: 3 2024-12-18T03:55:13.9788886Z Device Type: GPU 2024-12-18T03:55:13.9789473Z Cache Info: 2024-12-18T03:55:13.9789969Z L1: 16(0x10) KB 2024-12-18T03:55:13.9790588Z L2: 8192(0x2000) KB 2024-12-18T03:55:13.9791286Z Chip ID: 29711(0x740f) 2024-12-18T03:55:13.9792055Z ASIC Revision: 1(0x1) 2024-12-18T03:55:13.9792850Z Cacheline Size: 64(0x40) 2024-12-18T03:55:13.9793559Z Max Clock Freq. (MHz): 1700 2024-12-18T03:55:13.9794192Z BDFID: 33536 2024-12-18T03:55:13.9794829Z Internal Node ID: 3 2024-12-18T03:55:13.9795495Z Compute Unit: 104 2024-12-18T03:55:13.9796144Z SIMDs per CU: 4 2024-12-18T03:55:13.9796810Z Shader Engines: 8 2024-12-18T03:55:13.9797726Z Shader Arrs. per Eng.: 1 2024-12-18T03:55:13.9798440Z WatchPts on Addr. Ranges:4 2024-12-18T03:55:13.9799158Z Coherent Host Access: FALSE 2024-12-18T03:55:13.9799942Z Memory Properties: 2024-12-18T03:55:13.9800448Z Features: KERNEL_DISPATCH 2024-12-18T03:55:13.9801079Z Fast F16 Operation: TRUE 2024-12-18T03:55:13.9801770Z Wavefront Size: 64(0x40) 2024-12-18T03:55:13.9802457Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:55:13.9803084Z Workgroup Max Size per Dimension: 2024-12-18T03:55:13.9803623Z x 1024(0x400) 2024-12-18T03:55:13.9804180Z y 1024(0x400) 2024-12-18T03:55:13.9804727Z z 1024(0x400) 2024-12-18T03:55:13.9805351Z Max Waves Per CU: 32(0x20) 2024-12-18T03:55:13.9806035Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:55:13.9806717Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:55:13.9807327Z Grid Max Size per Dimension: 2024-12-18T03:55:13.9807828Z x 4294967295(0xffffffff) 2024-12-18T03:55:13.9808505Z y 4294967295(0xffffffff) 2024-12-18T03:55:13.9809079Z z 4294967295(0xffffffff) 2024-12-18T03:55:13.9809734Z Max fbarriers/Workgrp: 32 2024-12-18T03:55:13.9810479Z Packet Processor uCode:: 83 2024-12-18T03:55:13.9811323Z SDMA engine uCode:: 8 2024-12-18T03:55:13.9812148Z IOMMU Support:: None 2024-12-18T03:55:13.9812865Z Pool Info: 2024-12-18T03:55:13.9813402Z Pool 1 2024-12-18T03:55:13.9814074Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:55:13.9814876Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9816035Z Allocatable: TRUE 2024-12-18T03:55:13.9816751Z Alloc Granule: 4KB 2024-12-18T03:55:13.9817425Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9818113Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9818771Z Accessible by all: FALSE 2024-12-18T03:55:13.9819337Z Pool 2 2024-12-18T03:55:13.9819860Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:55:13.9820484Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9821116Z Allocatable: TRUE 2024-12-18T03:55:13.9821910Z Alloc Granule: 4KB 2024-12-18T03:55:13.9822756Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9823621Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9824347Z Accessible by all: FALSE 2024-12-18T03:55:13.9824948Z Pool 3 2024-12-18T03:55:13.9825496Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:55:13.9826138Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:55:13.9826790Z Allocatable: TRUE 2024-12-18T03:55:13.9827466Z Alloc Granule: 4KB 2024-12-18T03:55:13.9828177Z Alloc Recommended Granule:2048KB 2024-12-18T03:55:13.9829191Z Alloc Alignment: 4KB 2024-12-18T03:55:13.9829911Z Accessible by all: FALSE 2024-12-18T03:55:13.9830512Z Pool 4 2024-12-18T03:55:13.9831037Z Segment: GROUP 2024-12-18T03:55:13.9831667Z Size: 64(0x40) KB 2024-12-18T03:55:13.9832314Z Allocatable: FALSE 2024-12-18T03:55:13.9833004Z Alloc Granule: 0KB 2024-12-18T03:55:13.9833715Z Alloc Recommended Granule:0KB 2024-12-18T03:55:13.9834431Z Alloc Alignment: 0KB 2024-12-18T03:55:13.9835134Z Accessible by all: FALSE 2024-12-18T03:55:13.9835742Z ISA Info: 2024-12-18T03:55:13.9836187Z ISA 1 2024-12-18T03:55:13.9836764Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:55:13.9837517Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:55:13.9838234Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:55:13.9838956Z Default Rounding Mode: NEAR 2024-12-18T03:55:13.9839794Z Default Rounding Mode: NEAR 2024-12-18T03:55:13.9840473Z Fast f16: TRUE 2024-12-18T03:55:13.9841144Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:55:13.9841791Z Workgroup Max Size per Dimension: 2024-12-18T03:55:13.9842362Z x 1024(0x400) 2024-12-18T03:55:13.9842934Z y 1024(0x400) 2024-12-18T03:55:13.9843508Z z 1024(0x400) 2024-12-18T03:55:13.9844131Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:55:13.9844747Z Grid Max Size per Dimension: 2024-12-18T03:55:13.9845275Z x 4294967295(0xffffffff) 2024-12-18T03:55:13.9846261Z y 4294967295(0xffffffff) 2024-12-18T03:55:13.9846827Z z 4294967295(0xffffffff) 2024-12-18T03:55:13.9847460Z FBarrier Max Size: 32 2024-12-18T03:55:13.9848057Z *** Done *** 2024-12-18T03:55:13.9899173Z + rocminfo 2024-12-18T03:55:13.9902487Z + grep -E 'Name:.*\sgfx|Marketing' 2024-12-18T03:55:14.0679743Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:14.0680555Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-12-18T03:55:14.0681242Z Name: gfx90a 2024-12-18T03:55:14.0681927Z Marketing Name: AMD Instinct MI210 2024-12-18T03:55:14.0682636Z Name: gfx90a 2024-12-18T03:55:14.0683291Z Marketing Name: AMD Instinct MI210 2024-12-18T03:55:14.0901586Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-12-18T03:55:14.0902398Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-12-18T03:55:14.0903052Z + pip_install --user ninja==1.10.2 2024-12-18T03:55:14.0903809Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T03:55:14.0904646Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T03:55:14.4309577Z Collecting ninja==1.10.2 2024-12-18T03:55:14.4529532Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T03:55:14.4626411Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T03:55:14.7386830Z Installing collected packages: ninja 2024-12-18T03:55:14.7453773Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T03:55:14.7455617Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T03:55:14.7477517Z Successfully installed ninja-1.10.2 2024-12-18T03:55:14.7950840Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:55:14.7954194Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:55:14.7956135Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-12-18T03:55:14.7956726Z + install_tlparse 2024-12-18T03:55:14.7957219Z + pip_install --user tlparse==0.3.25 2024-12-18T03:55:14.7957921Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T03:55:14.7958772Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T03:55:15.1277579Z Collecting tlparse==0.3.25 2024-12-18T03:55:15.1496975Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T03:55:15.1576290Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T03:55:15.4909365Z Installing collected packages: tlparse 2024-12-18T03:55:15.5200905Z Successfully installed tlparse-0.3.25 2024-12-18T03:55:15.5707257Z ++ python -m site --user-base 2024-12-18T03:55:15.5932126Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:55:15.5934424Z + [[ linux-focal-rocm6.2-py3.10 == *asan* ]] 2024-12-18T03:55:15.5935108Z + [[ linux-focal-rocm6.2-py3.10 == *-debug* ]] 2024-12-18T03:55:15.5935754Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-12-18T03:55:15.5936647Z + echo 'We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass' 2024-12-18T03:55:15.5938683Z We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass 2024-12-18T03:55:15.5939452Z + cd test 2024-12-18T03:55:15.5940060Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T03:55:18.3045347Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T03:55:18.3046162Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T03:55:18.3051257Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T03:55:18.3051821Z + [[ distributed == *pr_time_benchmarks* ]] 2024-12-18T03:55:18.3052370Z + [[ distributed == *dynamo_eager* ]] 2024-12-18T03:55:18.3052890Z + [[ distributed == *aot_eager* ]] 2024-12-18T03:55:18.3053403Z + [[ distributed == *aot_inductor* ]] 2024-12-18T03:55:18.3053723Z + [[ distributed == *inductor* ]] 2024-12-18T03:55:18.3053943Z + [[ distributed == *dynamic* ]] 2024-12-18T03:55:18.3054159Z + [[ distributed == *cpu* ]] 2024-12-18T03:55:18.3054410Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T03:55:18.3099786Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-12-18T03:55:18.3100111Z + [[ linux-focal-rocm6.2-py3.10 == *-bazel-* ]] 2024-12-18T03:55:18.3104003Z + cd test 2024-12-18T03:55:18.3104789Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T03:55:20.5283085Z PyTorch built with: 2024-12-18T03:55:20.5283710Z - GCC 9.4 2024-12-18T03:55:20.5284100Z - C++ Version: 201703 2024-12-18T03:55:20.5284654Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T03:55:20.5285801Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T03:55:20.5286224Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T03:55:20.5286544Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T03:55:20.5286850Z - NNPACK is enabled 2024-12-18T03:55:20.5287100Z - CPU capability usage: AVX2 2024-12-18T03:55:20.5287365Z - HIP Runtime 6.2.41134 2024-12-18T03:55:20.5287620Z - MIOpen 3.2.0 2024-12-18T03:55:20.5287831Z - Magma 2.7.2 2024-12-18T03:55:20.5291971Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T03:55:20.5296304Z 2024-12-18T03:55:21.0465273Z + cd test 2024-12-18T03:55:21.0466075Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T03:55:22.1782426Z ATen/Parallel: 2024-12-18T03:55:23.7506651Z at::get_num_threads() : 64 2024-12-18T03:55:23.7507211Z at::get_num_interop_threads() : 64 2024-12-18T03:55:23.7507720Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T03:55:23.7508247Z omp_get_max_threads() : 64 2024-12-18T03:55:23.7509198Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T03:55:23.7510247Z mkl_get_max_threads() : 64 2024-12-18T03:55:23.7510937Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T03:55:23.7512577Z std::thread::hardware_concurrency() : 64 2024-12-18T03:55:23.7513134Z Environment variables: 2024-12-18T03:55:23.7513593Z OMP_NUM_THREADS : [not set] 2024-12-18T03:55:23.7514156Z MKL_NUM_THREADS : [not set] 2024-12-18T03:55:23.7514733Z ATen parallel backend: OpenMP 2024-12-18T03:55:23.7515116Z 2024-12-18T03:55:23.7515349Z + [[ distributed == *numpy_2* ]] 2024-12-18T03:55:23.7516038Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-12-18T03:55:23.7516750Z + [[ distributed == *backward* ]] 2024-12-18T03:55:23.7517260Z + [[ distributed == *xla* ]] 2024-12-18T03:55:23.7517742Z + [[ distributed == *executorch* ]] 2024-12-18T03:55:23.7518308Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T03:55:23.7518932Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-12-18T03:55:23.7519703Z + [[ distributed == distributed ]] 2024-12-18T03:55:23.7520209Z + test_distributed 2024-12-18T03:55:23.7520676Z + echo 'Testing distributed python tests' 2024-12-18T03:55:23.7521262Z Testing distributed python tests 2024-12-18T03:55:23.7523122Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-12-18T03:55:23.8375559Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T03:55:23.8377142Z import pkg_resources 2024-12-18T03:55:28.7711917Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T03:55:29.2825364Z Ignoring disabled issues: [''] 2024-12-18T03:55:29.3115292Z Found test times from artifacts 2024-12-18T03:55:29.3709879Z Found test times from artifacts 2024-12-18T03:55:29.3725409Z Running all tests 2024-12-18T03:55:29.3728819Z Running parallel tests on 2 processes 2024-12-18T03:55:29.3729189Z Name: tests to run (est. time: 31.51min) 2024-12-18T03:55:29.3729488Z Serial tests (4): 2024-12-18T03:55:29.3729741Z distributed/test_c10d_common 1/1 2024-12-18T03:55:29.3730041Z distributed/test_c10d_nccl 1/3 2024-12-18T03:55:29.3730344Z distributed/test_c10d_nccl 2/3 2024-12-18T03:55:29.3730617Z distributed/test_c10d_nccl 3/3 2024-12-18T03:55:29.3730882Z Parallel tests (0): 2024-12-18T03:55:29.3731136Z Name: excluded (est. time: 0.0min) 2024-12-18T03:55:29.3731397Z Serial tests (0): 2024-12-18T03:55:29.3731622Z Parallel tests (0): 2024-12-18T03:55:29.3785329Z Running distributed/test_c10d_common 1/1 ... [2024-12-18 03:55:29.378103] 2024-12-18T03:55:29.3786039Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:55:29.3789070Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:55:29.378361] 2024-12-18T03:59:17.7344152Z 2024-12-18T03:59:17.7345391Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_f60c5d46ad03e9a5_.log 2024-12-18T03:59:17.7354170Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-12-18T03:59:17.7365778Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-12-18T03:59:17.7366600Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-12-18T03:59:17.7367304Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-12-18T03:59:17.7367983Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-12-18T03:59:17.7368668Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-12-18T03:59:17.7369261Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-12-18T03:59:17.7369858Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-12-18T03:59:17.7370532Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-12-18T03:59:17.7371183Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-12-18T03:59:17.7371830Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-12-18T03:59:17.7372573Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-12-18T03:59:17.7373333Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-12-18T03:59:17.7373989Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-12-18T03:59:17.7374731Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T03:59:17.7375600Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T03:59:17.7376332Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-12-18T03:59:17.7376904Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-12-18T03:59:17.7377597Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-12-18T03:59:17.7379907Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-12-18T03:59:17.7380500Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-12-18T03:59:17.7381086Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-12-18T03:59:17.7381669Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-12-18T03:59:17.7382232Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-12-18T03:59:17.7382557Z 2024-12-18T03:59:17.7382721Z Running distributed/test_c10d_nccl 1/3 ... [2024-12-18 03:59:17.734598] 2024-12-18T03:59:17.7383035Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:59:17.7383809Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:59:17.734869] 2024-12-18T04:16:43.9599002Z 2024-12-18T04:16:43.9600383Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_1943fc01f355fcb8_.log 2024-12-18T04:16:43.9626055Z Running 82 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-12-18T04:16:43.9648598Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-12-18T04:16:43.9649323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-12-18T04:16:43.9649971Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-12-18T04:16:43.9650596Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-12-18T04:16:43.9651219Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-12-18T04:16:43.9651882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False 2024-12-18T04:16:43.9652535Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True 2024-12-18T04:16:43.9653163Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-12-18T04:16:43.9653838Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2024-12-18T04:16:43.9654503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-12-18T04:16:43.9655168Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-12-18T04:16:43.9655826Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-12-18T04:16:43.9656449Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2 2024-12-18T04:16:43.9657092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True 2024-12-18T04:16:43.9657723Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-12-18T04:16:43.9658377Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-12-18T04:16:43.9659064Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-12-18T04:16:43.9659716Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-12-18T04:16:43.9660540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-12-18T04:16:43.9661309Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-12-18T04:16:43.9662101Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-12-18T04:16:43.9662911Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-12-18T04:16:43.9663624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-12-18T04:16:43.9664266Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-12-18T04:16:43.9664975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-12-18T04:16:43.9665721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-12-18T04:16:43.9666475Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-12-18T04:16:43.9667378Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-12-18T04:16:43.9669842Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-12-18T04:16:43.9670575Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-12-18T04:16:43.9671228Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-12-18T04:16:43.9671886Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-12-18T04:16:43.9672586Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-12-18T04:16:43.9673320Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-12-18T04:16:43.9673980Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-12-18T04:16:43.9674702Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-12-18T04:16:43.9675506Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-12-18T04:16:43.9676247Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-12-18T04:16:43.9676879Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-12-18T04:16:43.9677500Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-12-18T04:16:43.9678174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-12-18T04:16:43.9678830Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-12-18T04:16:43.9679490Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-12-18T04:16:43.9680305Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-12-18T04:16:43.9680878Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-12-18T04:16:43.9681470Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result 2024-12-18T04:16:43.9682088Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-12-18T04:16:43.9682753Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-12-18T04:16:43.9683408Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-12-18T04:16:43.9684069Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier 2024-12-18T04:16:43.9684723Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-12-18T04:16:43.9685305Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-12-18T04:16:43.9685857Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-12-18T04:16:43.9686449Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-12-18T04:16:43.9687177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-12-18T04:16:43.9687797Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-12-18T04:16:43.9688330Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor 2024-12-18T04:16:43.9688953Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-12-18T04:16:43.9689754Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-12-18T04:16:43.9690613Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-12-18T04:16:43.9691389Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True 2024-12-18T04:16:43.9692097Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False 2024-12-18T04:16:43.9692848Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True 2024-12-18T04:16:43.9693532Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True 2024-12-18T04:16:43.9694134Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False 2024-12-18T04:16:43.9694779Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False 2024-12-18T04:16:43.9695423Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True 2024-12-18T04:16:43.9696107Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False 2024-12-18T04:16:43.9696850Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False 2024-12-18T04:16:43.9697697Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False 2024-12-18T04:16:43.9698399Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False 2024-12-18T04:16:43.9699033Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-12-18T04:16:43.9699571Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-12-18T04:16:43.9700178Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-12-18T04:16:43.9700765Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-12-18T04:16:43.9701373Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-12-18T04:16:43.9702125Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-12-18T04:16:43.9702868Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-12-18T04:16:43.9703599Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-12-18T04:16:43.9704429Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-12-18T04:16:43.9705092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-12-18T04:16:43.9705757Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-12-18T04:16:43.9706159Z 2024-12-18T04:16:43.9706327Z Running distributed/test_c10d_nccl 2/3 ... [2024-12-18 04:16:43.960890] 2024-12-18T04:16:43.9706641Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:16:43.9707407Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:16:43.961156] 2024-12-18T04:31:25.9504785Z 2024-12-18T04:31:25.9510276Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_82359ad3dd64b235_.log 2024-12-18T04:31:25.9541838Z Running 74 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-12-18T04:31:25.9578280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-12-18T04:31:25.9579040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-12-18T04:31:25.9579802Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-12-18T04:31:25.9580599Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-12-18T04:31:25.9581445Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2024-12-18T04:31:25.9582260Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-12-18T04:31:25.9583009Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-12-18T04:31:25.9583800Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn 2024-12-18T04:31:25.9584555Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check 2024-12-18T04:31:25.9585276Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter 2024-12-18T04:31:25.9586051Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False 2024-12-18T04:31:25.9586894Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p 2024-12-18T04:31:25.9588547Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-12-18T04:31:25.9590112Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-12-18T04:31:25.9591996Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True 2024-12-18T04:31:25.9593744Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-12-18T04:31:25.9595529Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-12-18T04:31:25.9597186Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-12-18T04:31:25.9598903Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-12-18T04:31:25.9600774Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-12-18T04:31:25.9602593Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-12-18T04:31:25.9604503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-12-18T04:31:25.9606424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-12-18T04:31:25.9608528Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-12-18T04:31:25.9610005Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-12-18T04:31:25.9610910Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-12-18T04:31:25.9611739Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-12-18T04:31:25.9612542Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-12-18T04:31:25.9613433Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-12-18T04:31:25.9614318Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-12-18T04:31:25.9615153Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-12-18T04:31:25.9616002Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-12-18T04:31:25.9616954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-12-18T04:31:25.9617955Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-12-18T04:31:25.9618946Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-12-18T04:31:25.9619808Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-12-18T04:31:25.9620657Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-12-18T04:31:25.9621482Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-12-18T04:31:25.9622245Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-12-18T04:31:25.9623199Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-12-18T04:31:25.9623961Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-12-18T04:31:25.9624685Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-12-18T04:31:25.9625384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-12-18T04:31:25.9626063Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-12-18T04:31:25.9626731Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-12-18T04:31:25.9627889Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-12-18T04:31:25.9629313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-12-18T04:31:25.9630632Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-12-18T04:31:25.9631992Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-12-18T04:31:25.9633365Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-12-18T04:31:25.9634703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-12-18T04:31:25.9636243Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited 2024-12-18T04:31:25.9637713Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-12-18T04:31:25.9639518Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False 2024-12-18T04:31:25.9641220Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True 2024-12-18T04:31:25.9642873Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False 2024-12-18T04:31:25.9644367Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True 2024-12-18T04:31:25.9645892Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False 2024-12-18T04:31:25.9647622Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False 2024-12-18T04:31:25.9648819Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-12-18T04:31:25.9649592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True 2024-12-18T04:31:25.9650473Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True 2024-12-18T04:31:25.9651385Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True 2024-12-18T04:31:25.9652266Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True 2024-12-18T04:31:25.9653170Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-12-18T04:31:25.9654107Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-12-18T04:31:25.9655205Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-12-18T04:31:25.9656116Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-12-18T04:31:25.9657079Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-12-18T04:31:25.9658106Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-12-18T04:31:25.9659018Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-12-18T04:31:25.9659939Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-12-18T04:31:25.9660855Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-12-18T04:31:25.9661738Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-12-18T04:31:25.9662217Z 2024-12-18T04:31:25.9662414Z Running distributed/test_c10d_nccl 3/3 ... [2024-12-18 04:31:25.951734] 2024-12-18T04:31:25.9662804Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:31:25.9663903Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:31:25.952362] 2024-12-18T04:44:28.3164389Z 2024-12-18T04:44:28.3166339Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_cbf0c4915ceaa63c_.log 2024-12-18T04:44:28.3206981Z Running 64 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T04:44:28.3242885Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str 2024-12-18T04:44:28.3244520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-12-18T04:44:28.3246230Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup 2024-12-18T04:44:28.3247775Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context 2024-12-18T04:44:28.3249200Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-12-18T04:44:28.3250606Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-12-18T04:44:28.3252099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-12-18T04:44:28.3253586Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-12-18T04:44:28.3254523Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg 2024-12-18T04:44:28.3255335Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-12-18T04:44:28.3256186Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False 2024-12-18T04:44:28.3257123Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-12-18T04:44:28.3258021Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-12-18T04:44:28.3258837Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-12-18T04:44:28.3259662Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-12-18T04:44:28.3260582Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T04:44:28.3261577Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-12-18T04:44:28.3263043Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T04:44:28.3264968Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T04:44:28.3266803Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-12-18T04:44:28.3268682Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T04:44:28.3270633Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T04:44:28.3272462Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-12-18T04:44:28.3274391Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-12-18T04:44:28.3275865Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-12-18T04:44:28.3277575Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-12-18T04:44:28.3279375Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-12-18T04:44:28.3281258Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-12-18T04:44:28.3283130Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-12-18T04:44:28.3284870Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T04:44:28.3286533Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-12-18T04:44:28.3288168Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-12-18T04:44:28.3289697Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-12-18T04:44:28.3291400Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-12-18T04:44:28.3293041Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration 2024-12-18T04:44:28.3294088Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-12-18T04:44:28.3294830Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-12-18T04:44:28.3295593Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-12-18T04:44:28.3296323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-12-18T04:44:28.3297088Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-12-18T04:44:28.3297859Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-12-18T04:44:28.3298555Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-12-18T04:44:28.3299216Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-12-18T04:44:28.3300033Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-12-18T04:44:28.3301041Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-12-18T04:44:28.3302054Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-12-18T04:44:28.3303971Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T04:44:28.3305963Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False 2024-12-18T04:44:28.3307655Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True 2024-12-18T04:44:28.3309486Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-12-18T04:44:28.3310957Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-12-18T04:44:28.3312402Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True 2024-12-18T04:44:28.3313862Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False 2024-12-18T04:44:28.3315447Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True 2024-12-18T04:44:28.3316940Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling 2024-12-18T04:44:28.3318491Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-12-18T04:44:28.3320412Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-12-18T04:44:28.3322106Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-12-18T04:44:28.3323805Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-12-18T04:44:28.3325895Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-12-18T04:44:28.3327688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-12-18T04:44:28.3329450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-12-18T04:44:28.3331101Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-12-18T04:44:28.3332630Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T04:44:28.3333471Z 2024-12-18T04:44:29.2968672Z Running test batch 'tests to run' cost 2939.92 seconds 2024-12-18T04:44:30.0820969Z 2024-12-18T04:44:30.0821495Z real 49m6.331s 2024-12-18T04:44:30.0821911Z user 66m19.666s 2024-12-18T04:44:30.0822301Z sys 67m49.681s 2024-12-18T04:44:30.0822677Z + assert_git_not_dirty 2024-12-18T04:44:30.0823141Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-12-18T04:44:30.0823668Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T04:44:30.0824163Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T04:44:30.0824609Z + [[ 1 == 1 ]] 2024-12-18T04:44:30.0824989Z + echo 'Testing distributed C++ tests' 2024-12-18T04:44:30.0825457Z Testing distributed C++ tests 2024-12-18T04:44:30.0828831Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:44:30.0852969Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:44:30.0869837Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:44:30.0870693Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:44:30.0871108Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-12-18T04:44:30.1712536Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T04:44:30.1714049Z import pkg_resources 2024-12-18T04:44:35.0476044Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T04:44:35.0590710Z Found test times from artifacts 2024-12-18T04:44:35.1075348Z Found test times from artifacts 2024-12-18T04:44:35.1090607Z Running all tests 2024-12-18T04:44:35.1096251Z Running parallel tests on 2 processes 2024-12-18T04:44:35.1096947Z Name: tests to run (est. time: 0.0min) 2024-12-18T04:44:35.1097541Z Serial tests (1): 2024-12-18T04:44:35.1098009Z cpp/FileStoreTest 1/1 2024-12-18T04:44:35.1098533Z Parallel tests (0): 2024-12-18T04:44:35.1099027Z Name: excluded (est. time: 0.0min) 2024-12-18T04:44:35.1099537Z Serial tests (0): 2024-12-18T04:44:35.1099963Z Parallel tests (0): 2024-12-18T04:44:35.1152864Z Running cpp/FileStoreTest 1/1 ... [2024-12-18 04:44:35.114792] 2024-12-18T04:44:35.1153643Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:44:35.1161195Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-bdda4a36f8eaeb90.xml', '-x', '--reruns=2'] ... [2024-12-18 04:44:35.115284] 2024-12-18T04:44:37.5854723Z 2024-12-18T04:44:37.5856426Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_5133f0d3b8cd77a9_.log 2024-12-18T04:44:37.5857605Z 2024-12-18T04:44:39.8849550Z Running test batch 'tests to run' cost 4.77 seconds 2024-12-18T04:44:40.6698884Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-12-18T04:44:40.7555274Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T04:44:40.7556056Z import pkg_resources 2024-12-18T04:44:45.6487962Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T04:44:45.6600935Z Found test times from artifacts 2024-12-18T04:44:45.7077070Z Found test times from artifacts 2024-12-18T04:44:45.7093734Z Running all tests 2024-12-18T04:44:45.7094836Z Running parallel tests on 2 processes 2024-12-18T04:44:45.7095564Z Name: tests to run (est. time: 0.0min) 2024-12-18T04:44:45.7096107Z Serial tests (1): 2024-12-18T04:44:45.7096620Z cpp/HashStoreTest 1/1 2024-12-18T04:44:45.7097095Z Parallel tests (0): 2024-12-18T04:44:45.7097571Z Name: excluded (est. time: 0.0min) 2024-12-18T04:44:45.7098086Z Serial tests (0): 2024-12-18T04:44:45.7098506Z Parallel tests (0): 2024-12-18T04:44:45.7151193Z Running cpp/HashStoreTest 1/1 ... [2024-12-18 04:44:45.714471] 2024-12-18T04:44:45.7152048Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:44:45.7154863Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-3500dd0f740de994.xml', '-x', '--reruns=2'] ... [2024-12-18 04:44:45.714877] 2024-12-18T04:44:48.0850669Z 2024-12-18T04:44:48.0852438Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_d00300fe35249b02_.log 2024-12-18T04:44:48.0853715Z 2024-12-18T04:44:50.5890954Z Running test batch 'tests to run' cost 4.88 seconds 2024-12-18T04:44:51.3830015Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-12-18T04:44:51.4682983Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T04:44:51.4684538Z import pkg_resources 2024-12-18T04:44:56.3264672Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T04:44:56.3378009Z Found test times from artifacts 2024-12-18T04:44:56.3859558Z Found test times from artifacts 2024-12-18T04:44:56.3874948Z Running all tests 2024-12-18T04:44:56.3877713Z Running parallel tests on 2 processes 2024-12-18T04:44:56.3878330Z Name: tests to run (est. time: 0.0min) 2024-12-18T04:44:56.3878875Z Serial tests (1): 2024-12-18T04:44:56.3879327Z cpp/TCPStoreTest 1/1 2024-12-18T04:44:56.3879938Z Parallel tests (0): 2024-12-18T04:44:56.3880442Z Name: excluded (est. time: 0.0min) 2024-12-18T04:44:56.3880952Z Serial tests (0): 2024-12-18T04:44:56.3881361Z Parallel tests (0): 2024-12-18T04:44:56.3929854Z Running cpp/TCPStoreTest 1/1 ... [2024-12-18 04:44:56.392606] 2024-12-18T04:44:56.3930324Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:44:56.3933572Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-896389f33bf4a874.xml', '-x', '--reruns=2'] ... [2024-12-18 04:44:56.392998] 2024-12-18T04:45:10.3108443Z 2024-12-18T04:45:10.3110715Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_0cc12500d5d2ad11_.log 2024-12-18T04:45:10.3111959Z 2024-12-18T04:45:11.1897188Z Running test batch 'tests to run' cost 14.8 seconds 2024-12-18T04:45:11.9729875Z + echo 'Testing multi-GPU linalg tests' 2024-12-18T04:45:11.9730594Z Testing multi-GPU linalg tests 2024-12-18T04:45:11.9731530Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose 2024-12-18T04:45:12.0589978Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T04:45:12.0591510Z import pkg_resources 2024-12-18T04:45:16.9189255Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T04:45:16.9303120Z Found test times from artifacts 2024-12-18T04:45:16.9784552Z Found test times from artifacts 2024-12-18T04:45:16.9798591Z Running all tests 2024-12-18T04:45:16.9800788Z Running parallel tests on 2 processes 2024-12-18T04:45:16.9801444Z Name: tests to run (est. time: 0.0min) 2024-12-18T04:45:16.9801988Z Serial tests (1): 2024-12-18T04:45:16.9802508Z test_linalg 1/1 2024-12-18T04:45:16.9802968Z Parallel tests (0): 2024-12-18T04:45:16.9803463Z Name: excluded (est. time: 0.0min) 2024-12-18T04:45:16.9803971Z Serial tests (0): 2024-12-18T04:45:16.9804390Z Parallel tests (0): 2024-12-18T04:45:16.9856526Z Running test_linalg 1/1 ... [2024-12-18 04:45:16.985240] 2024-12-18T04:45:16.9857251Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:45:16.9859741Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:45:16.985526] 2024-12-18T04:45:22.8126067Z 2024-12-18T04:45:22.8127324Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_c53e125c729246b0_.log 2024-12-18T04:45:22.8129089Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32 2024-12-18T04:45:22.8131085Z 2024-12-18T04:45:23.6338238Z Running test batch 'tests to run' cost 6.65 seconds 2024-12-18T04:45:24.4374545Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T04:45:24.4375215Z + [[ 1 == 1 ]] 2024-12-18T04:45:24.4375617Z + test_rpc 2024-12-18T04:45:24.4376046Z + echo 'Testing RPC C++ tests' 2024-12-18T04:45:24.4376580Z Testing RPC C++ tests 2024-12-18T04:45:24.4380655Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:45:24.4416615Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:45:24.4450358Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T04:45:24.4451459Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-12-18T04:45:24.5285884Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T04:45:24.5287373Z import pkg_resources 2024-12-18T04:45:29.4142312Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T04:45:29.4255217Z Found test times from artifacts 2024-12-18T04:45:29.4736121Z Found test times from artifacts 2024-12-18T04:45:29.4750153Z Running all tests 2024-12-18T04:45:29.4751775Z Running parallel tests on 2 processes 2024-12-18T04:45:29.4752408Z Name: tests to run (est. time: 0.0min) 2024-12-18T04:45:29.4752968Z Serial tests (1): 2024-12-18T04:45:29.4753467Z cpp/test_cpp_rpc 1/1 2024-12-18T04:45:29.4754030Z Parallel tests (0): 2024-12-18T04:45:29.4754500Z Name: excluded (est. time: 0.0min) 2024-12-18T04:45:29.4755002Z Serial tests (0): 2024-12-18T04:45:29.4755414Z Parallel tests (0): 2024-12-18T04:45:29.4807288Z Running cpp/test_cpp_rpc 1/1 ... [2024-12-18 04:45:29.480339] 2024-12-18T04:45:29.4807755Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:45:29.4810399Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-43f27a049ffed665.xml', '-x', '--reruns=2'] ... [2024-12-18 04:45:29.480738] 2024-12-18T04:45:38.8154346Z 2024-12-18T04:45:38.8156205Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_4244dfd40961d921_.log 2024-12-18T04:45:38.8157385Z 2024-12-18T04:45:39.6385133Z Running test batch 'tests to run' cost 10.16 seconds 2024-12-18T04:45:40.4587305Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-12-18T04:45:40.4588126Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-12-18T04:45:40.4589032Z docker exec -t "548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-12-18T04:45:40.4712125Z shell: /usr/bin/bash -e {0} 2024-12-18T04:45:40.4712653Z env: 2024-12-18T04:45:40.4713257Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:40.4713838Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:40.4714841Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:40.4716240Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:40.4716738Z AWS_REGION: us-east-1 2024-12-18T04:45:40.4717478Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:40.4718187Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:40.4728533Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:40.4729435Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:40.4730386Z ##[endgroup] 2024-12-18T04:45:40.5229075Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T04:45:40.5445238Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T04:45:40.5446019Z cat test/**/*_toprint.log || true 2024-12-18T04:45:40.5490347Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T04:45:40.5491087Z env: 2024-12-18T04:45:40.5491339Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:40.5491683Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:40.5492192Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:40.5492693Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:40.5492954Z AWS_REGION: us-east-1 2024-12-18T04:45:40.5493266Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:40.5493605Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:40.5498636Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:40.5499017Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:40.5499437Z ##[endgroup] 2024-12-18T04:45:40.5635516Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T04:45:40.5735679Z Prepare all required actions 2024-12-18T04:45:40.5736084Z Getting action download info 2024-12-18T04:45:41.1315605Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-18T04:45:42.0367669Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T04:45:42.0367940Z with: 2024-12-18T04:45:42.0368105Z use-gha: true 2024-12-18T04:45:42.0368390Z file-suffix: test-distributed-1-1-linux.rocm.gpu_34566663857 2024-12-18T04:45:42.0368688Z s3-bucket: gha-artifacts 2024-12-18T04:45:42.0368888Z env: 2024-12-18T04:45:42.0369051Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:42.0369275Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:42.0369681Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:42.0370062Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:42.0370272Z AWS_REGION: us-east-1 2024-12-18T04:45:42.0370542Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:42.0370822Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:42.0374839Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:42.0375147Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:42.0375506Z ##[endgroup] 2024-12-18T04:45:42.0434383Z ##[group]Run actions/upload-artifact@v4 2024-12-18T04:45:42.0434637Z with: 2024-12-18T04:45:42.0434968Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip 2024-12-18T04:45:42.0435348Z retention-days: 14 2024-12-18T04:45:42.0435552Z if-no-files-found: warn 2024-12-18T04:45:42.0435748Z path: test/**/*.json 2024-12-18T04:45:42.0435933Z compression-level: 6 2024-12-18T04:45:42.0436121Z overwrite: false 2024-12-18T04:45:42.0436311Z include-hidden-files: false 2024-12-18T04:45:42.0436521Z env: 2024-12-18T04:45:42.0436684Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:42.0436920Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:42.0437324Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:42.0437702Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:42.0437903Z AWS_REGION: us-east-1 2024-12-18T04:45:42.0438117Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:42.0438392Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:42.0442504Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:42.0442815Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:42.0443375Z ##[endgroup] 2024-12-18T04:45:42.6381004Z With the provided path, there will be 11 files uploaded 2024-12-18T04:45:42.6386338Z Artifact name is valid! 2024-12-18T04:45:42.6387035Z Root directory input is valid! 2024-12-18T04:45:44.2129572Z Beginning upload of artifact content to blob storage 2024-12-18T04:45:44.7127326Z Uploaded bytes 44442 2024-12-18T04:45:44.7961948Z Finished uploading artifact content to blob storage! 2024-12-18T04:45:44.7967523Z SHA256 hash of uploaded artifact zip is ca2a0564ac124ad05238b8a36dbb8bbc5955efe30f0a051c79e036351239dc10 2024-12-18T04:45:44.7970250Z Finalizing artifact upload 2024-12-18T04:45:44.9370901Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip.zip successfully finalized. Artifact ID 2335744262 2024-12-18T04:45:44.9373106Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip has been successfully uploaded! Final size is 44442 bytes. Artifact ID is 2335744262 2024-12-18T04:45:44.9387462Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255596/artifacts/2335744262 2024-12-18T04:45:44.9718522Z ##[group]Run actions/upload-artifact@v4 2024-12-18T04:45:44.9719095Z with: 2024-12-18T04:45:44.9720016Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip 2024-12-18T04:45:44.9720929Z retention-days: 14 2024-12-18T04:45:44.9721393Z if-no-files-found: ignore 2024-12-18T04:45:44.9721913Z path: test/**/*.xml test/**/*.csv 2024-12-18T04:45:44.9722918Z compression-level: 6 2024-12-18T04:45:44.9723386Z overwrite: false 2024-12-18T04:45:44.9723840Z include-hidden-files: false 2024-12-18T04:45:44.9724324Z env: 2024-12-18T04:45:44.9724699Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:44.9725241Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:44.9726218Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:44.9727148Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:44.9727650Z AWS_REGION: us-east-1 2024-12-18T04:45:44.9728213Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:44.9728868Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:44.9738977Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:44.9739724Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:44.9740528Z ##[endgroup] 2024-12-18T04:45:45.5834195Z With the provided path, there will be 250 files uploaded 2024-12-18T04:45:45.5838902Z Artifact name is valid! 2024-12-18T04:45:45.5839565Z Root directory input is valid! 2024-12-18T04:45:47.1009567Z Beginning upload of artifact content to blob storage 2024-12-18T04:45:47.7854718Z Uploaded bytes 132064 2024-12-18T04:45:47.8691708Z Finished uploading artifact content to blob storage! 2024-12-18T04:45:47.8697525Z SHA256 hash of uploaded artifact zip is 87b6aa8e96bc35599c1646eb85e431e0fb8ad4443db3f35145876afa8e1d7f48 2024-12-18T04:45:47.8700281Z Finalizing artifact upload 2024-12-18T04:45:48.0171600Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip.zip successfully finalized. Artifact ID 2335744357 2024-12-18T04:45:48.0173893Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip has been successfully uploaded! Final size is 132064 bytes. Artifact ID is 2335744357 2024-12-18T04:45:48.0185494Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255596/artifacts/2335744357 2024-12-18T04:45:48.0532942Z ##[group]Run actions/upload-artifact@v4 2024-12-18T04:45:48.0533537Z with: 2024-12-18T04:45:48.0534217Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip 2024-12-18T04:45:48.0535034Z retention-days: 14 2024-12-18T04:45:48.0535487Z if-no-files-found: ignore 2024-12-18T04:45:48.0535988Z path: usage_log.txt test/**/*.log 2024-12-18T04:45:48.0536517Z compression-level: 6 2024-12-18T04:45:48.0536953Z overwrite: false 2024-12-18T04:45:48.0537889Z include-hidden-files: false 2024-12-18T04:45:48.0538366Z env: 2024-12-18T04:45:48.0538742Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:48.0539282Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:48.0540254Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:48.0541156Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:48.0541647Z AWS_REGION: us-east-1 2024-12-18T04:45:48.0542200Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:48.0542846Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:48.0552650Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:48.0553395Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:48.0554201Z ##[endgroup] 2024-12-18T04:45:48.6853835Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-12-18T04:45:48.6856110Z The least common ancestor is /home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-12-18T04:45:48.6857671Z With the provided path, there will be 9 files uploaded 2024-12-18T04:45:48.6859848Z Artifact name is valid! 2024-12-18T04:45:48.6860918Z Root directory input is valid! 2024-12-18T04:45:50.2931159Z Beginning upload of artifact content to blob storage 2024-12-18T04:45:50.7451396Z Uploaded bytes 29867 2024-12-18T04:45:50.8330352Z Finished uploading artifact content to blob storage! 2024-12-18T04:45:50.8335848Z SHA256 hash of uploaded artifact zip is f1fb7df57c66abc497c66bbdee7e082c9f93110f41668c0d065f1a94a85669e1 2024-12-18T04:45:50.8339367Z Finalizing artifact upload 2024-12-18T04:45:50.9721330Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip.zip successfully finalized. Artifact ID 2335744453 2024-12-18T04:45:50.9723418Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_34566663857.zip has been successfully uploaded! Final size is 29867 bytes. Artifact ID is 2335744453 2024-12-18T04:45:50.9734696Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255596/artifacts/2335744453 2024-12-18T04:45:51.0056746Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T04:45:51.0057447Z # shellcheck disable=SC2156 2024-12-18T04:45:51.0058431Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T04:45:51.0115780Z shell: /usr/bin/bash -e {0} 2024-12-18T04:45:51.0116293Z env: 2024-12-18T04:45:51.0116688Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:51.0117262Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:51.0118244Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:51.0119156Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:51.0119781Z AWS_REGION: us-east-1 2024-12-18T04:45:51.0120361Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:51.0121016Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:51.0130865Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:51.0131716Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:51.0132535Z ##[endgroup] 2024-12-18T04:45:51.3101789Z Prepare all required actions 2024-12-18T04:45:51.3102705Z Getting action download info 2024-12-18T04:45:51.3149051Z ##[group]Run ./.github/actions/teardown-rocm 2024-12-18T04:45:51.3149626Z env: 2024-12-18T04:45:51.3150018Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:51.3150568Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:51.3151544Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:51.3152466Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:51.3152962Z AWS_REGION: us-east-1 2024-12-18T04:45:51.3153523Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:51.3154183Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:51.3164043Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:51.3164788Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:51.3166062Z ##[endgroup] 2024-12-18T04:45:51.3190686Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T04:45:51.3191666Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T04:45:51.3192406Z # shellcheck disable=SC2046 2024-12-18T04:45:51.3193007Z docker stop $(docker ps -q) || true 2024-12-18T04:45:51.3193614Z # Prune all stopped containers. 2024-12-18T04:45:51.3194196Z docker container prune -f 2024-12-18T04:45:51.3240951Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T04:45:51.3241615Z env: 2024-12-18T04:45:51.3242008Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:45:51.3242537Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:45:51.3243480Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:45:51.3244354Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:45:51.3244836Z AWS_REGION: us-east-1 2024-12-18T04:45:51.3245378Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:45:51.3246026Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:45:51.3255967Z AWS_SESSION_TOKEN: *** 2024-12-18T04:45:51.3256348Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:45:51.3256771Z ##[endgroup] 2024-12-18T04:45:51.7829717Z 548d3077c0cd 2024-12-18T04:46:09.1085518Z Deleted Containers: 2024-12-18T04:46:09.1086400Z 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:46:09.1087101Z 2024-12-18T04:46:09.1087400Z Total reclaimed space: 7.125GB 2024-12-18T04:46:09.1165953Z Prepare all required actions 2024-12-18T04:46:09.1220699Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-12-18T04:46:09.1221291Z with: 2024-12-18T04:46:09.1221691Z diskspace-cutoff: 70 2024-12-18T04:46:09.1222113Z env: 2024-12-18T04:46:09.1222481Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:46:09.1223015Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:46:09.1223982Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:46:09.1224943Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:46:09.1225435Z AWS_REGION: us-east-1 2024-12-18T04:46:09.1225981Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:46:09.1226643Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:46:09.1236445Z AWS_SESSION_TOKEN: *** 2024-12-18T04:46:09.1237189Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:46:09.1237981Z ##[endgroup] 2024-12-18T04:46:09.1267938Z ##[group]Run set -ex 2024-12-18T04:46:09.1268495Z set -ex 2024-12-18T04:46:09.1268933Z diskspace_cutoff=70 2024-12-18T04:46:09.1269581Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-12-18T04:46:09.1270614Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T04:46:09.1272157Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T04:46:09.1273914Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-12-18T04:46:09.1274621Z  docker system prune -af 2024-12-18T04:46:09.1275553Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T04:46:09.1276587Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-12-18T04:46:09.1277637Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-12-18T04:46:09.1278552Z  echo "$msg" 2024-12-18T04:46:09.1279022Z  exit 1 2024-12-18T04:46:09.1279590Z  else 2024-12-18T04:46:09.1280114Z  difference=$((diskspace - diskspace_new)) 2024-12-18T04:46:09.1280864Z  echo "Diskspace saved: $difference percent" 2024-12-18T04:46:09.1281564Z  fi 2024-12-18T04:46:09.1282532Z fi 2024-12-18T04:46:09.1333138Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T04:46:09.1333875Z env: 2024-12-18T04:46:09.1334254Z GIT_DEFAULT_BRANCH: main 2024-12-18T04:46:09.1334791Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-12-18T04:46:09.1335739Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T04:46:09.1336654Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T04:46:09.1337153Z AWS_REGION: us-east-1 2024-12-18T04:46:09.1337690Z AWS_ACCESS_KEY_ID: *** 2024-12-18T04:46:09.1338332Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T04:46:09.1348659Z AWS_SESSION_TOKEN: *** 2024-12-18T04:46:09.1349400Z CONTAINER_NAME: 548d3077c0cd4508397dff76211f99b504061a7fc3c815805dfd74a8a233ab28 2024-12-18T04:46:09.1350201Z ##[endgroup] 2024-12-18T04:46:09.1428548Z + diskspace_cutoff=70 2024-12-18T04:46:09.1434420Z ++ docker info -f '{{.DockerRootDir}}' 2024-12-18T04:46:09.3131631Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-12-18T04:46:09.3138035Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-12-18T04:46:09.3140567Z ++ sed -n 2p 2024-12-18T04:46:09.3142988Z ++ sed s/%// 2024-12-18T04:46:09.3143309Z ++ sed 's/ //' 2024-12-18T04:46:09.3169591Z + diskspace=19 2024-12-18T04:46:09.3170155Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-12-18T04:46:09.3170739Z + [[ 19 -ge 70 ]] 2024-12-18T04:46:09.3214223Z Post job cleanup. 2024-12-18T04:46:09.3241014Z Post job cleanup. 2024-12-18T04:46:09.3598200Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T04:46:09.3891044Z Post job cleanup. 2024-12-18T04:46:09.5103499Z Post job cleanup. 2024-12-18T04:46:09.5194342Z Post job cleanup. 2024-12-18T04:46:09.5270586Z Post job cleanup. 2024-12-18T04:46:09.5941271Z [command]/usr/bin/git version 2024-12-18T04:46:09.5972079Z git version 2.34.1 2024-12-18T04:46:09.6008126Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/68a4d976-6b39-4a81-bcd7-b34dbe8180d2' before making global git config changes 2024-12-18T04:46:09.6008847Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T04:46:09.6011436Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T04:46:09.6036579Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T04:46:09.6060012Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T04:46:09.6378248Z Entering 'android/libs/fbjni' 2024-12-18T04:46:09.6425476Z Entering 'third_party/FP16' 2024-12-18T04:46:09.6478811Z Entering 'third_party/FXdiv' 2024-12-18T04:46:09.6518355Z Entering 'third_party/NNPACK' 2024-12-18T04:46:09.6557631Z Entering 'third_party/NVTX' 2024-12-18T04:46:09.6596183Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T04:46:09.6637941Z Entering 'third_party/XNNPACK' 2024-12-18T04:46:09.6700951Z Entering 'third_party/benchmark' 2024-12-18T04:46:09.6742469Z Entering 'third_party/composable_kernel' 2024-12-18T04:46:09.6791454Z Entering 'third_party/cpp-httplib' 2024-12-18T04:46:09.6833223Z Entering 'third_party/cpuinfo' 2024-12-18T04:46:09.6876546Z Entering 'third_party/cudnn_frontend' 2024-12-18T04:46:09.6918249Z Entering 'third_party/cutlass' 2024-12-18T04:46:09.6964604Z Entering 'third_party/eigen' 2024-12-18T04:46:09.7015484Z Entering 'third_party/fbgemm' 2024-12-18T04:46:09.7057802Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T04:46:09.7110350Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T04:46:09.7156257Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T04:46:09.7206231Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T04:46:09.7252453Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T04:46:09.7300302Z Entering 'third_party/flatbuffers' 2024-12-18T04:46:09.7344292Z Entering 'third_party/fmt' 2024-12-18T04:46:09.7392239Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T04:46:09.7436389Z Entering 'third_party/gloo' 2024-12-18T04:46:09.7478066Z Entering 'third_party/googletest' 2024-12-18T04:46:09.7519162Z Entering 'third_party/ideep' 2024-12-18T04:46:09.7563477Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T04:46:09.7612753Z Entering 'third_party/ittapi' 2024-12-18T04:46:09.7657006Z Entering 'third_party/kineto' 2024-12-18T04:46:09.7704081Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T04:46:09.7750196Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T04:46:09.7796246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T04:46:09.7842603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T04:46:09.7881083Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T04:46:09.7921304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T04:46:09.7964714Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T04:46:09.8013264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T04:46:09.8057298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T04:46:09.8098374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T04:46:09.8146063Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T04:46:09.8191218Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T04:46:09.8237773Z Entering 'third_party/mimalloc' 2024-12-18T04:46:09.8288418Z Entering 'third_party/nccl/nccl' 2024-12-18T04:46:09.8328645Z Entering 'third_party/nlohmann' 2024-12-18T04:46:09.8369323Z Entering 'third_party/onnx' 2024-12-18T04:46:09.8430173Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T04:46:09.8480829Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T04:46:09.8529488Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T04:46:09.8576170Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T04:46:09.8618923Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T04:46:09.8660646Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T04:46:09.8711685Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T04:46:09.8746302Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T04:46:09.8785408Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T04:46:09.8821840Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T04:46:09.8864798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T04:46:09.8915786Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T04:46:09.8976310Z Entering 'third_party/pocketfft' 2024-12-18T04:46:09.9019458Z Entering 'third_party/protobuf' 2024-12-18T04:46:09.9061543Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T04:46:09.9103477Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T04:46:09.9142641Z Entering 'third_party/psimd' 2024-12-18T04:46:09.9181420Z Entering 'third_party/pthreadpool' 2024-12-18T04:46:09.9220197Z Entering 'third_party/pybind11' 2024-12-18T04:46:09.9259426Z Entering 'third_party/python-peachpy' 2024-12-18T04:46:09.9296890Z Entering 'third_party/sleef' 2024-12-18T04:46:09.9334127Z Entering 'third_party/tensorpipe' 2024-12-18T04:46:09.9381543Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T04:46:09.9423094Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T04:46:09.9461289Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T04:46:09.9499396Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T04:46:09.9534835Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T04:46:09.9604031Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T04:46:09.9630590Z http.https://github.com/.extraheader 2024-12-18T04:46:09.9645454Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T04:46:09.9694356Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T04:46:09.9966327Z Entering 'android/libs/fbjni' 2024-12-18T04:46:09.9992759Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0032516Z Entering 'third_party/FP16' 2024-12-18T04:46:10.0056752Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0110008Z Entering 'third_party/FXdiv' 2024-12-18T04:46:10.0136812Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0177812Z Entering 'third_party/NNPACK' 2024-12-18T04:46:10.0198813Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0229377Z Entering 'third_party/NVTX' 2024-12-18T04:46:10.0251122Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0287122Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T04:46:10.0311612Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0361584Z Entering 'third_party/XNNPACK' 2024-12-18T04:46:10.0381719Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0436730Z Entering 'third_party/benchmark' 2024-12-18T04:46:10.0465721Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0503024Z Entering 'third_party/composable_kernel' 2024-12-18T04:46:10.0525638Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0565471Z Entering 'third_party/cpp-httplib' 2024-12-18T04:46:10.0591115Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0617578Z Entering 'third_party/cpuinfo' 2024-12-18T04:46:10.0638799Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0669369Z Entering 'third_party/cudnn_frontend' 2024-12-18T04:46:10.0694149Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0742827Z Entering 'third_party/cutlass' 2024-12-18T04:46:10.0768490Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0810192Z Entering 'third_party/eigen' 2024-12-18T04:46:10.0834840Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0871826Z Entering 'third_party/fbgemm' 2024-12-18T04:46:10.0896452Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0935792Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T04:46:10.0965594Z http.https://github.com/.extraheader 2024-12-18T04:46:10.0995432Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T04:46:10.1020665Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1059126Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T04:46:10.1079784Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1121354Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T04:46:10.1141472Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1181393Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T04:46:10.1204416Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1240101Z Entering 'third_party/flatbuffers' 2024-12-18T04:46:10.1265710Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1301675Z Entering 'third_party/fmt' 2024-12-18T04:46:10.1323114Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1358461Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T04:46:10.1379999Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1411084Z Entering 'third_party/gloo' 2024-12-18T04:46:10.1436694Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1473265Z Entering 'third_party/googletest' 2024-12-18T04:46:10.1497675Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1528241Z Entering 'third_party/ideep' 2024-12-18T04:46:10.1558708Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1596930Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T04:46:10.1623853Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1665551Z Entering 'third_party/ittapi' 2024-12-18T04:46:10.1689567Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1726746Z Entering 'third_party/kineto' 2024-12-18T04:46:10.1753053Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1790506Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T04:46:10.1815067Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1857309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T04:46:10.1876282Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1917316Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T04:46:10.1938096Z http.https://github.com/.extraheader 2024-12-18T04:46:10.1975656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T04:46:10.2003131Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2035098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T04:46:10.2061083Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2095655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T04:46:10.2116904Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2156584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T04:46:10.2178708Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2215050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T04:46:10.2236525Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2266806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T04:46:10.2287216Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2323198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T04:46:10.2344284Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2381808Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T04:46:10.2405231Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2440586Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T04:46:10.2461393Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2512555Z Entering 'third_party/mimalloc' 2024-12-18T04:46:10.2537837Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2573990Z Entering 'third_party/nccl/nccl' 2024-12-18T04:46:10.2599053Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2636305Z Entering 'third_party/nlohmann' 2024-12-18T04:46:10.2661492Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2697784Z Entering 'third_party/onnx' 2024-12-18T04:46:10.2721583Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2768438Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T04:46:10.2789931Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2835717Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T04:46:10.2859065Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2892673Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T04:46:10.2916003Z http.https://github.com/.extraheader 2024-12-18T04:46:10.2953733Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T04:46:10.2976179Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3010441Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T04:46:10.3030416Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3064261Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T04:46:10.3089976Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3127168Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T04:46:10.3149810Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3180680Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T04:46:10.3200751Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3229479Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T04:46:10.3249833Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3278994Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T04:46:10.3304123Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3349360Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T04:46:10.3376470Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3418989Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T04:46:10.3440911Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3493410Z Entering 'third_party/pocketfft' 2024-12-18T04:46:10.3520537Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3550695Z Entering 'third_party/protobuf' 2024-12-18T04:46:10.3571820Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3618723Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T04:46:10.3644342Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3677540Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T04:46:10.3701266Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3744623Z Entering 'third_party/psimd' 2024-12-18T04:46:10.3769575Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3803995Z Entering 'third_party/pthreadpool' 2024-12-18T04:46:10.3825035Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3869198Z Entering 'third_party/pybind11' 2024-12-18T04:46:10.3893995Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3927696Z Entering 'third_party/python-peachpy' 2024-12-18T04:46:10.3949904Z http.https://github.com/.extraheader 2024-12-18T04:46:10.3979842Z Entering 'third_party/sleef' 2024-12-18T04:46:10.4003855Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4037726Z Entering 'third_party/tensorpipe' 2024-12-18T04:46:10.4058929Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4105658Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T04:46:10.4131644Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4163720Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T04:46:10.4184714Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4218893Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T04:46:10.4238484Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4289552Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T04:46:10.4310216Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4349484Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T04:46:10.4369419Z http.https://github.com/.extraheader 2024-12-18T04:46:10.4560646Z Cleaning up orphan processes