2024-08-21T03:06:16.3755949Z Current runner version: '2.319.1' 2024-08-21T03:06:16.3761287Z Runner name: 'pytorch-rocm-hw-50' 2024-08-21T03:06:16.3761936Z Runner group name: 'linux.rocm.gpu.group' 2024-08-21T03:06:16.3762715Z Machine name: 'pytorch-rocm-hw-50' 2024-08-21T03:06:16.3776689Z Testing runner upgrade compatibility 2024-08-21T03:06:16.4438884Z ##[group]GITHUB_TOKEN Permissions 2024-08-21T03:06:16.4443006Z Contents: read 2024-08-21T03:06:16.4443985Z Metadata: read 2024-08-21T03:06:16.4444903Z ##[endgroup] 2024-08-21T03:06:16.4452041Z Secret source: Actions 2024-08-21T03:06:16.4453307Z Prepare workflow directory 2024-08-21T03:06:16.8285485Z Prepare all required actions 2024-08-21T03:06:16.8447057Z Getting action download info 2024-08-21T03:06:17.1262360Z Download action repository 'pytorch/pytorch@main' (SHA:e8fc1e0118da85c3529f71fa60823fdbbbebee56) 2024-08-21T03:06:25.2542098Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-08-21T03:06:25.7960625Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-08-21T03:06:26.3351154Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-21T03:06:27.2501026Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T03:06:27.8016680Z Getting action download info 2024-08-21T03:06:27.9769624Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-21T03:06:28.5655586Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-21T03:06:28.5657751Z ##[group] Inputs 2024-08-21T03:06:28.5658151Z build-environment: linux-focal-rocm6.1-py3.8 2024-08-21T03:06:28.5659249Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-08-21T03:06:28.5660622Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:06:28.5661331Z sync-tag: 2024-08-21T03:06:28.5662146Z timeout-minutes: 300 2024-08-21T03:06:28.5662978Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-21T03:06:28.5663836Z ##[endgroup] 2024-08-21T03:06:28.5664493Z Complete job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:06:28.7308407Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-21T03:06:28.7308861Z with: 2024-08-21T03:06:28.7309085Z no-sudo: true 2024-08-21T03:06:28.7309337Z submodules: recursive 2024-08-21T03:06:28.7309575Z fetch-depth: 0 2024-08-21T03:06:28.7309813Z env: 2024-08-21T03:06:28.7310032Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:06:28.7310292Z ##[endgroup] 2024-08-21T03:06:28.7495082Z ##[group]Run retry () { 2024-08-21T03:06:28.7495413Z retry () { 2024-08-21T03:06:28.7495818Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-21T03:06:28.7496273Z } 2024-08-21T03:06:28.7496502Z echo "${GITHUB_WORKSPACE}" 2024-08-21T03:06:28.7496822Z if [ -z "${NO_SUDO}" ]; then 2024-08-21T03:06:28.7497188Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-21T03:06:28.7497523Z else 2024-08-21T03:06:28.7497786Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-21T03:06:28.7498187Z fi 2024-08-21T03:06:28.7498433Z mkdir "${GITHUB_WORKSPACE}" 2024-08-21T03:06:28.7524484Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:06:28.7524899Z env: 2024-08-21T03:06:28.7525135Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:06:28.7525720Z NO_SUDO: true 2024-08-21T03:06:28.7525954Z ##[endgroup] 2024-08-21T03:06:28.7727538Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:06:32.2812683Z ##[group]Run malfet/checkout@silent-checkout 2024-08-21T03:06:32.2813315Z with: 2024-08-21T03:06:32.2813723Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:06:32.2814287Z fetch-depth: 0 2024-08-21T03:06:32.2814698Z submodules: recursive 2024-08-21T03:06:32.2815129Z quiet-checkout: true 2024-08-21T03:06:32.2815581Z repository: pytorch/pytorch 2024-08-21T03:06:32.2816266Z token: *** 2024-08-21T03:06:32.2816628Z ssh-strict: true 2024-08-21T03:06:32.2817050Z persist-credentials: true 2024-08-21T03:06:32.2817530Z clean: true 2024-08-21T03:06:32.2817925Z sparse-checkout-cone-mode: true 2024-08-21T03:06:32.2818415Z lfs: false 2024-08-21T03:06:32.2818788Z set-safe-directory: true 2024-08-21T03:06:32.2819197Z env: 2024-08-21T03:06:32.2819531Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:06:32.2819963Z ##[endgroup] 2024-08-21T03:06:32.3729673Z Syncing repository: pytorch/pytorch 2024-08-21T03:06:32.3732024Z ##[group]Getting Git version info 2024-08-21T03:06:32.3733053Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-08-21T03:06:32.3734226Z [command]/usr/bin/git version 2024-08-21T03:06:32.3734714Z git version 2.34.1 2024-08-21T03:06:32.3736416Z ##[endgroup] 2024-08-21T03:06:32.3753576Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/bfd954fc-f7b1-466e-9b4f-c73bef05846b' before making global git config changes 2024-08-21T03:06:32.3755241Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T03:06:32.3756761Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:06:32.3758822Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-08-21T03:06:32.3760014Z ##[group]Initializing the repository 2024-08-21T03:06:32.3761079Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:06:32.3762273Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-21T03:06:32.3763371Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-21T03:06:32.3764392Z hint: of your new repositories, which will suppress this warning, call: 2024-08-21T03:06:32.3765117Z hint: 2024-08-21T03:06:32.3765876Z hint: git config --global init.defaultBranch 2024-08-21T03:06:32.3766496Z hint: 2024-08-21T03:06:32.3767157Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-21T03:06:32.3768717Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-21T03:06:32.3769510Z hint: 2024-08-21T03:06:32.3769921Z hint: git branch -m 2024-08-21T03:06:32.3770914Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-21T03:06:32.3772824Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-21T03:06:32.3800656Z ##[endgroup] 2024-08-21T03:06:32.3801480Z ##[group]Disabling automatic garbage collection 2024-08-21T03:06:32.3802367Z [command]/usr/bin/git config --local gc.auto 0 2024-08-21T03:06:32.3838130Z ##[endgroup] 2024-08-21T03:06:32.3838709Z ##[group]Setting up auth 2024-08-21T03:06:32.3844630Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T03:06:32.3870170Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T03:06:32.4134936Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T03:06:32.4156007Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T03:06:32.4377199Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-21T03:06:32.4411479Z ##[endgroup] 2024-08-21T03:06:32.4411937Z ##[group]Fetching the repository 2024-08-21T03:06:32.4416482Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-21T03:06:35.9288197Z remote: Enumerating objects: 1007039 2024-08-21T03:06:35.9289347Z remote: Enumerating objects: 1009630, done. 2024-08-21T03:06:35.9290469Z remote: Counting objects: 0% (1/2591) 2024-08-21T03:06:35.9291643Z remote: Counting objects: 1% (26/2591) 2024-08-21T03:06:35.9292671Z remote: Counting objects: 2% (52/2591) 2024-08-21T03:06:35.9293722Z remote: Counting objects: 3% (78/2591) 2024-08-21T03:06:35.9294715Z remote: Counting objects: 4% (104/2591) 2024-08-21T03:06:35.9295542Z remote: Counting objects: 5% (130/2591) 2024-08-21T03:06:35.9296362Z remote: Counting objects: 6% (156/2591) 2024-08-21T03:06:35.9297128Z remote: Counting objects: 7% (182/2591) 2024-08-21T03:06:35.9297895Z remote: Counting objects: 8% (208/2591) 2024-08-21T03:06:35.9298646Z remote: Counting objects: 9% (234/2591) 2024-08-21T03:06:35.9299460Z remote: Counting objects: 10% (260/2591) 2024-08-21T03:06:35.9300241Z remote: Counting objects: 11% (286/2591) 2024-08-21T03:06:35.9300997Z remote: Counting objects: 12% (311/2591) 2024-08-21T03:06:35.9301777Z remote: Counting objects: 13% (337/2591) 2024-08-21T03:06:35.9302592Z remote: Counting objects: 14% (363/2591) 2024-08-21T03:06:35.9303414Z remote: Counting objects: 15% (389/2591) 2024-08-21T03:06:35.9304260Z remote: Counting objects: 16% (415/2591) 2024-08-21T03:06:35.9305101Z remote: Counting objects: 17% (441/2591) 2024-08-21T03:06:35.9305921Z remote: Counting objects: 18% (467/2591) 2024-08-21T03:06:35.9306760Z remote: Counting objects: 19% (493/2591) 2024-08-21T03:06:35.9307554Z remote: Counting objects: 20% (519/2591) 2024-08-21T03:06:35.9308440Z remote: Counting objects: 21% (545/2591) 2024-08-21T03:06:35.9309430Z remote: Counting objects: 22% (571/2591) 2024-08-21T03:06:35.9310390Z remote: Counting objects: 23% (596/2591) 2024-08-21T03:06:35.9311250Z remote: Counting objects: 24% (622/2591) 2024-08-21T03:06:35.9312061Z remote: Counting objects: 25% (648/2591) 2024-08-21T03:06:35.9312921Z remote: Counting objects: 26% (674/2591) 2024-08-21T03:06:35.9368329Z remote: Counting objects: 27% (700/2591) 2024-08-21T03:06:35.9369482Z remote: Counting objects: 28% (726/2591) 2024-08-21T03:06:35.9370390Z remote: Counting objects: 29% (752/2591) 2024-08-21T03:06:35.9371227Z remote: Counting objects: 30% (778/2591) 2024-08-21T03:06:35.9372045Z remote: Counting objects: 31% (804/2591) 2024-08-21T03:06:35.9372873Z remote: Counting objects: 32% (830/2591) 2024-08-21T03:06:35.9373717Z remote: Counting objects: 33% (856/2591) 2024-08-21T03:06:35.9374517Z remote: Counting objects: 34% (881/2591) 2024-08-21T03:06:35.9375346Z remote: Counting objects: 35% (907/2591) 2024-08-21T03:06:35.9376242Z remote: Counting objects: 36% (933/2591) 2024-08-21T03:06:35.9377125Z remote: Counting objects: 37% (959/2591) 2024-08-21T03:06:35.9377955Z remote: Counting objects: 38% (985/2591) 2024-08-21T03:06:35.9378840Z remote: Counting objects: 39% (1011/2591) 2024-08-21T03:06:35.9379741Z remote: Counting objects: 40% (1037/2591) 2024-08-21T03:06:35.9380612Z remote: Counting objects: 41% (1063/2591) 2024-08-21T03:06:35.9381474Z remote: Counting objects: 42% (1089/2591) 2024-08-21T03:06:35.9382328Z remote: Counting objects: 43% (1115/2591) 2024-08-21T03:06:35.9383665Z remote: Counting objects: 44% (1141/2591) 2024-08-21T03:06:35.9384534Z remote: Counting objects: 45% (1166/2591) 2024-08-21T03:06:35.9385392Z remote: Counting objects: 46% (1192/2591) 2024-08-21T03:06:35.9386272Z remote: Counting objects: 47% (1218/2591) 2024-08-21T03:06:35.9387099Z remote: Counting objects: 48% (1244/2591) 2024-08-21T03:06:35.9387953Z remote: Counting objects: 49% (1270/2591) 2024-08-21T03:06:35.9388777Z remote: Counting objects: 50% (1296/2591) 2024-08-21T03:06:35.9389605Z remote: Counting objects: 51% (1322/2591) 2024-08-21T03:06:35.9390439Z remote: Counting objects: 52% (1348/2591) 2024-08-21T03:06:35.9391276Z remote: Counting objects: 53% (1374/2591) 2024-08-21T03:06:35.9392102Z remote: Counting objects: 54% (1400/2591) 2024-08-21T03:06:35.9392924Z remote: Counting objects: 55% (1426/2591) 2024-08-21T03:06:35.9393778Z remote: Counting objects: 56% (1451/2591) 2024-08-21T03:06:35.9394606Z remote: Counting objects: 57% (1477/2591) 2024-08-21T03:06:35.9395423Z remote: Counting objects: 58% (1503/2591) 2024-08-21T03:06:35.9396247Z remote: Counting objects: 59% (1529/2591) 2024-08-21T03:06:35.9397064Z remote: Counting objects: 60% (1555/2591) 2024-08-21T03:06:35.9397860Z remote: Counting objects: 61% (1581/2591) 2024-08-21T03:06:35.9398666Z remote: Counting objects: 62% (1607/2591) 2024-08-21T03:06:35.9399498Z remote: Counting objects: 63% (1633/2591) 2024-08-21T03:06:35.9400471Z remote: Counting objects: 64% (1659/2591) 2024-08-21T03:06:35.9401293Z remote: Counting objects: 65% (1685/2591) 2024-08-21T03:06:35.9402108Z remote: Counting objects: 66% (1711/2591) 2024-08-21T03:06:35.9402908Z remote: Counting objects: 67% (1736/2591) 2024-08-21T03:06:35.9403747Z remote: Counting objects: 68% (1762/2591) 2024-08-21T03:06:35.9404590Z remote: Counting objects: 69% (1788/2591) 2024-08-21T03:06:35.9405393Z remote: Counting objects: 70% (1814/2591) 2024-08-21T03:06:35.9406214Z remote: Counting objects: 71% (1840/2591) 2024-08-21T03:06:35.9407029Z remote: Counting objects: 72% (1866/2591) 2024-08-21T03:06:35.9407825Z remote: Counting objects: 73% (1892/2591) 2024-08-21T03:06:35.9408641Z remote: Counting objects: 74% (1918/2591) 2024-08-21T03:06:35.9409470Z remote: Counting objects: 75% (1944/2591) 2024-08-21T03:06:35.9410315Z remote: Counting objects: 76% (1970/2591) 2024-08-21T03:06:35.9411121Z remote: Counting objects: 77% (1996/2591) 2024-08-21T03:06:35.9412279Z remote: Counting objects: 78% (2021/2591) 2024-08-21T03:06:35.9413148Z remote: Counting objects: 79% (2047/2591) 2024-08-21T03:06:35.9413972Z remote: Counting objects: 80% (2073/2591) 2024-08-21T03:06:35.9414823Z remote: Counting objects: 81% (2099/2591) 2024-08-21T03:06:35.9415631Z remote: Counting objects: 82% (2125/2591) 2024-08-21T03:06:35.9416450Z remote: Counting objects: 83% (2151/2591) 2024-08-21T03:06:35.9417266Z remote: Counting objects: 84% (2177/2591) 2024-08-21T03:06:35.9418063Z remote: Counting objects: 85% (2203/2591) 2024-08-21T03:06:35.9418879Z remote: Counting objects: 86% (2229/2591) 2024-08-21T03:06:35.9419727Z remote: Counting objects: 87% (2255/2591) 2024-08-21T03:06:35.9420529Z remote: Counting objects: 88% (2281/2591) 2024-08-21T03:06:35.9421348Z remote: Counting objects: 89% (2306/2591) 2024-08-21T03:06:35.9422169Z remote: Counting objects: 90% (2332/2591) 2024-08-21T03:06:35.9422983Z remote: Counting objects: 91% (2358/2591) 2024-08-21T03:06:35.9423797Z remote: Counting objects: 92% (2384/2591) 2024-08-21T03:06:35.9424615Z remote: Counting objects: 93% (2410/2591) 2024-08-21T03:06:35.9425409Z remote: Counting objects: 94% (2436/2591) 2024-08-21T03:06:35.9426530Z remote: Counting objects: 95% (2462/2591) 2024-08-21T03:06:35.9427339Z remote: Counting objects: 96% (2488/2591) 2024-08-21T03:06:35.9428138Z remote: Counting objects: 97% (2514/2591) 2024-08-21T03:06:35.9428949Z remote: Counting objects: 98% (2540/2591) 2024-08-21T03:06:35.9429750Z remote: Counting objects: 99% (2566/2591) 2024-08-21T03:06:35.9818970Z remote: Counting objects: 100% (2591/2591) 2024-08-21T03:06:35.9819990Z remote: Counting objects: 100% (2591/2591), done. 2024-08-21T03:06:35.9820907Z remote: Compressing objects: 0% (1/1205) 2024-08-21T03:06:35.9821745Z remote: Compressing objects: 1% (13/1205) 2024-08-21T03:06:36.0352191Z remote: Compressing objects: 2% (25/1205) 2024-08-21T03:06:36.0887319Z remote: Compressing objects: 3% (37/1205) 2024-08-21T03:06:36.4610586Z remote: Compressing objects: 4% (49/1205) 2024-08-21T03:06:36.4611638Z remote: Compressing objects: 5% (61/1205) 2024-08-21T03:06:36.4612652Z remote: Compressing objects: 6% (73/1205) 2024-08-21T03:06:36.4613531Z remote: Compressing objects: 7% (85/1205) 2024-08-21T03:06:36.4614422Z remote: Compressing objects: 8% (97/1205) 2024-08-21T03:06:36.5150428Z remote: Compressing objects: 9% (109/1205) 2024-08-21T03:06:36.5151516Z remote: Compressing objects: 10% (121/1205) 2024-08-21T03:06:36.5685270Z remote: Compressing objects: 11% (133/1205) 2024-08-21T03:06:36.5686334Z remote: Compressing objects: 12% (145/1205) 2024-08-21T03:06:36.5687328Z remote: Compressing objects: 13% (157/1205) 2024-08-21T03:06:36.6212077Z remote: Compressing objects: 14% (169/1205) 2024-08-21T03:06:36.6213173Z remote: Compressing objects: 15% (181/1205) 2024-08-21T03:06:36.6214132Z remote: Compressing objects: 16% (193/1205) 2024-08-21T03:06:36.6215029Z remote: Compressing objects: 17% (205/1205) 2024-08-21T03:06:36.6610449Z remote: Compressing objects: 18% (217/1205) 2024-08-21T03:06:36.6611492Z remote: Compressing objects: 19% (229/1205) 2024-08-21T03:06:36.6612468Z remote: Compressing objects: 20% (241/1205) 2024-08-21T03:06:36.6613367Z remote: Compressing objects: 21% (254/1205) 2024-08-21T03:06:36.6614294Z remote: Compressing objects: 22% (266/1205) 2024-08-21T03:06:36.6615166Z remote: Compressing objects: 23% (278/1205) 2024-08-21T03:06:36.6616060Z remote: Compressing objects: 24% (290/1205) 2024-08-21T03:06:36.6616930Z remote: Compressing objects: 25% (302/1205) 2024-08-21T03:06:36.6617804Z remote: Compressing objects: 26% (314/1205) 2024-08-21T03:06:36.6619161Z remote: Compressing objects: 27% (326/1205) 2024-08-21T03:06:36.6620078Z remote: Compressing objects: 28% (338/1205) 2024-08-21T03:06:36.6620965Z remote: Compressing objects: 29% (350/1205) 2024-08-21T03:06:36.6621868Z remote: Compressing objects: 30% (362/1205) 2024-08-21T03:06:36.6622777Z remote: Compressing objects: 31% (374/1205) 2024-08-21T03:06:36.6623621Z remote: Compressing objects: 32% (386/1205) 2024-08-21T03:06:36.6624491Z remote: Compressing objects: 33% (398/1205) 2024-08-21T03:06:36.6625366Z remote: Compressing objects: 34% (410/1205) 2024-08-21T03:06:36.6626207Z remote: Compressing objects: 35% (422/1205) 2024-08-21T03:06:36.6627066Z remote: Compressing objects: 36% (434/1205) 2024-08-21T03:06:36.6627939Z remote: Compressing objects: 37% (446/1205) 2024-08-21T03:06:36.6747682Z remote: Compressing objects: 38% (458/1205) 2024-08-21T03:06:36.6748791Z remote: Compressing objects: 39% (470/1205) 2024-08-21T03:06:36.6749762Z remote: Compressing objects: 40% (482/1205) 2024-08-21T03:06:36.6750674Z remote: Compressing objects: 41% (495/1205) 2024-08-21T03:06:36.6751577Z remote: Compressing objects: 42% (507/1205) 2024-08-21T03:06:36.6752950Z remote: Compressing objects: 43% (519/1205) 2024-08-21T03:06:36.6753850Z remote: Compressing objects: 44% (531/1205) 2024-08-21T03:06:36.6754756Z remote: Compressing objects: 45% (543/1205) 2024-08-21T03:06:36.6755619Z remote: Compressing objects: 46% (555/1205) 2024-08-21T03:06:36.6756491Z remote: Compressing objects: 47% (567/1205) 2024-08-21T03:06:36.6757339Z remote: Compressing objects: 48% (579/1205) 2024-08-21T03:06:36.6758208Z remote: Compressing objects: 49% (591/1205) 2024-08-21T03:06:36.6759083Z remote: Compressing objects: 50% (603/1205) 2024-08-21T03:06:36.6759924Z remote: Compressing objects: 51% (615/1205) 2024-08-21T03:06:36.6761005Z remote: Compressing objects: 52% (627/1205) 2024-08-21T03:06:36.6827856Z remote: Compressing objects: 53% (639/1205) 2024-08-21T03:06:36.6828913Z remote: Compressing objects: 54% (651/1205) 2024-08-21T03:06:36.6829849Z remote: Compressing objects: 55% (663/1205) 2024-08-21T03:06:36.6830789Z remote: Compressing objects: 56% (675/1205) 2024-08-21T03:06:36.6831683Z remote: Compressing objects: 57% (687/1205) 2024-08-21T03:06:36.6832575Z remote: Compressing objects: 58% (699/1205) 2024-08-21T03:06:36.6833473Z remote: Compressing objects: 59% (711/1205) 2024-08-21T03:06:36.6834358Z remote: Compressing objects: 60% (723/1205) 2024-08-21T03:06:36.6835225Z remote: Compressing objects: 61% (736/1205) 2024-08-21T03:06:36.6836094Z remote: Compressing objects: 62% (748/1205) 2024-08-21T03:06:36.6836967Z remote: Compressing objects: 63% (760/1205) 2024-08-21T03:06:36.6837830Z remote: Compressing objects: 64% (772/1205) 2024-08-21T03:06:36.6838705Z remote: Compressing objects: 65% (784/1205) 2024-08-21T03:06:36.6839565Z remote: Compressing objects: 66% (796/1205) 2024-08-21T03:06:36.6840552Z remote: Compressing objects: 67% (808/1205) 2024-08-21T03:06:36.6841467Z remote: Compressing objects: 68% (820/1205) 2024-08-21T03:06:36.6842332Z remote: Compressing objects: 69% (832/1205) 2024-08-21T03:06:36.6843177Z remote: Compressing objects: 70% (844/1205) 2024-08-21T03:06:36.6844026Z remote: Compressing objects: 71% (856/1205) 2024-08-21T03:06:36.6844881Z remote: Compressing objects: 72% (868/1205) 2024-08-21T03:06:36.6902077Z remote: Compressing objects: 73% (880/1205) 2024-08-21T03:06:36.6903146Z remote: Compressing objects: 74% (892/1205) 2024-08-21T03:06:36.6904199Z remote: Compressing objects: 75% (904/1205) 2024-08-21T03:06:36.6905105Z remote: Compressing objects: 76% (916/1205) 2024-08-21T03:06:36.6906431Z remote: Compressing objects: 77% (928/1205) 2024-08-21T03:06:36.6907377Z remote: Compressing objects: 78% (940/1205) 2024-08-21T03:06:36.6908279Z remote: Compressing objects: 79% (952/1205) 2024-08-21T03:06:36.6909146Z remote: Compressing objects: 80% (964/1205) 2024-08-21T03:06:36.6910044Z remote: Compressing objects: 81% (977/1205) 2024-08-21T03:06:36.6910904Z remote: Compressing objects: 82% (989/1205) 2024-08-21T03:06:36.6911781Z remote: Compressing objects: 83% (1001/1205) 2024-08-21T03:06:36.6912710Z remote: Compressing objects: 84% (1013/1205) 2024-08-21T03:06:36.6913612Z remote: Compressing objects: 85% (1025/1205) 2024-08-21T03:06:36.6914518Z remote: Compressing objects: 86% (1037/1205) 2024-08-21T03:06:36.6915400Z remote: Compressing objects: 87% (1049/1205) 2024-08-21T03:06:36.6916276Z remote: Compressing objects: 88% (1061/1205) 2024-08-21T03:06:36.6917190Z remote: Compressing objects: 89% (1073/1205) 2024-08-21T03:06:36.6918057Z remote: Compressing objects: 90% (1085/1205) 2024-08-21T03:06:36.6918924Z remote: Compressing objects: 91% (1097/1205) 2024-08-21T03:06:36.6919804Z remote: Compressing objects: 92% (1109/1205) 2024-08-21T03:06:36.6923364Z remote: Compressing objects: 93% (1121/1205) 2024-08-21T03:06:36.6924345Z remote: Compressing objects: 94% (1133/1205) 2024-08-21T03:06:36.6925361Z remote: Compressing objects: 95% (1145/1205) 2024-08-21T03:06:36.6926408Z remote: Compressing objects: 96% (1157/1205) 2024-08-21T03:06:36.6927407Z remote: Compressing objects: 97% (1169/1205) 2024-08-21T03:06:36.6928282Z remote: Compressing objects: 98% (1181/1205) 2024-08-21T03:06:36.6929155Z remote: Compressing objects: 99% (1193/1205) 2024-08-21T03:06:36.6930048Z remote: Compressing objects: 100% (1205/1205) 2024-08-21T03:06:36.6931062Z remote: Compressing objects: 100% (1205/1205), done. 2024-08-21T03:07:02.1877774Z remote: Total 1009630 (delta 1737), reused 2039 (delta 1378), pack-reused 1007039 (from 1) 2024-08-21T03:07:13.4846607Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-21T03:07:13.4887147Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:07:13.4895946Z ##[endgroup] 2024-08-21T03:07:13.4896434Z ##[group]Determining the checkout info 2024-08-21T03:07:13.4896972Z ##[endgroup] 2024-08-21T03:07:13.4897424Z ##[group]Checking out the ref 2024-08-21T03:07:13.4901151Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:07:14.8610661Z ##[endgroup] 2024-08-21T03:07:14.8611897Z ##[group]Setting up auth for fetching submodules 2024-08-21T03:07:14.8613695Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-21T03:07:14.8681431Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-21T03:07:14.8725602Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-21T03:07:14.8759513Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-21T03:07:14.8808783Z ##[endgroup] 2024-08-21T03:07:14.8809833Z ##[group]Fetching submodules 2024-08-21T03:07:14.8814537Z [command]/usr/bin/git submodule sync --recursive 2024-08-21T03:07:14.9140091Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-21T03:07:14.9381962Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-21T03:07:14.9384627Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-21T03:07:14.9388438Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-21T03:07:14.9391846Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-21T03:07:14.9395422Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-21T03:07:14.9398258Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-21T03:07:14.9405183Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-21T03:07:14.9406251Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-21T03:07:14.9409252Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-21T03:07:14.9412955Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-21T03:07:14.9416547Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-21T03:07:14.9420288Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-21T03:07:14.9424115Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-21T03:07:14.9428055Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-21T03:07:14.9431903Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-21T03:07:14.9436024Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:07:14.9440006Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-21T03:07:14.9444656Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-21T03:07:14.9448770Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-21T03:07:14.9453105Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-21T03:07:14.9457403Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-21T03:07:14.9461842Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-21T03:07:14.9466169Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-21T03:07:14.9470667Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-21T03:07:14.9475136Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-21T03:07:14.9480030Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-21T03:07:14.9485112Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-21T03:07:14.9489923Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-21T03:07:14.9494710Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-21T03:07:14.9499670Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-21T03:07:14.9504584Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-21T03:07:14.9509457Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-21T03:07:14.9514389Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-21T03:07:14.9519540Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-21T03:07:14.9591983Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-21T03:07:15.5988401Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-21T03:07:16.1099124Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-21T03:07:16.5951144Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-21T03:07:17.1883598Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-21T03:07:19.5864041Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-21T03:07:27.4802947Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-21T03:07:28.2925495Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-21T03:07:29.0507425Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-21T03:07:30.0547544Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-21T03:07:32.3969989Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-21T03:07:34.6408010Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-21T03:07:38.6265567Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-21T03:07:40.1702603Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-21T03:07:42.2441082Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-21T03:07:44.5471283Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-21T03:07:45.3374413Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-21T03:07:46.0131925Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-21T03:07:47.2318250Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-21T03:07:47.9694812Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-21T03:07:48.5719916Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-21T03:07:50.9554022Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-21T03:07:51.9426551Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-21T03:07:53.2137914Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-21T03:07:59.6657581Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-21T03:08:01.7768738Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-21T03:08:06.5650013Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-21T03:08:07.1227570Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-21T03:08:15.1826111Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-21T03:08:15.6515030Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-21T03:08:16.2102341Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-21T03:08:17.3401387Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-21T03:08:17.9749797Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-21T03:08:18.9348858Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-21T03:08:19.8626697Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-21T03:08:19.8898879Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-21T03:08:19.9143875Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-21T03:08:19.9531473Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-21T03:08:20.0044438Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-21T03:08:20.9833317Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-21T03:08:21.0284439Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-21T03:08:21.0907482Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-21T03:08:21.2031238Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-21T03:08:21.2531301Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-21T03:08:21.7652063Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-21T03:08:22.0163584Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-21T03:08:22.1089131Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-21T03:08:22.1141515Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:08:22.1144074Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:08:22.1147380Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:08:22.1150766Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:08:22.1154349Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:08:22.1202179Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-21T03:08:23.4069747Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-21T03:08:24.3401019Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-21T03:08:26.9893439Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-21T03:08:28.2727520Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-21T03:08:28.9503035Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-21T03:08:29.0551454Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-21T03:08:29.4794500Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-21T03:08:29.5562450Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-21T03:08:29.5839710Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-21T03:08:29.7198182Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-21T03:08:29.7826677Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-21T03:08:29.8405501Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-21T03:08:29.8816167Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-21T03:08:29.9410330Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-21T03:08:29.9692001Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-21T03:08:29.9743152Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-21T03:08:29.9784420Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-21T03:08:41.4936843Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-21T03:08:41.5297931Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-21T03:08:41.6293319Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-21T03:08:41.6378566Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:08:41.6382430Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:08:41.6383999Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:08:41.6426270Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-21T03:08:42.5028908Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-21T03:08:43.9076695Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-21T03:08:45.1734564Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-21T03:08:45.1796587Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:08:45.1800059Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:08:45.1803626Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:08:45.1807512Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:08:45.1811034Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:08:45.1814769Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:08:45.1818563Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:08:45.1822316Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:08:45.1869805Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-21T03:08:46.2902215Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-21T03:08:47.0563882Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-21T03:08:48.3267664Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-21T03:08:49.2131754Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-21T03:08:50.1657547Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-21T03:08:51.4687738Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-21T03:08:58.0363105Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-21T03:08:58.8710419Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-21T03:08:58.9030106Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-21T03:08:58.9525141Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-21T03:08:58.9798535Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-21T03:08:58.9850728Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:08:58.9898570Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-21T03:09:00.1667320Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-21T03:09:00.2001500Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-21T03:09:00.2542570Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-21T03:09:00.3630666Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-21T03:09:00.3964621Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-21T03:09:00.4507004Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-21T03:09:00.5210905Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-21T03:09:00.5727089Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-21T03:09:00.6139208Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-21T03:09:00.7335661Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-21T03:09:01.1464645Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-21T03:09:01.1552233Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:01.1555617Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:01.1614910Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-21T03:09:02.0034807Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-21T03:09:03.3834833Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-21T03:09:03.4374091Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-21T03:09:03.5199450Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-21T03:09:03.5262328Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:03.5265352Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:03.5268367Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:03.5271850Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:03.5275789Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:03.5279328Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:03.5287500Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:03.5289756Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:03.5337500Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-21T03:09:04.3422668Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-21T03:09:05.5383893Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-21T03:09:06.2422158Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-21T03:09:14.1777966Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-21T03:09:14.8215477Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-21T03:09:15.3841392Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-21T03:09:16.0372864Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-21T03:09:24.8775887Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-21T03:09:24.9378242Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-21T03:09:24.9684094Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-21T03:09:25.0851052Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-21T03:09:25.1169128Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-21T03:09:25.1422603Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-21T03:09:25.1736491Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-21T03:09:25.1787186Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:25.1790425Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:25.1836849Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-21T03:09:27.0271860Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-21T03:09:28.5592059Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-21T03:09:28.6215383Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-21T03:09:29.1368255Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-21T03:09:29.1730947Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-21T03:09:29.4468502Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-21T03:09:29.4543511Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:29.4546146Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:29.4631954Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-21T03:09:30.3177127Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-21T03:09:31.6108683Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-21T03:09:31.6892790Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-21T03:09:31.7158362Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-21T03:09:31.7412965Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-21T03:09:31.7914443Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-21T03:09:31.8376470Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-21T03:09:31.8945738Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-21T03:09:31.9405120Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-21T03:09:31.9465541Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:31.9467988Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:31.9470287Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:31.9474082Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:31.9522251Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-21T03:09:33.1619362Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-21T03:09:33.7341769Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-21T03:09:35.1799675Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-21T03:09:36.4657043Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-21T03:09:36.4940546Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-21T03:09:36.5723535Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-21T03:09:36.6214248Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-21T03:09:36.6285902Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:36.6330934Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-21T03:09:37.1753105Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-21T03:09:37.1857468Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-21T03:09:37.2129078Z Entering 'android/libs/fbjni' 2024-08-21T03:09:37.2172051Z Entering 'third_party/FP16' 2024-08-21T03:09:37.2209203Z Entering 'third_party/FXdiv' 2024-08-21T03:09:37.2247956Z Entering 'third_party/NNPACK' 2024-08-21T03:09:37.2286640Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:09:37.2326118Z Entering 'third_party/XNNPACK' 2024-08-21T03:09:37.2381018Z Entering 'third_party/benchmark' 2024-08-21T03:09:37.2429436Z Entering 'third_party/cpp-httplib' 2024-08-21T03:09:37.2475231Z Entering 'third_party/cpuinfo' 2024-08-21T03:09:37.2526665Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:09:37.2565984Z Entering 'third_party/cutlass' 2024-08-21T03:09:37.2610341Z Entering 'third_party/eigen' 2024-08-21T03:09:37.2657930Z Entering 'third_party/fbgemm' 2024-08-21T03:09:37.2693398Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:09:37.2730225Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:09:37.2765534Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:09:37.2809481Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:09:37.2845761Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:09:37.2885363Z Entering 'third_party/flatbuffers' 2024-08-21T03:09:37.2925596Z Entering 'third_party/fmt' 2024-08-21T03:09:37.2964467Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:09:37.2999363Z Entering 'third_party/gloo' 2024-08-21T03:09:37.3033646Z Entering 'third_party/googletest' 2024-08-21T03:09:37.3073116Z Entering 'third_party/ideep' 2024-08-21T03:09:37.3115584Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:09:37.3161917Z Entering 'third_party/ittapi' 2024-08-21T03:09:37.3201817Z Entering 'third_party/kineto' 2024-08-21T03:09:37.3242786Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:09:37.3274812Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:09:37.3315036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:09:37.3352790Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:09:37.3389167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:09:37.3420794Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:09:37.3462367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:09:37.3504233Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:09:37.3544122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:09:37.3586459Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:09:37.3623276Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:09:37.3659503Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:09:37.3694625Z Entering 'third_party/mimalloc' 2024-08-21T03:09:37.3730218Z Entering 'third_party/nccl/nccl' 2024-08-21T03:09:37.3774023Z Entering 'third_party/nlohmann' 2024-08-21T03:09:37.3821905Z Entering 'third_party/onnx' 2024-08-21T03:09:37.3874952Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:37.3915391Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:37.3955721Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:09:37.4003784Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:37.4045182Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:37.4077524Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:37.4114500Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:37.4150341Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:37.4181318Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:37.4215143Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:37.4247657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:37.4288935Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:37.4329320Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:37.4379234Z Entering 'third_party/pocketfft' 2024-08-21T03:09:37.4421328Z Entering 'third_party/protobuf' 2024-08-21T03:09:37.4467965Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:37.4507326Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:37.4547260Z Entering 'third_party/psimd' 2024-08-21T03:09:37.4583217Z Entering 'third_party/pthreadpool' 2024-08-21T03:09:37.4624247Z Entering 'third_party/pybind11' 2024-08-21T03:09:37.4664888Z Entering 'third_party/python-peachpy' 2024-08-21T03:09:37.4703071Z Entering 'third_party/sleef' 2024-08-21T03:09:37.4745746Z Entering 'third_party/tensorpipe' 2024-08-21T03:09:37.4787165Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:37.4827145Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:37.4865108Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:37.4899032Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:37.4935627Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:37.4998201Z ##[endgroup] 2024-08-21T03:09:37.5005061Z ##[group]Persisting credentials for submodules 2024-08-21T03:09:37.5006609Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-21T03:09:37.5246035Z Entering 'android/libs/fbjni' 2024-08-21T03:09:37.5287951Z Entering 'third_party/FP16' 2024-08-21T03:09:37.5340913Z Entering 'third_party/FXdiv' 2024-08-21T03:09:37.5387146Z Entering 'third_party/NNPACK' 2024-08-21T03:09:37.5428375Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:09:37.5468584Z Entering 'third_party/XNNPACK' 2024-08-21T03:09:37.5526554Z Entering 'third_party/benchmark' 2024-08-21T03:09:37.5570604Z Entering 'third_party/cpp-httplib' 2024-08-21T03:09:37.5614553Z Entering 'third_party/cpuinfo' 2024-08-21T03:09:37.5657953Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:09:37.5703186Z Entering 'third_party/cutlass' 2024-08-21T03:09:37.5751768Z Entering 'third_party/eigen' 2024-08-21T03:09:37.5797252Z Entering 'third_party/fbgemm' 2024-08-21T03:09:37.5845496Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:09:37.5884961Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:09:37.5925451Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:09:37.5967033Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:09:37.6006729Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:09:37.6048117Z Entering 'third_party/flatbuffers' 2024-08-21T03:09:37.6089286Z Entering 'third_party/fmt' 2024-08-21T03:09:37.6126857Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:09:37.6176325Z Entering 'third_party/gloo' 2024-08-21T03:09:37.6216180Z Entering 'third_party/googletest' 2024-08-21T03:09:37.6257208Z Entering 'third_party/ideep' 2024-08-21T03:09:37.6296086Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:09:37.6347068Z Entering 'third_party/ittapi' 2024-08-21T03:09:37.6387121Z Entering 'third_party/kineto' 2024-08-21T03:09:37.6424546Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:09:37.6461806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:09:37.6502097Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:09:37.6539532Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:09:37.6575095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:09:37.6610489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:09:37.6650546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:09:37.6692166Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:09:37.6732208Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:09:37.6773052Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:09:37.6816196Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:09:37.6852730Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:09:37.6897591Z Entering 'third_party/mimalloc' 2024-08-21T03:09:37.6935694Z Entering 'third_party/nccl/nccl' 2024-08-21T03:09:37.6978148Z Entering 'third_party/nlohmann' 2024-08-21T03:09:37.7017669Z Entering 'third_party/onnx' 2024-08-21T03:09:37.7067673Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:37.7109886Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:37.7150303Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:09:37.7190445Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:37.7227577Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:37.7264459Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:37.7300445Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:37.7345475Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:37.7386520Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:37.7423095Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:37.7469387Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:37.7524163Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:37.7582263Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:37.7645996Z Entering 'third_party/pocketfft' 2024-08-21T03:09:37.7694388Z Entering 'third_party/protobuf' 2024-08-21T03:09:37.7746073Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:37.7803168Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:37.7846829Z Entering 'third_party/psimd' 2024-08-21T03:09:37.7891281Z Entering 'third_party/pthreadpool' 2024-08-21T03:09:37.7933640Z Entering 'third_party/pybind11' 2024-08-21T03:09:37.7971725Z Entering 'third_party/python-peachpy' 2024-08-21T03:09:37.8013834Z Entering 'third_party/sleef' 2024-08-21T03:09:37.8055580Z Entering 'third_party/tensorpipe' 2024-08-21T03:09:37.8093703Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:37.8135778Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:37.8172742Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:37.8215718Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:37.8259746Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:37.8323392Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-21T03:09:37.8586618Z Entering 'android/libs/fbjni' 2024-08-21T03:09:37.8628064Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-21T03:09:37.8645742Z Entering 'third_party/FP16' 2024-08-21T03:09:37.8686703Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-21T03:09:37.8703348Z Entering 'third_party/FXdiv' 2024-08-21T03:09:37.8743195Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-21T03:09:37.8759843Z Entering 'third_party/NNPACK' 2024-08-21T03:09:37.8801129Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-21T03:09:37.8823419Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:09:37.8865107Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-21T03:09:37.8886617Z Entering 'third_party/XNNPACK' 2024-08-21T03:09:37.8927161Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-21T03:09:37.8955064Z Entering 'third_party/benchmark' 2024-08-21T03:09:37.8996250Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-21T03:09:37.9018647Z Entering 'third_party/cpp-httplib' 2024-08-21T03:09:37.9056108Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-21T03:09:37.9073282Z Entering 'third_party/cpuinfo' 2024-08-21T03:09:37.9109276Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-21T03:09:37.9127571Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:09:37.9166405Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-21T03:09:37.9181949Z Entering 'third_party/cutlass' 2024-08-21T03:09:37.9228181Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-21T03:09:37.9251896Z Entering 'third_party/eigen' 2024-08-21T03:09:37.9292295Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-21T03:09:37.9311483Z Entering 'third_party/fbgemm' 2024-08-21T03:09:37.9356818Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-21T03:09:37.9375602Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:09:37.9414483Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-21T03:09:37.9431152Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:09:37.9468206Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-21T03:09:37.9485011Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:09:37.9518661Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-21T03:09:37.9545511Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:09:37.9585983Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:37.9605275Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:09:37.9638330Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-21T03:09:37.9659661Z Entering 'third_party/flatbuffers' 2024-08-21T03:09:37.9698866Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-21T03:09:37.9719088Z Entering 'third_party/fmt' 2024-08-21T03:09:37.9758533Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-21T03:09:37.9780158Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:09:37.9818485Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-21T03:09:37.9835942Z Entering 'third_party/gloo' 2024-08-21T03:09:37.9869909Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-21T03:09:37.9891741Z Entering 'third_party/googletest' 2024-08-21T03:09:37.9926995Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:37.9944705Z Entering 'third_party/ideep' 2024-08-21T03:09:37.9982928Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-21T03:09:38.0007228Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:09:38.0040283Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-21T03:09:38.0065094Z Entering 'third_party/ittapi' 2024-08-21T03:09:38.0108945Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-21T03:09:38.0127138Z Entering 'third_party/kineto' 2024-08-21T03:09:38.0163587Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-21T03:09:38.0182300Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:09:38.0233275Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-21T03:09:38.0259796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:09:38.0299612Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-21T03:09:38.0324577Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:09:38.0360466Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-21T03:09:38.0378128Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:09:38.0410145Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-21T03:09:38.0427829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:09:38.0462860Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-21T03:09:38.0484124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:09:38.0526352Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-21T03:09:38.0543060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:09:38.0577921Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-21T03:09:38.0596236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:09:38.0630812Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:38.0647941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:09:38.0680400Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-21T03:09:38.0699613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:09:38.0737344Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-21T03:09:38.0762407Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:09:38.0821520Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-21T03:09:38.0847384Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:09:38.0889867Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-21T03:09:38.0910618Z Entering 'third_party/mimalloc' 2024-08-21T03:09:38.0944983Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-21T03:09:38.0963391Z Entering 'third_party/nccl/nccl' 2024-08-21T03:09:38.1009158Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-21T03:09:38.1029008Z Entering 'third_party/nlohmann' 2024-08-21T03:09:38.1065513Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-21T03:09:38.1083016Z Entering 'third_party/onnx' 2024-08-21T03:09:38.1115090Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-21T03:09:38.1146413Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:38.1187728Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-21T03:09:38.1205123Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:38.1244400Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-21T03:09:38.1266036Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:09:38.1303207Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-21T03:09:38.1325744Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:38.1370254Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-21T03:09:38.1388091Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:38.1423555Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:38.1445246Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:38.1479307Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-21T03:09:38.1497911Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:38.1536969Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-21T03:09:38.1557049Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:38.1597904Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-21T03:09:38.1615698Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:38.1649388Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-21T03:09:38.1671186Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:38.1712089Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-21T03:09:38.1730041Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:38.1767313Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-21T03:09:38.1787184Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:38.1821472Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-21T03:09:38.1846858Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:38.1878737Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-21T03:09:38.1920197Z Entering 'third_party/pocketfft' 2024-08-21T03:09:38.1963047Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-21T03:09:38.1981506Z Entering 'third_party/protobuf' 2024-08-21T03:09:38.2022162Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-21T03:09:38.2044768Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:38.2086157Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-21T03:09:38.2099855Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:38.2136320Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:38.2158008Z Entering 'third_party/psimd' 2024-08-21T03:09:38.2198449Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-21T03:09:38.2222823Z Entering 'third_party/pthreadpool' 2024-08-21T03:09:38.2264240Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-21T03:09:38.2283319Z Entering 'third_party/pybind11' 2024-08-21T03:09:38.2320588Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-21T03:09:38.2343767Z Entering 'third_party/python-peachpy' 2024-08-21T03:09:38.2384750Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-21T03:09:38.2405849Z Entering 'third_party/sleef' 2024-08-21T03:09:38.2446046Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-21T03:09:38.2460525Z Entering 'third_party/tensorpipe' 2024-08-21T03:09:38.2499335Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-21T03:09:38.2523836Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:38.2560076Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-21T03:09:38.2579774Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:38.2613270Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-21T03:09:38.2630119Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:38.2662924Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-21T03:09:38.2680046Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:38.2718524Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-21T03:09:38.2735451Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:38.2776131Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-21T03:09:38.3083004Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-21T03:09:38.3331345Z Entering 'android/libs/fbjni' 2024-08-21T03:09:38.3366544Z Entering 'third_party/FP16' 2024-08-21T03:09:38.3404239Z Entering 'third_party/FXdiv' 2024-08-21T03:09:38.3440785Z Entering 'third_party/NNPACK' 2024-08-21T03:09:38.3473976Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:09:38.3510324Z Entering 'third_party/XNNPACK' 2024-08-21T03:09:38.3558605Z Entering 'third_party/benchmark' 2024-08-21T03:09:38.3593679Z Entering 'third_party/cpp-httplib' 2024-08-21T03:09:38.3629918Z Entering 'third_party/cpuinfo' 2024-08-21T03:09:38.3666440Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:09:38.3702639Z Entering 'third_party/cutlass' 2024-08-21T03:09:38.3750715Z Entering 'third_party/eigen' 2024-08-21T03:09:38.3782543Z Entering 'third_party/fbgemm' 2024-08-21T03:09:38.3816990Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:09:38.3863189Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:09:38.3903640Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:09:38.3950454Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:09:38.3984787Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:09:38.4025285Z Entering 'third_party/flatbuffers' 2024-08-21T03:09:38.4063372Z Entering 'third_party/fmt' 2024-08-21T03:09:38.4102721Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:09:38.4141253Z Entering 'third_party/gloo' 2024-08-21T03:09:38.4181404Z Entering 'third_party/googletest' 2024-08-21T03:09:38.4219095Z Entering 'third_party/ideep' 2024-08-21T03:09:38.4255016Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:09:38.4301274Z Entering 'third_party/ittapi' 2024-08-21T03:09:38.4335957Z Entering 'third_party/kineto' 2024-08-21T03:09:38.4380960Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:09:38.4417998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:09:38.4453452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:09:38.4494510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:09:38.4529966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:09:38.4562063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:09:38.4609119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:09:38.4643378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:09:38.4676115Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:09:38.4712718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:09:38.4752856Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:09:38.4792458Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:09:38.4834263Z Entering 'third_party/mimalloc' 2024-08-21T03:09:38.4871476Z Entering 'third_party/nccl/nccl' 2024-08-21T03:09:38.4910395Z Entering 'third_party/nlohmann' 2024-08-21T03:09:38.4945158Z Entering 'third_party/onnx' 2024-08-21T03:09:38.4995471Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:38.5034405Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:38.5085385Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:09:38.5122264Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:38.5167034Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:38.5200184Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:38.5234944Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:38.5269873Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:38.5302592Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:38.5338972Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:38.5370774Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:38.5407689Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:38.5444425Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:38.5502075Z Entering 'third_party/pocketfft' 2024-08-21T03:09:38.5538932Z Entering 'third_party/protobuf' 2024-08-21T03:09:38.5575080Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:38.5620524Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:38.5664224Z Entering 'third_party/psimd' 2024-08-21T03:09:38.5702965Z Entering 'third_party/pthreadpool' 2024-08-21T03:09:38.5740379Z Entering 'third_party/pybind11' 2024-08-21T03:09:38.5774277Z Entering 'third_party/python-peachpy' 2024-08-21T03:09:38.5812657Z Entering 'third_party/sleef' 2024-08-21T03:09:38.5852451Z Entering 'third_party/tensorpipe' 2024-08-21T03:09:38.5896327Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:38.5941181Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:38.5972908Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:38.6011196Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:38.6043781Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:38.6104206Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-21T03:09:38.6388092Z Entering 'android/libs/fbjni' 2024-08-21T03:09:38.6433249Z Entering 'third_party/FP16' 2024-08-21T03:09:38.6469899Z Entering 'third_party/FXdiv' 2024-08-21T03:09:38.6503966Z Entering 'third_party/NNPACK' 2024-08-21T03:09:38.6540911Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:09:38.6575556Z Entering 'third_party/XNNPACK' 2024-08-21T03:09:38.6624636Z Entering 'third_party/benchmark' 2024-08-21T03:09:38.6660648Z Entering 'third_party/cpp-httplib' 2024-08-21T03:09:38.6695662Z Entering 'third_party/cpuinfo' 2024-08-21T03:09:38.6731954Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:09:38.6767056Z Entering 'third_party/cutlass' 2024-08-21T03:09:38.6808223Z Entering 'third_party/eigen' 2024-08-21T03:09:38.6845966Z Entering 'third_party/fbgemm' 2024-08-21T03:09:38.6885785Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:09:38.6918754Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:09:38.6952891Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:09:38.6997156Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:09:38.7033048Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:09:38.7074574Z Entering 'third_party/flatbuffers' 2024-08-21T03:09:38.7111701Z Entering 'third_party/fmt' 2024-08-21T03:09:38.7149373Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:09:38.7185043Z Entering 'third_party/gloo' 2024-08-21T03:09:38.7223427Z Entering 'third_party/googletest' 2024-08-21T03:09:38.7261737Z Entering 'third_party/ideep' 2024-08-21T03:09:38.7297748Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:09:38.7348792Z Entering 'third_party/ittapi' 2024-08-21T03:09:38.7384934Z Entering 'third_party/kineto' 2024-08-21T03:09:38.7419779Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:09:38.7456144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:09:38.7501766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:09:38.7539339Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:09:38.7573395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:09:38.7605488Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:09:38.7652832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:09:38.7692217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:09:38.7736000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:09:38.7771304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:09:38.7809524Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:09:38.7844680Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:09:38.7882982Z Entering 'third_party/mimalloc' 2024-08-21T03:09:38.7921912Z Entering 'third_party/nccl/nccl' 2024-08-21T03:09:38.7959424Z Entering 'third_party/nlohmann' 2024-08-21T03:09:38.8011157Z Entering 'third_party/onnx' 2024-08-21T03:09:38.8068399Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:09:38.8110323Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:09:38.8158793Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:09:38.8199237Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:09:38.8235248Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:09:38.8271582Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:09:38.8313783Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:09:38.8354889Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:09:38.8397519Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:09:38.8437088Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:09:38.8474375Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:09:38.8516227Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:09:38.8554496Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:09:38.8614083Z Entering 'third_party/pocketfft' 2024-08-21T03:09:38.8655012Z Entering 'third_party/protobuf' 2024-08-21T03:09:38.8698410Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:09:38.8738307Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:09:38.8785682Z Entering 'third_party/psimd' 2024-08-21T03:09:38.8829529Z Entering 'third_party/pthreadpool' 2024-08-21T03:09:38.8863034Z Entering 'third_party/pybind11' 2024-08-21T03:09:38.8901515Z Entering 'third_party/python-peachpy' 2024-08-21T03:09:38.8934927Z Entering 'third_party/sleef' 2024-08-21T03:09:38.8977186Z Entering 'third_party/tensorpipe' 2024-08-21T03:09:38.9015819Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:09:38.9059205Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:09:38.9099441Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:09:38.9133776Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:09:38.9170807Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:09:38.9233018Z ##[endgroup] 2024-08-21T03:09:38.9277216Z [command]/usr/bin/git log -1 --format='%H' 2024-08-21T03:09:38.9303731Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-21T03:09:38.9582067Z Prepare all required actions 2024-08-21T03:09:38.9582957Z Getting action download info 2024-08-21T03:09:38.9730558Z ##[group]Run ./.github/actions/setup-rocm 2024-08-21T03:09:38.9731245Z env: 2024-08-21T03:09:38.9731730Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:38.9732303Z ##[endgroup] 2024-08-21T03:09:38.9771922Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-08-21T03:09:38.9773399Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-08-21T03:09:38.9816669Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:38.9817305Z env: 2024-08-21T03:09:38.9817658Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:38.9818081Z ##[endgroup] 2024-08-21T03:09:38.9964255Z ##[group]Run set -ex 2024-08-21T03:09:38.9964807Z set -ex 2024-08-21T03:09:38.9965279Z  2024-08-21T03:09:38.9965819Z cat ~/.docker/config.json || true 2024-08-21T03:09:38.9967780Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-08-21T03:09:38.9969861Z rm -f ~/.docker/config.json 2024-08-21T03:09:39.0013763Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.0014226Z env: 2024-08-21T03:09:39.0014458Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.0014817Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.0015204Z ##[endgroup] 2024-08-21T03:09:39.0081713Z + cat /home/pytorchci/.docker/config.json 2024-08-21T03:09:39.0094178Z { 2024-08-21T03:09:39.0094400Z "auths": {} 2024-08-21T03:09:39.0095649Z + rm -f /home/pytorchci/.docker/config.json 2024-08-21T03:09:39.0107470Z } 2024-08-21T03:09:39.0140490Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T03:09:39.0141163Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T03:09:39.0141675Z # shellcheck disable=SC2046 2024-08-21T03:09:39.0142090Z docker stop $(docker ps -q) || true 2024-08-21T03:09:39.0142504Z # Prune all stopped containers. 2024-08-21T03:09:39.0142884Z docker container prune -f 2024-08-21T03:09:39.0178963Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.0179967Z env: 2024-08-21T03:09:39.0180494Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.0181192Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.0181928Z ##[endgroup] 2024-08-21T03:09:39.0519577Z "docker stop" requires at least 1 argument. 2024-08-21T03:09:39.0520091Z See 'docker stop --help'. 2024-08-21T03:09:39.0520457Z 2024-08-21T03:09:39.0520659Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-21T03:09:39.0520980Z 2024-08-21T03:09:39.0521106Z Stop one or more running containers 2024-08-21T03:09:39.0673275Z Total reclaimed space: 0B 2024-08-21T03:09:39.0716549Z ##[group]Run cat /etc/os-release || true 2024-08-21T03:09:39.0716924Z cat /etc/os-release || true 2024-08-21T03:09:39.0717267Z cat /etc/apt/sources.list.d/rocm.list || true 2024-08-21T03:09:39.0717633Z cat /opt/rocm/.info/version || true 2024-08-21T03:09:39.0717929Z whoami 2024-08-21T03:09:39.0739563Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.0739920Z env: 2024-08-21T03:09:39.0740114Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.0740400Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.0740709Z ##[endgroup] 2024-08-21T03:09:39.0781663Z PRETTY_NAME="Ubuntu 22.04.3 LTS" 2024-08-21T03:09:39.0781951Z NAME="Ubuntu" 2024-08-21T03:09:39.0782157Z VERSION_ID="22.04" 2024-08-21T03:09:39.0782391Z VERSION="22.04.3 LTS (Jammy Jellyfish)" 2024-08-21T03:09:39.0782682Z VERSION_CODENAME=jammy 2024-08-21T03:09:39.0783787Z ID=ubuntu 2024-08-21T03:09:39.0783982Z ID_LIKE=debian 2024-08-21T03:09:39.0784269Z HOME_URL="https://www.ubuntu.com/" 2024-08-21T03:09:39.0784746Z SUPPORT_URL="https://help.ubuntu.com/" 2024-08-21T03:09:39.0785624Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-08-21T03:09:39.0786795Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-08-21T03:09:39.0787510Z UBUNTU_CODENAME=jammy 2024-08-21T03:09:39.0796606Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.1 jammy main 2024-08-21T03:09:39.0805919Z 6.1.0-82 2024-08-21T03:09:39.0816503Z pytorchci 2024-08-21T03:09:39.0837764Z ##[group]Run rocm-smi 2024-08-21T03:09:39.0838004Z rocm-smi 2024-08-21T03:09:39.0856007Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.0856351Z env: 2024-08-21T03:09:39.0856549Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.0856843Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.0857142Z ##[endgroup] 2024-08-21T03:09:39.1659406Z 2024-08-21T03:09:39.1659652Z 2024-08-21T03:09:39.1660274Z ====================================== ROCm System Management Interface ====================================== 2024-08-21T03:09:39.1660990Z ================================================ Concise Info ================================================ 2024-08-21T03:09:39.1661933Z Device [Model : Revision] Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-08-21T03:09:39.1663333Z  Name (20 chars) (Edge) (Avg) (Mem, Compute)  2024-08-21T03:09:39.1663989Z ============================================================================================================== 2024-08-21T03:09:39.1665381Z 0 [0x0c34 : 0x02] 46.0°C 43.0W N/A, N/A 800Mhz 1600Mhz 0% auto 300.0W 4% 0% 2024-08-21T03:09:39.1671116Z Instinct MI210 2024-08-21T03:09:39.1672053Z 1 [0x0c34 : 0x02] 46.0°C 43.0W N/A, N/A 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-08-21T03:09:39.1682213Z Instinct MI210 2024-08-21T03:09:39.1685155Z ============================================================================================================== 2024-08-21T03:09:39.1686336Z ============================================ End of ROCm SMI Log ============================================= 2024-08-21T03:09:39.1781326Z ##[group]Run rocminfo 2024-08-21T03:09:39.1781608Z rocminfo 2024-08-21T03:09:39.1816167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.1816606Z env: 2024-08-21T03:09:39.1816840Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.1817213Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.1817600Z ##[endgroup] 2024-08-21T03:09:39.2567803Z ROCk module version 6.7.0 is loaded 2024-08-21T03:09:39.2568603Z ===================== 2024-08-21T03:09:39.2569226Z HSA System Attributes 2024-08-21T03:09:39.2569881Z ===================== 2024-08-21T03:09:39.2570535Z Runtime Version: 1.13 2024-08-21T03:09:39.2571223Z Runtime Ext Version: 1.4 2024-08-21T03:09:39.2571850Z System Timestamp Freq.: 1000.000000MHz 2024-08-21T03:09:39.2572963Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-08-21T03:09:39.2574801Z Machine Model: LARGE 2024-08-21T03:09:39.2575822Z System Endianness: LITTLE 2024-08-21T03:09:39.2576603Z Mwaitx: DISABLED 2024-08-21T03:09:39.2576998Z DMAbuf Support: YES 2024-08-21T03:09:39.2577214Z 2024-08-21T03:09:39.2577317Z ========== 2024-08-21T03:09:39.2577591Z HSA Agents 2024-08-21T03:09:39.2577863Z ========== 2024-08-21T03:09:39.2578132Z ******* 2024-08-21T03:09:39.2578407Z Agent 1 2024-08-21T03:09:39.2578682Z ******* 2024-08-21T03:09:39.2579112Z Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:09:39.2580484Z Uuid: CPU-XX 2024-08-21T03:09:39.2581147Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:09:39.2581675Z Vendor Name: CPU 2024-08-21T03:09:39.2582249Z Feature: None specified 2024-08-21T03:09:39.2582739Z Profile: FULL_PROFILE 2024-08-21T03:09:39.2583245Z Float Round Mode: NEAR 2024-08-21T03:09:39.2583783Z Max Queue Number: 0(0x0) 2024-08-21T03:09:39.2584286Z Queue Min Size: 0(0x0) 2024-08-21T03:09:39.2584767Z Queue Max Size: 0(0x0) 2024-08-21T03:09:39.2585228Z Queue Type: MULTI 2024-08-21T03:09:39.2585670Z Node: 0 2024-08-21T03:09:39.2586130Z Device Type: CPU 2024-08-21T03:09:39.2586532Z Cache Info: 2024-08-21T03:09:39.2586884Z L1: 32768(0x8000) KB 2024-08-21T03:09:39.2587354Z Chip ID: 0(0x0) 2024-08-21T03:09:39.2587808Z ASIC Revision: 0(0x0) 2024-08-21T03:09:39.2588282Z Cacheline Size: 64(0x40) 2024-08-21T03:09:39.2588775Z Max Clock Freq. (MHz): 2600 2024-08-21T03:09:39.2589243Z BDFID: 0 2024-08-21T03:09:39.2589685Z Internal Node ID: 0 2024-08-21T03:09:39.2590153Z Compute Unit: 32 2024-08-21T03:09:39.2590613Z SIMDs per CU: 0 2024-08-21T03:09:39.2591075Z Shader Engines: 0 2024-08-21T03:09:39.2591556Z Shader Arrs. per Eng.: 0 2024-08-21T03:09:39.2592059Z WatchPts on Addr. Ranges:1 2024-08-21T03:09:39.2592492Z Features: None 2024-08-21T03:09:39.2592815Z Pool Info: 2024-08-21T03:09:39.2593114Z Pool 1 2024-08-21T03:09:39.2593507Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:09:39.2593998Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:09:39.2594494Z Allocatable: TRUE 2024-08-21T03:09:39.2594988Z Alloc Granule: 4KB 2024-08-21T03:09:39.2595500Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2596027Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2596540Z Accessible by all: TRUE 2024-08-21T03:09:39.2596963Z Pool 2 2024-08-21T03:09:39.2597357Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-21T03:09:39.2597840Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:09:39.2598551Z Allocatable: TRUE 2024-08-21T03:09:39.2599045Z Alloc Granule: 4KB 2024-08-21T03:09:39.2599558Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2600074Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2600691Z Accessible by all: TRUE 2024-08-21T03:09:39.2601113Z Pool 3 2024-08-21T03:09:39.2601504Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:09:39.2601973Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:09:39.2602601Z Allocatable: TRUE 2024-08-21T03:09:39.2603100Z Alloc Granule: 4KB 2024-08-21T03:09:39.2603607Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2604134Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2604641Z Accessible by all: TRUE 2024-08-21T03:09:39.2605046Z ISA Info: 2024-08-21T03:09:39.2605333Z ******* 2024-08-21T03:09:39.2605606Z Agent 2 2024-08-21T03:09:39.2605871Z ******* 2024-08-21T03:09:39.2606294Z Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:09:39.2606808Z Uuid: CPU-XX 2024-08-21T03:09:39.2607342Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:09:39.2607863Z Vendor Name: CPU 2024-08-21T03:09:39.2608328Z Feature: None specified 2024-08-21T03:09:39.2608810Z Profile: FULL_PROFILE 2024-08-21T03:09:39.2609310Z Float Round Mode: NEAR 2024-08-21T03:09:39.2609819Z Max Queue Number: 0(0x0) 2024-08-21T03:09:39.2610294Z Queue Min Size: 0(0x0) 2024-08-21T03:09:39.2610762Z Queue Max Size: 0(0x0) 2024-08-21T03:09:39.2611229Z Queue Type: MULTI 2024-08-21T03:09:39.2611689Z Node: 1 2024-08-21T03:09:39.2612131Z Device Type: CPU 2024-08-21T03:09:39.2612530Z Cache Info: 2024-08-21T03:09:39.2612887Z L1: 32768(0x8000) KB 2024-08-21T03:09:39.2613339Z Chip ID: 0(0x0) 2024-08-21T03:09:39.2613813Z ASIC Revision: 0(0x0) 2024-08-21T03:09:39.2614364Z Cacheline Size: 64(0x40) 2024-08-21T03:09:39.2614932Z Max Clock Freq. (MHz): 2600 2024-08-21T03:09:39.2615478Z BDFID: 0 2024-08-21T03:09:39.2616012Z Internal Node ID: 1 2024-08-21T03:09:39.2616476Z Compute Unit: 32 2024-08-21T03:09:39.2616935Z SIMDs per CU: 0 2024-08-21T03:09:39.2617408Z Shader Engines: 0 2024-08-21T03:09:39.2617879Z Shader Arrs. per Eng.: 0 2024-08-21T03:09:39.2618383Z WatchPts on Addr. Ranges:1 2024-08-21T03:09:39.2618811Z Features: None 2024-08-21T03:09:39.2619118Z Pool Info: 2024-08-21T03:09:39.2619418Z Pool 1 2024-08-21T03:09:39.2619809Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:09:39.2620469Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:09:39.2620952Z Allocatable: TRUE 2024-08-21T03:09:39.2621460Z Alloc Granule: 4KB 2024-08-21T03:09:39.2621981Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2622500Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2623003Z Accessible by all: TRUE 2024-08-21T03:09:39.2623420Z Pool 2 2024-08-21T03:09:39.2623810Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-21T03:09:39.2624450Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:09:39.2625177Z Allocatable: TRUE 2024-08-21T03:09:39.2626024Z Alloc Granule: 4KB 2024-08-21T03:09:39.2626911Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2627714Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2628429Z Accessible by all: TRUE 2024-08-21T03:09:39.2629054Z Pool 3 2024-08-21T03:09:39.2629608Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:09:39.2630294Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:09:39.2630972Z Allocatable: TRUE 2024-08-21T03:09:39.2631681Z Alloc Granule: 4KB 2024-08-21T03:09:39.2632428Z Alloc Recommended Granule:4KB 2024-08-21T03:09:39.2633199Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2633956Z Accessible by all: TRUE 2024-08-21T03:09:39.2634588Z ISA Info: 2024-08-21T03:09:39.2634997Z ******* 2024-08-21T03:09:39.2635401Z Agent 3 2024-08-21T03:09:39.2635891Z ******* 2024-08-21T03:09:39.2636374Z Name: gfx90a 2024-08-21T03:09:39.2637137Z Uuid: GPU-09623e4b9a2a2fde 2024-08-21T03:09:39.2637851Z Marketing Name: AMD Instinct MI210 2024-08-21T03:09:39.2638569Z Vendor Name: AMD 2024-08-21T03:09:39.2639270Z Feature: KERNEL_DISPATCH 2024-08-21T03:09:39.2639957Z Profile: BASE_PROFILE 2024-08-21T03:09:39.2640771Z Float Round Mode: NEAR 2024-08-21T03:09:39.2641499Z Max Queue Number: 128(0x80) 2024-08-21T03:09:39.2642194Z Queue Min Size: 64(0x40) 2024-08-21T03:09:39.2642898Z Queue Max Size: 131072(0x20000) 2024-08-21T03:09:39.2643584Z Queue Type: MULTI 2024-08-21T03:09:39.2644225Z Node: 2 2024-08-21T03:09:39.2644871Z Device Type: GPU 2024-08-21T03:09:39.2645456Z Cache Info: 2024-08-21T03:09:39.2645962Z L1: 16(0x10) KB 2024-08-21T03:09:39.2646606Z L2: 8192(0x2000) KB 2024-08-21T03:09:39.2647346Z Chip ID: 29711(0x740f) 2024-08-21T03:09:39.2648159Z ASIC Revision: 1(0x1) 2024-08-21T03:09:39.2649006Z Cacheline Size: 64(0x40) 2024-08-21T03:09:39.2649842Z Max Clock Freq. (MHz): 1700 2024-08-21T03:09:39.2650940Z BDFID: 768 2024-08-21T03:09:39.2651731Z Internal Node ID: 2 2024-08-21T03:09:39.2652367Z Compute Unit: 104 2024-08-21T03:09:39.2652847Z SIMDs per CU: 4 2024-08-21T03:09:39.2653309Z Shader Engines: 8 2024-08-21T03:09:39.2653796Z Shader Arrs. per Eng.: 1 2024-08-21T03:09:39.2654295Z WatchPts on Addr. Ranges:4 2024-08-21T03:09:39.2654787Z Coherent Host Access: FALSE 2024-08-21T03:09:39.2655394Z Features: KERNEL_DISPATCH 2024-08-21T03:09:39.2655859Z Fast F16 Operation: TRUE 2024-08-21T03:09:39.2656341Z Wavefront Size: 64(0x40) 2024-08-21T03:09:39.2656832Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:09:39.2657311Z Workgroup Max Size per Dimension: 2024-08-21T03:09:39.2657910Z x 1024(0x400) 2024-08-21T03:09:39.2658633Z y 1024(0x400) 2024-08-21T03:09:39.2659316Z z 1024(0x400) 2024-08-21T03:09:39.2659984Z Max Waves Per CU: 32(0x20) 2024-08-21T03:09:39.2660802Z Max Work-item Per CU: 2048(0x800) 2024-08-21T03:09:39.2661530Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:09:39.2662148Z Grid Max Size per Dimension: 2024-08-21T03:09:39.2662677Z x 4294967295(0xffffffff) 2024-08-21T03:09:39.2663298Z y 4294967295(0xffffffff) 2024-08-21T03:09:39.2663914Z z 4294967295(0xffffffff) 2024-08-21T03:09:39.2664641Z Max fbarriers/Workgrp: 32 2024-08-21T03:09:39.2681635Z Packet Processor uCode:: 82 2024-08-21T03:09:39.2682493Z SDMA engine uCode:: 8 2024-08-21T03:09:39.2683239Z IOMMU Support:: None 2024-08-21T03:09:39.2683841Z Pool Info: 2024-08-21T03:09:39.2684282Z Pool 1 2024-08-21T03:09:39.2684868Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:09:39.2685596Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:09:39.2686307Z Allocatable: TRUE 2024-08-21T03:09:39.2687099Z Alloc Granule: 4KB 2024-08-21T03:09:39.2688011Z Alloc Recommended Granule:2048KB 2024-08-21T03:09:39.2688927Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2689827Z Accessible by all: FALSE 2024-08-21T03:09:39.2690550Z Pool 2 2024-08-21T03:09:39.2691236Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-21T03:09:39.2692048Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:09:39.2692490Z Allocatable: TRUE 2024-08-21T03:09:39.2692898Z Alloc Granule: 4KB 2024-08-21T03:09:39.2693311Z Alloc Recommended Granule:2048KB 2024-08-21T03:09:39.2693730Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2694134Z Accessible by all: FALSE 2024-08-21T03:09:39.2694477Z Pool 3 2024-08-21T03:09:39.2694782Z Segment: GROUP 2024-08-21T03:09:39.2695353Z Size: 64(0x40) KB 2024-08-21T03:09:39.2695741Z Allocatable: FALSE 2024-08-21T03:09:39.2696147Z Alloc Granule: 0KB 2024-08-21T03:09:39.2696553Z Alloc Recommended Granule:0KB 2024-08-21T03:09:39.2697001Z Alloc Alignment: 0KB 2024-08-21T03:09:39.2697481Z Accessible by all: FALSE 2024-08-21T03:09:39.2697891Z ISA Info: 2024-08-21T03:09:39.2698175Z ISA 1 2024-08-21T03:09:39.2698814Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-21T03:09:39.2699312Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-21T03:09:39.2699731Z Profiles: HSA_PROFILE_BASE 2024-08-21T03:09:39.2700144Z Default Rounding Mode: NEAR 2024-08-21T03:09:39.2700580Z Default Rounding Mode: NEAR 2024-08-21T03:09:39.2700978Z Fast f16: TRUE 2024-08-21T03:09:39.2701386Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:09:39.2701760Z Workgroup Max Size per Dimension: 2024-08-21T03:09:39.2702083Z x 1024(0x400) 2024-08-21T03:09:39.2702440Z y 1024(0x400) 2024-08-21T03:09:39.2702787Z z 1024(0x400) 2024-08-21T03:09:39.2703170Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:09:39.2703529Z Grid Max Size per Dimension: 2024-08-21T03:09:39.2703834Z x 4294967295(0xffffffff) 2024-08-21T03:09:39.2704197Z y 4294967295(0xffffffff) 2024-08-21T03:09:39.2704562Z z 4294967295(0xffffffff) 2024-08-21T03:09:39.2704950Z FBarrier Max Size: 32 2024-08-21T03:09:39.2705283Z ******* 2024-08-21T03:09:39.2705506Z Agent 4 2024-08-21T03:09:39.2705726Z ******* 2024-08-21T03:09:39.2705994Z Name: gfx90a 2024-08-21T03:09:39.2706414Z Uuid: GPU-17f2a9b321b37cb8 2024-08-21T03:09:39.2706808Z Marketing Name: AMD Instinct MI210 2024-08-21T03:09:39.2707208Z Vendor Name: AMD 2024-08-21T03:09:39.2707583Z Feature: KERNEL_DISPATCH 2024-08-21T03:09:39.2707968Z Profile: BASE_PROFILE 2024-08-21T03:09:39.2708357Z Float Round Mode: NEAR 2024-08-21T03:09:39.2708748Z Max Queue Number: 128(0x80) 2024-08-21T03:09:39.2709137Z Queue Min Size: 64(0x40) 2024-08-21T03:09:39.2709512Z Queue Max Size: 131072(0x20000) 2024-08-21T03:09:39.2709879Z Queue Type: MULTI 2024-08-21T03:09:39.2710241Z Node: 3 2024-08-21T03:09:39.2710598Z Device Type: GPU 2024-08-21T03:09:39.2710914Z Cache Info: 2024-08-21T03:09:39.2711202Z L1: 16(0x10) KB 2024-08-21T03:09:39.2711561Z L2: 8192(0x2000) KB 2024-08-21T03:09:39.2711920Z Chip ID: 29711(0x740f) 2024-08-21T03:09:39.2712301Z ASIC Revision: 1(0x1) 2024-08-21T03:09:39.2714856Z Cacheline Size: 64(0x40) 2024-08-21T03:09:39.2715252Z Max Clock Freq. (MHz): 1700 2024-08-21T03:09:39.2715619Z BDFID: 33536 2024-08-21T03:09:39.2715989Z Internal Node ID: 3 2024-08-21T03:09:39.2716369Z Compute Unit: 104 2024-08-21T03:09:39.2716732Z SIMDs per CU: 4 2024-08-21T03:09:39.2717106Z Shader Engines: 8 2024-08-21T03:09:39.2717494Z Shader Arrs. per Eng.: 1 2024-08-21T03:09:39.2717990Z WatchPts on Addr. Ranges:4 2024-08-21T03:09:39.2718399Z Coherent Host Access: FALSE 2024-08-21T03:09:39.2718757Z Features: KERNEL_DISPATCH 2024-08-21T03:09:39.2719119Z Fast F16 Operation: TRUE 2024-08-21T03:09:39.2719522Z Wavefront Size: 64(0x40) 2024-08-21T03:09:39.2719919Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:09:39.2720313Z Workgroup Max Size per Dimension: 2024-08-21T03:09:39.2720630Z x 1024(0x400) 2024-08-21T03:09:39.2720969Z y 1024(0x400) 2024-08-21T03:09:39.2721299Z z 1024(0x400) 2024-08-21T03:09:39.2721670Z Max Waves Per CU: 32(0x20) 2024-08-21T03:09:39.2722125Z Max Work-item Per CU: 2048(0x800) 2024-08-21T03:09:39.2722536Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:09:39.2722871Z Grid Max Size per Dimension: 2024-08-21T03:09:39.2723164Z x 4294967295(0xffffffff) 2024-08-21T03:09:39.2723505Z y 4294967295(0xffffffff) 2024-08-21T03:09:39.2723837Z z 4294967295(0xffffffff) 2024-08-21T03:09:39.2724222Z Max fbarriers/Workgrp: 32 2024-08-21T03:09:39.2724633Z Packet Processor uCode:: 82 2024-08-21T03:09:39.2725026Z SDMA engine uCode:: 8 2024-08-21T03:09:39.2725429Z IOMMU Support:: None 2024-08-21T03:09:39.2725766Z Pool Info: 2024-08-21T03:09:39.2726004Z Pool 1 2024-08-21T03:09:39.2726327Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:09:39.2726733Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:09:39.2727119Z Allocatable: TRUE 2024-08-21T03:09:39.2727519Z Alloc Granule: 4KB 2024-08-21T03:09:39.2727949Z Alloc Recommended Granule:2048KB 2024-08-21T03:09:39.2728372Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2728771Z Accessible by all: FALSE 2024-08-21T03:09:39.2729107Z Pool 2 2024-08-21T03:09:39.2729426Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-21T03:09:39.2729813Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:09:39.2730185Z Allocatable: TRUE 2024-08-21T03:09:39.2730580Z Alloc Granule: 4KB 2024-08-21T03:09:39.2730992Z Alloc Recommended Granule:2048KB 2024-08-21T03:09:39.2731411Z Alloc Alignment: 4KB 2024-08-21T03:09:39.2731809Z Accessible by all: FALSE 2024-08-21T03:09:39.2732275Z Pool 3 2024-08-21T03:09:39.2732573Z Segment: GROUP 2024-08-21T03:09:39.2732937Z Size: 64(0x40) KB 2024-08-21T03:09:39.2733300Z Allocatable: FALSE 2024-08-21T03:09:39.2733696Z Alloc Granule: 0KB 2024-08-21T03:09:39.2734136Z Alloc Recommended Granule:0KB 2024-08-21T03:09:39.2734630Z Alloc Alignment: 0KB 2024-08-21T03:09:39.2735098Z Accessible by all: FALSE 2024-08-21T03:09:39.2735630Z ISA Info: 2024-08-21T03:09:39.2735919Z ISA 1 2024-08-21T03:09:39.2736316Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-21T03:09:39.2736747Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-21T03:09:39.2737166Z Profiles: HSA_PROFILE_BASE 2024-08-21T03:09:39.2737566Z Default Rounding Mode: NEAR 2024-08-21T03:09:39.2737980Z Default Rounding Mode: NEAR 2024-08-21T03:09:39.2738373Z Fast f16: TRUE 2024-08-21T03:09:39.2738765Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:09:39.2739125Z Workgroup Max Size per Dimension: 2024-08-21T03:09:39.2739456Z x 1024(0x400) 2024-08-21T03:09:39.2739808Z y 1024(0x400) 2024-08-21T03:09:39.2740162Z z 1024(0x400) 2024-08-21T03:09:39.2740544Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:09:39.2740894Z Grid Max Size per Dimension: 2024-08-21T03:09:39.2741209Z x 4294967295(0xffffffff) 2024-08-21T03:09:39.2741554Z y 4294967295(0xffffffff) 2024-08-21T03:09:39.2741922Z z 4294967295(0xffffffff) 2024-08-21T03:09:39.2742301Z FBarrier Max Size: 32 2024-08-21T03:09:39.2742644Z *** Done *** 2024-08-21T03:09:39.2842401Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-08-21T03:09:39.2842958Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-08-21T03:09:39.2843921Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-21T03:09:39.2844794Z if [[ $ngpu -eq 0 ]]; then 2024-08-21T03:09:39.2845290Z  echo "Error: Failed to detect any GPUs on the runner" 2024-08-21T03:09:39.2845759Z  echo "$msg" 2024-08-21T03:09:39.2846077Z  exit 1 2024-08-21T03:09:39.2846346Z fi 2024-08-21T03:09:39.2846603Z if [[ $ngpu -eq 1 ]]; then 2024-08-21T03:09:39.2847202Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-08-21T03:09:39.2847798Z  echo "$msg" 2024-08-21T03:09:39.2848080Z  exit 1 2024-08-21T03:09:39.2848347Z fi 2024-08-21T03:09:39.2889982Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.2890490Z env: 2024-08-21T03:09:39.2890732Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.2891092Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.2891486Z ##[endgroup] 2024-08-21T03:09:39.3934684Z Prepare all required actions 2024-08-21T03:09:39.3987720Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-08-21T03:09:39.3988465Z with: 2024-08-21T03:09:39.3988922Z diskspace-cutoff: 70 2024-08-21T03:09:39.3989445Z env: 2024-08-21T03:09:39.3990239Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.3990924Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.3991664Z ##[endgroup] 2024-08-21T03:09:39.4092632Z ##[group]Run set -ex 2024-08-21T03:09:39.4092933Z set -ex 2024-08-21T03:09:39.4093211Z diskspace_cutoff=70 2024-08-21T03:09:39.4093653Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-08-21T03:09:39.4094389Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-21T03:09:39.4095520Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-21T03:09:39.4096692Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-08-21T03:09:39.4097149Z  docker system prune -af 2024-08-21T03:09:39.4097780Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-21T03:09:39.4098521Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-08-21T03:09:39.4099306Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-08-21T03:09:39.4099950Z  echo "$msg" 2024-08-21T03:09:39.4100255Z  exit 1 2024-08-21T03:09:39.4100535Z  else 2024-08-21T03:09:39.4100878Z  difference=$((diskspace - diskspace_new)) 2024-08-21T03:09:39.4101402Z  echo "Diskspace saved: $difference percent" 2024-08-21T03:09:39.4101834Z  fi 2024-08-21T03:09:39.4102076Z fi 2024-08-21T03:09:39.4136768Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.4137214Z env: 2024-08-21T03:09:39.4137451Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.4137801Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.4138181Z ##[endgroup] 2024-08-21T03:09:39.4191833Z + diskspace_cutoff=70 2024-08-21T03:09:39.4197017Z ++ docker info -f '{{.DockerRootDir}}' 2024-08-21T03:09:39.4702791Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-08-21T03:09:39.4710858Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-08-21T03:09:39.4711647Z ++ sed -n 2p 2024-08-21T03:09:39.4712383Z ++ sed s/%// 2024-08-21T03:09:39.4716258Z ++ sed 's/ //' 2024-08-21T03:09:39.4739185Z + diskspace=22 2024-08-21T03:09:39.4740165Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-08-21T03:09:39.4741044Z + [[ 22 -ge 70 ]] 2024-08-21T03:09:39.4779364Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-21T03:09:39.4780018Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-21T03:09:39.4780591Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-21T03:09:39.4814294Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.4814976Z env: 2024-08-21T03:09:39.4815332Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.4815865Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.4816440Z ##[endgroup] 2024-08-21T03:09:39.4928614Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-08-21T03:09:39.4929538Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-08-21T03:09:39.4930570Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-08-21T03:09:39.4963250Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:39.4963699Z env: 2024-08-21T03:09:39.4963984Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.4964406Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.4964874Z ##[endgroup] 2024-08-21T03:09:39.5069097Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-08-21T03:09:39.5069681Z with: 2024-08-21T03:09:39.5070038Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-08-21T03:09:39.5070478Z aws-region: us-east-1 2024-08-21T03:09:39.5070725Z role-duration-seconds: 18000 2024-08-21T03:09:39.5071003Z audience: sts.amazonaws.com 2024-08-21T03:09:39.5071248Z env: 2024-08-21T03:09:39.5071435Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:39.5071729Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:39.5072265Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:39.5072754Z ##[endgroup] 2024-08-21T03:09:39.9511031Z Assuming role with OIDC 2024-08-21T03:09:40.2910375Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-08-21T03:09:40.4099538Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-08-21T03:09:40.4100375Z with: 2024-08-21T03:09:40.4100876Z mask-password: true 2024-08-21T03:09:40.4101492Z registry-type: private 2024-08-21T03:09:40.4102145Z skip-logout: false 2024-08-21T03:09:40.4102684Z env: 2024-08-21T03:09:40.4103156Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:40.4103879Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:40.4105222Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:40.4106481Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:40.4107139Z AWS_REGION: us-east-1 2024-08-21T03:09:40.4108032Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:40.4108933Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:40.4121560Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:40.4122180Z ##[endgroup] 2024-08-21T03:09:41.0047309Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8271355Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-21T03:09:41.8271847Z with: 2024-08-21T03:09:41.8272498Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8273259Z docker-build-dir: .ci/docker 2024-08-21T03:09:41.8273553Z working-directory: . 2024-08-21T03:09:41.8273912Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8274311Z force-push: false 2024-08-21T03:09:41.8274539Z env: 2024-08-21T03:09:41.8274755Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:41.8275064Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:41.8275634Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:41.8276157Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:41.8276439Z AWS_REGION: us-east-1 2024-08-21T03:09:41.8276800Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:41.8277172Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:41.8282304Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:41.8282557Z ##[endgroup] 2024-08-21T03:09:41.8298919Z ##[group]Run set -ex 2024-08-21T03:09:41.8299189Z set -ex 2024-08-21T03:09:41.8299422Z  2024-08-21T03:09:41.8299845Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-21T03:09:41.8300545Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-21T03:09:41.8301143Z # job could then download the pre-built image as usual 2024-08-21T03:09:41.8301695Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-21T03:09:41.8302207Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8302670Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8303075Z  2024-08-21T03:09:41.8303469Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-21T03:09:41.8303929Z  exit 0 2024-08-21T03:09:41.8304166Z else 2024-08-21T03:09:41.8304446Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8304991Z fi 2024-08-21T03:09:41.8305205Z  2024-08-21T03:09:41.8305558Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-21T03:09:41.8306186Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-21T03:09:41.8306754Z  # use it as it is, but first let's extract the tag 2024-08-21T03:09:41.8307268Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-21T03:09:41.8307794Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8308308Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8308723Z else 2024-08-21T03:09:41.8309050Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-21T03:09:41.8309526Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8310171Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8310738Z fi 2024-08-21T03:09:41.8331941Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:41.8332319Z env: 2024-08-21T03:09:41.8332534Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:41.8332850Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:41.8333410Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:41.8333930Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:41.8334223Z AWS_REGION: us-east-1 2024-08-21T03:09:41.8334529Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:41.8334902Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:41.8340517Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:41.8340795Z REPO_NAME: pytorch 2024-08-21T03:09:41.8341478Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8342208Z DOCKER_BUILD_DIR: .ci/docker 2024-08-21T03:09:41.8342582Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8342966Z ##[endgroup] 2024-08-21T03:09:41.8386631Z + [[ ! -d .ci/docker ]] 2024-08-21T03:09:41.8387251Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-21T03:09:41.8387770Z + echo skip=false 2024-08-21T03:09:41.8389873Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-21T03:09:41.8396472Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8397832Z ++ awk -F '[:,]' '{print $2}' 2024-08-21T03:09:41.8417372Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8418384Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8421570Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8478516Z ##[group]Run set +e 2024-08-21T03:09:41.8479182Z set +e 2024-08-21T03:09:41.8479711Z set -x 2024-08-21T03:09:41.8480349Z  2024-08-21T03:09:41.8480839Z login() { 2024-08-21T03:09:41.8482035Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-21T03:09:41.8483328Z } 2024-08-21T03:09:41.8483816Z  2024-08-21T03:09:41.8484302Z retry () { 2024-08-21T03:09:41.8485030Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-21T03:09:41.8485820Z } 2024-08-21T03:09:41.8486293Z  2024-08-21T03:09:41.8486842Z retry login "${DOCKER_REGISTRY}" 2024-08-21T03:09:41.8487698Z  2024-08-21T03:09:41.8488682Z # Check if image already exists, if it does then skip building it 2024-08-21T03:09:41.8490680Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-21T03:09:41.8491726Z  exit 0 2024-08-21T03:09:41.8492347Z fi 2024-08-21T03:09:41.8492882Z  2024-08-21T03:09:41.8493459Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-21T03:09:41.8494227Z # be empty. The default action would be to continue rebuild the image 2024-08-21T03:09:41.8494904Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-21T03:09:41.8495524Z  # if we're on the base branch then use the parent commit 2024-08-21T03:09:41.8496086Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-21T03:09:41.8496481Z else 2024-08-21T03:09:41.8496915Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-21T03:09:41.8497545Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-21T03:09:41.8498014Z fi 2024-08-21T03:09:41.8498301Z  2024-08-21T03:09:41.8498649Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-21T03:09:41.8499665Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8500606Z  2024-08-21T03:09:41.8501843Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-21T03:09:41.8503188Z  exit 0 2024-08-21T03:09:41.8503711Z fi 2024-08-21T03:09:41.8504184Z  2024-08-21T03:09:41.8504947Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-21T03:09:41.8507072Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-21T03:09:41.8508542Z  exit 1 2024-08-21T03:09:41.8509066Z fi 2024-08-21T03:09:41.8509539Z  2024-08-21T03:09:41.8510403Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-21T03:09:41.8512103Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-21T03:09:41.8513671Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-21T03:09:41.8515416Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-21T03:09:41.8517366Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-21T03:09:41.8518501Z fi 2024-08-21T03:09:41.8518982Z  2024-08-21T03:09:41.8519606Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:41.8563566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:41.8564476Z env: 2024-08-21T03:09:41.8564977Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:41.8565853Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:41.8567455Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:41.8568969Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:41.8569788Z AWS_REGION: us-east-1 2024-08-21T03:09:41.8570515Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:41.8571057Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:41.8578623Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:41.8579266Z DOCKER_BUILD_DIR: .ci/docker 2024-08-21T03:09:41.8580097Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:09:41.8581977Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8583818Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:41.8584898Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8585976Z ##[endgroup] 2024-08-21T03:09:41.8655139Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8655745Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:41.8658497Z + aws ecr get-login-password --region us-east-1 2024-08-21T03:09:41.8659890Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:43.2513758Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-08-21T03:09:43.2515294Z Configure a credential helper to remove this warning. See 2024-08-21T03:09:43.2516927Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-21T03:09:43.2517865Z 2024-08-21T03:09:43.2518085Z Login Succeeded 2024-08-21T03:09:43.2554695Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:44.6026565Z { 2024-08-21T03:09:44.6027476Z "schemaVersion": 2, 2024-08-21T03:09:44.6028534Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-21T03:09:44.6029937Z "config": { 2024-08-21T03:09:44.6030766Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-21T03:09:44.6031746Z "size": 23393, 2024-08-21T03:09:44.6032717Z "digest": "sha256:89b1fce3ca1e521a81a11b09c8916745293c11b95b183947aa60747dd56e931f" 2024-08-21T03:09:44.6033798Z }, 2024-08-21T03:09:44.6034242Z "layers": [ 2024-08-21T03:09:44.6034727Z { 2024-08-21T03:09:44.6035483Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6036473Z "size": 28584223, 2024-08-21T03:09:44.6037411Z "digest": "sha256:560c024910bebac6b404791af28ebd48a8289303b8377d17b67ffdfe52754f2a" 2024-08-21T03:09:44.6038437Z }, 2024-08-21T03:09:44.6038831Z { 2024-08-21T03:09:44.6040582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6041659Z "size": 1822, 2024-08-21T03:09:44.6042610Z "digest": "sha256:adc140d8bb4958da5af5f0a199a9d8e7654beec0a119b03f8a8b55b1d1db77db" 2024-08-21T03:09:44.6043642Z }, 2024-08-21T03:09:44.6044038Z { 2024-08-21T03:09:44.6044745Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6045649Z "size": 312575450, 2024-08-21T03:09:44.6046574Z "digest": "sha256:10b1a547d6d14e48b479888e2ac03addd47deb1a5b09b0754950c69bb8392345" 2024-08-21T03:09:44.6047586Z }, 2024-08-21T03:09:44.6047981Z { 2024-08-21T03:09:44.6048684Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6049588Z "size": 804, 2024-08-21T03:09:44.6050472Z "digest": "sha256:968b3429655e3d61cbd5eaa6617d34f5b15b65f6aed68c15cebe6adfc7deffdb" 2024-08-21T03:09:44.6051489Z }, 2024-08-21T03:09:44.6051881Z { 2024-08-21T03:09:44.6052597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6053538Z "size": 106, 2024-08-21T03:09:44.6054467Z "digest": "sha256:1c80fa2ef2487c309cbbbfd0cf30b813fdca0c473f4629a8899de02bf5e481e8" 2024-08-21T03:09:44.6055505Z }, 2024-08-21T03:09:44.6055900Z { 2024-08-21T03:09:44.6056605Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6057665Z "size": 544, 2024-08-21T03:09:44.6058810Z "digest": "sha256:aab0246dd5e6ef59bacce7ed901d6c067b74aaa1b876b4af9601973259afcd18" 2024-08-21T03:09:44.6059898Z }, 2024-08-21T03:09:44.6060313Z { 2024-08-21T03:09:44.6061055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6062208Z "size": 1260, 2024-08-21T03:09:44.6063147Z "digest": "sha256:9f3064f84f02d6108e40afa561a87b4000446a7ed5d3d591df0fd3129af7d66a" 2024-08-21T03:09:44.6064195Z }, 2024-08-21T03:09:44.6064603Z { 2024-08-21T03:09:44.6065333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6066273Z "size": 3390, 2024-08-21T03:09:44.6067237Z "digest": "sha256:5ea6f7933401ced22b172cc141d60b912d8b6de0706dccd36f82b99dc4b78124" 2024-08-21T03:09:44.6068301Z }, 2024-08-21T03:09:44.6068721Z { 2024-08-21T03:09:44.6069467Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6070893Z "size": 1909, 2024-08-21T03:09:44.6071944Z "digest": "sha256:6f3dd4e0602685a51ac26eb509df7cfcd024eaa0cda962d1ab8a9f9d972a7bd0" 2024-08-21T03:09:44.6073024Z }, 2024-08-21T03:09:44.6073445Z { 2024-08-21T03:09:44.6074199Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6075134Z "size": 701, 2024-08-21T03:09:44.6076064Z "digest": "sha256:8460b8ecc249a61b01984d14a2cc5f800488d0f7dbb6be3b4d149737d9393be5" 2024-08-21T03:09:44.6077113Z }, 2024-08-21T03:09:44.6077525Z { 2024-08-21T03:09:44.6078260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6079206Z "size": 2518634113, 2024-08-21T03:09:44.6080288Z "digest": "sha256:25df6ea774889a11232c0186a0edc6930586f5a377a450b09c908253019bfdc1" 2024-08-21T03:09:44.6081314Z }, 2024-08-21T03:09:44.6081724Z { 2024-08-21T03:09:44.6082448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6083379Z "size": 380, 2024-08-21T03:09:44.6084325Z "digest": "sha256:11186a70bc632c16e089051cc26a3a6719eeddf0d414f98251f96a91bb86a97c" 2024-08-21T03:09:44.6085384Z }, 2024-08-21T03:09:44.6085808Z { 2024-08-21T03:09:44.6086560Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6087515Z "size": 12127, 2024-08-21T03:09:44.6088447Z "digest": "sha256:98501b418069d2a003344f96d50f1a4cb1641faebe7765a06624f8398fcd5885" 2024-08-21T03:09:44.6089491Z }, 2024-08-21T03:09:44.6089916Z { 2024-08-21T03:09:44.6090661Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6091602Z "size": 503, 2024-08-21T03:09:44.6092870Z "digest": "sha256:b3bb152b37362be7d52ffac6fe500abdcd68c35670773a6b7039e4410f210f61" 2024-08-21T03:09:44.6093956Z }, 2024-08-21T03:09:44.6094373Z { 2024-08-21T03:09:44.6095105Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6096055Z "size": 121477532, 2024-08-21T03:09:44.6097043Z "digest": "sha256:86e384e02fbfa7952717d87cfa1c0421fa3743a6c51f0b50e370813ce23375c4" 2024-08-21T03:09:44.6098129Z }, 2024-08-21T03:09:44.6098543Z { 2024-08-21T03:09:44.6099283Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6100226Z "size": 109, 2024-08-21T03:09:44.6101174Z "digest": "sha256:59ef70eb96dcb639c6b5bd83377ab51ad1056848ea6094bbbf126fb92196e96e" 2024-08-21T03:09:44.6102249Z }, 2024-08-21T03:09:44.6102668Z { 2024-08-21T03:09:44.6103411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6104369Z "size": 488, 2024-08-21T03:09:44.6105287Z "digest": "sha256:e7b60c94055030c47c467fe32c9f240f38cc375600a430fd4249123e4770ca05" 2024-08-21T03:09:44.6106356Z }, 2024-08-21T03:09:44.6106769Z { 2024-08-21T03:09:44.6107505Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6108475Z "size": 296, 2024-08-21T03:09:44.6109433Z "digest": "sha256:bd8e43536d18f708597ade75d6ef265048de72b4cd8f8389f88dad23c08111a0" 2024-08-21T03:09:44.6110526Z }, 2024-08-21T03:09:44.6110944Z { 2024-08-21T03:09:44.6111694Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6112642Z "size": 103, 2024-08-21T03:09:44.6113593Z "digest": "sha256:0ec75ea7f23f0cc2d91d75e35ad5be9972c22e38b880b2f61db5559d1e7c634c" 2024-08-21T03:09:44.6114683Z }, 2024-08-21T03:09:44.6115102Z { 2024-08-21T03:09:44.6115840Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6116797Z "size": 1473, 2024-08-21T03:09:44.6117756Z "digest": "sha256:bea45dbe8894b7630b6de998adf6c47506a46b577c4beacf11167ee7b728ad2c" 2024-08-21T03:09:44.6118844Z }, 2024-08-21T03:09:44.6119265Z { 2024-08-21T03:09:44.6119997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6121070Z "size": 592973916, 2024-08-21T03:09:44.6122034Z "digest": "sha256:a326ca949f681959c0726e28c9cb812664b4f1a0c9a8fbe033e571d0995a5bdc" 2024-08-21T03:09:44.6123400Z }, 2024-08-21T03:09:44.6123813Z { 2024-08-21T03:09:44.6124548Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6125487Z "size": 163, 2024-08-21T03:09:44.6126427Z "digest": "sha256:eaed78ff7978a9fdc73c5edc69f691bd607e399234acc5a732b2c3b369d3297b" 2024-08-21T03:09:44.6127507Z }, 2024-08-21T03:09:44.6127926Z { 2024-08-21T03:09:44.6128655Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6129592Z "size": 1638, 2024-08-21T03:09:44.6130524Z "digest": "sha256:5636a85a0644ae75dfdd063330f907cb3254bd43bd150be122e152bd7a40dc6f" 2024-08-21T03:09:44.6131579Z }, 2024-08-21T03:09:44.6131987Z { 2024-08-21T03:09:44.6132728Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6133661Z "size": 6892252452, 2024-08-21T03:09:44.6134629Z "digest": "sha256:249ef9eea6138d46b925866ba97868d9a7a5b16f3dd2fc36c5477dd31c8e6213" 2024-08-21T03:09:44.6135694Z }, 2024-08-21T03:09:44.6136113Z { 2024-08-21T03:09:44.6136838Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6137775Z "size": 105, 2024-08-21T03:09:44.6138698Z "digest": "sha256:f0aa54fd67039a900c631e7bd0b35d244c7fa5769e7ffde45035ffe3e2dab2bf" 2024-08-21T03:09:44.6139753Z }, 2024-08-21T03:09:44.6140163Z { 2024-08-21T03:09:44.6140886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6141810Z "size": 975, 2024-08-21T03:09:44.6142778Z "digest": "sha256:fa2cf33cab305aab8bdbfa9d8bdf827ec32172ff5e17218d30fb348bb9d37541" 2024-08-21T03:09:44.6143885Z }, 2024-08-21T03:09:44.6144299Z { 2024-08-21T03:09:44.6145328Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6146311Z "size": 320951411, 2024-08-21T03:09:44.6147254Z "digest": "sha256:3b93495907dc86ef378dae7ec197825a74237998e4075e9c9cdd806478f57cbe" 2024-08-21T03:09:44.6148298Z }, 2024-08-21T03:09:44.6148712Z { 2024-08-21T03:09:44.6149450Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6150384Z "size": 111, 2024-08-21T03:09:44.6151289Z "digest": "sha256:1dc84102243c71ec2a67f0d6e76d27330ad358088e03a7efac8e0f4897bda570" 2024-08-21T03:09:44.6152336Z }, 2024-08-21T03:09:44.6152749Z { 2024-08-21T03:09:44.6153465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6154388Z "size": 165, 2024-08-21T03:09:44.6155308Z "digest": "sha256:4bc902cbefd3433a7700582aeee0a2144d18ab878f96072f4beb2527671d43bf" 2024-08-21T03:09:44.6156379Z }, 2024-08-21T03:09:44.6156789Z { 2024-08-21T03:09:44.6157511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6158474Z "size": 3892797, 2024-08-21T03:09:44.6159423Z "digest": "sha256:93c646cacff5607899f60f4fceef4959846ff19fcfe88c135d6604914816b483" 2024-08-21T03:09:44.6160575Z }, 2024-08-21T03:09:44.6160994Z { 2024-08-21T03:09:44.6161723Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6162655Z "size": 107, 2024-08-21T03:09:44.6163597Z "digest": "sha256:5ed0f9bd8591bf4c31b08e99d6cdc708b29750eef4c4326ac672e787cc84a7e4" 2024-08-21T03:09:44.6164661Z }, 2024-08-21T03:09:44.6165071Z { 2024-08-21T03:09:44.6165797Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6166725Z "size": 564, 2024-08-21T03:09:44.6167622Z "digest": "sha256:1de68e32f431a3c823508da3f2257a29f042a4791332b8ae35336a5e52f33935" 2024-08-21T03:09:44.6168674Z }, 2024-08-21T03:09:44.6169088Z { 2024-08-21T03:09:44.6169812Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6170760Z "size": 43195627, 2024-08-21T03:09:44.6171726Z "digest": "sha256:9c69923981ffa472ceaabc7a31857883bbfbd291f02157c54c642afc35b17d90" 2024-08-21T03:09:44.6172786Z }, 2024-08-21T03:09:44.6173198Z { 2024-08-21T03:09:44.6173921Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6175188Z "size": 106, 2024-08-21T03:09:44.6176129Z "digest": "sha256:e4a0f7e7f19d4d397735eda013b29f48f4834a090ad03bffb6f6f5b62f67ab13" 2024-08-21T03:09:44.6177205Z }, 2024-08-21T03:09:44.6177615Z { 2024-08-21T03:09:44.6178354Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6179296Z "size": 294, 2024-08-21T03:09:44.6180233Z "digest": "sha256:da48c85fcfb44c23b72f080fa7585fe83f3514d98edd3c93ad32aefc007dbe02" 2024-08-21T03:09:44.6181311Z }, 2024-08-21T03:09:44.6181723Z { 2024-08-21T03:09:44.6182447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6183386Z "size": 88290, 2024-08-21T03:09:44.6184351Z "digest": "sha256:8aa36fbfb6172da62794cc2cd55d6b56927f052d91b888f539ede156cef2bd66" 2024-08-21T03:09:44.6185424Z }, 2024-08-21T03:09:44.6185834Z { 2024-08-21T03:09:44.6186565Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6187501Z "size": 106, 2024-08-21T03:09:44.6188442Z "digest": "sha256:cab524a4584434cb1dbffc7fcb8cb5376aa730bda4c7485ecf649487280ac7ea" 2024-08-21T03:09:44.6189524Z }, 2024-08-21T03:09:44.6189942Z { 2024-08-21T03:09:44.6190671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6191589Z "size": 1379, 2024-08-21T03:09:44.6192530Z "digest": "sha256:84ab4d27e41ec5584747b5eff61e28d481f6191b4109fe0999c8bc1238ea29e2" 2024-08-21T03:09:44.6193596Z }, 2024-08-21T03:09:44.6194011Z { 2024-08-21T03:09:44.6194731Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6195665Z "size": 701, 2024-08-21T03:09:44.6196883Z "digest": "sha256:8460b8ecc249a61b01984d14a2cc5f800488d0f7dbb6be3b4d149737d9393be5" 2024-08-21T03:09:44.6197964Z }, 2024-08-21T03:09:44.6198377Z { 2024-08-21T03:09:44.6199109Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6200059Z "size": 142, 2024-08-21T03:09:44.6201073Z "digest": "sha256:f46c4b1feb2ff227e542ae4cbbdc454231e870b4076c00a5f330462a3a341d57" 2024-08-21T03:09:44.6202160Z }, 2024-08-21T03:09:44.6202578Z { 2024-08-21T03:09:44.6203297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6204230Z "size": 119, 2024-08-21T03:09:44.6205154Z "digest": "sha256:5a35a60de089ab202a69fa12eb32b77865215cd9888c35fbef65b7a6946dfd69" 2024-08-21T03:09:44.6206213Z }, 2024-08-21T03:09:44.6206625Z { 2024-08-21T03:09:44.6207371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6208314Z "size": 4403652906, 2024-08-21T03:09:44.6209271Z "digest": "sha256:609425ba242f8f593be9203a838f9de6900e6bd39c720c23c1834d7693333557" 2024-08-21T03:09:44.6210330Z }, 2024-08-21T03:09:44.6210747Z { 2024-08-21T03:09:44.6211467Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6212393Z "size": 180, 2024-08-21T03:09:44.6213344Z "digest": "sha256:3d5f39a6f9e0ba9f949dc7dca3b66be34fc675ab3360aaed3b53f3e3b2fd8c77" 2024-08-21T03:09:44.6214432Z }, 2024-08-21T03:09:44.6214849Z { 2024-08-21T03:09:44.6215573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6216502Z "size": 208, 2024-08-21T03:09:44.6217450Z "digest": "sha256:5af385ad82da6ef4fcbb862f15f1e84b1bea881f9180e42eb0ec0a2801957c88" 2024-08-21T03:09:44.6218526Z }, 2024-08-21T03:09:44.6218936Z { 2024-08-21T03:09:44.6219663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6220586Z "size": 701, 2024-08-21T03:09:44.6221499Z "digest": "sha256:8460b8ecc249a61b01984d14a2cc5f800488d0f7dbb6be3b4d149737d9393be5" 2024-08-21T03:09:44.6222552Z }, 2024-08-21T03:09:44.6222963Z { 2024-08-21T03:09:44.6223698Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6224632Z "size": 632, 2024-08-21T03:09:44.6225557Z "digest": "sha256:a65259a40d7ceff50e49bfcab06ded50a34214fa46999c4ab9be80888a430341" 2024-08-21T03:09:44.6226952Z }, 2024-08-21T03:09:44.6227365Z { 2024-08-21T03:09:44.6228085Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6229035Z "size": 229509078, 2024-08-21T03:09:44.6229998Z "digest": "sha256:73bc9ab6d300d7063c88e9c24149749e2b67c359f0a5a5ad5d8b9f1217cf9343" 2024-08-21T03:09:44.6231056Z }, 2024-08-21T03:09:44.6231468Z { 2024-08-21T03:09:44.6232182Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6233106Z "size": 1841, 2024-08-21T03:09:44.6234058Z "digest": "sha256:6b227ee02daf104c53c83335e59ebe7abbddddd124e4184df88c410ed98e416a" 2024-08-21T03:09:44.6235135Z }, 2024-08-21T03:09:44.6235556Z { 2024-08-21T03:09:44.6236284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-21T03:09:44.6237223Z "size": 15982996, 2024-08-21T03:09:44.6238205Z "digest": "sha256:712ae1d160e2ed7a6cfcb6ce1890d7db8608393ffa32a0f971c021a3f8a87045" 2024-08-21T03:09:44.6239267Z } 2024-08-21T03:09:44.6239689Z ] 2024-08-21T03:09:44.6240203Z } 2024-08-21T03:09:44.6240654Z + exit 0 2024-08-21T03:09:44.6480611Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-21T03:09:44.6481733Z with: 2024-08-21T03:09:44.6483251Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:44.6485266Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:44.6486185Z env: 2024-08-21T03:09:44.6486682Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:44.6487342Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:44.6488355Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:44.6489323Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:44.6489838Z AWS_REGION: us-east-1 2024-08-21T03:09:44.6490504Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:44.6491193Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:44.6500555Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:44.6501014Z ##[endgroup] 2024-08-21T03:09:44.6527005Z ##[group]Run set -x 2024-08-21T03:09:44.6527471Z set -x 2024-08-21T03:09:44.6527876Z set +e 2024-08-21T03:09:44.6528255Z  2024-08-21T03:09:44.6528629Z login() { 2024-08-21T03:09:44.6529539Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-21T03:09:44.6530503Z } 2024-08-21T03:09:44.6530875Z  2024-08-21T03:09:44.6531252Z retry () { 2024-08-21T03:09:44.6531780Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-21T03:09:44.6532370Z } 2024-08-21T03:09:44.6532741Z  2024-08-21T03:09:44.6533171Z retry login "${DOCKER_REGISTRY}" 2024-08-21T03:09:44.6533728Z  2024-08-21T03:09:44.6534082Z set -e 2024-08-21T03:09:44.6534708Z # ignore output since only exit code is used for conditional 2024-08-21T03:09:44.6535644Z # only pull docker image if it's not available locally 2024-08-21T03:09:44.6536674Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-21T03:09:44.6537593Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-21T03:09:44.6538169Z fi 2024-08-21T03:09:44.6572965Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:44.6573642Z env: 2024-08-21T03:09:44.6574020Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:44.6574588Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:44.6575589Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:44.6576575Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:44.6577083Z AWS_REGION: us-east-1 2024-08-21T03:09:44.6577631Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:44.6578296Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:44.6587657Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:44.6588899Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:44.6590644Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:44.6591332Z ##[endgroup] 2024-08-21T03:09:44.6652130Z + set +e 2024-08-21T03:09:44.6652630Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:44.6653114Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:44.6657970Z + aws ecr get-login-password --region us-east-1 2024-08-21T03:09:44.6659406Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:09:46.0736201Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-08-21T03:09:46.0737805Z Configure a credential helper to remove this warning. See 2024-08-21T03:09:46.0739468Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-21T03:09:46.0740415Z 2024-08-21T03:09:46.0747645Z Login Succeeded 2024-08-21T03:09:46.0774949Z + set -e 2024-08-21T03:09:46.0779621Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:09:46.1065583Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-21T03:09:46.1066182Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-21T03:09:46.1066709Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-21T03:09:46.1067202Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-21T03:09:46.1089118Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:09:46.1089492Z env: 2024-08-21T03:09:46.1089710Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:46.1090029Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:46.1090602Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:46.1091151Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:46.1091440Z AWS_REGION: us-east-1 2024-08-21T03:09:46.1091775Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:46.1092157Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:46.1097620Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:46.1097931Z ##[endgroup] 2024-08-21T03:09:46.3633891Z Defaulting to user installation because normal site-packages is not writeable 2024-08-21T03:09:46.4350062Z Requirement already satisfied: psutil==5.9.1 in /home/pytorchci/.local/lib/python3.10/site-packages (5.9.1) 2024-08-21T03:09:46.4354580Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/pytorchci/.local/lib/python3.10/site-packages (11.525.84) 2024-08-21T03:09:46.5775025Z Prepare all required actions 2024-08-21T03:09:46.5775553Z Getting action download info 2024-08-21T03:09:46.7389429Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-21T03:09:47.3469391Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-21T03:09:47.8781347Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-21T03:09:47.8781844Z with: 2024-08-21T03:09:47.8782166Z name: linux-focal-rocm6.1-py3.8 2024-08-21T03:09:47.8782561Z s3-bucket: gha-artifacts 2024-08-21T03:09:47.8782944Z env: 2024-08-21T03:09:47.8783248Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:47.8783664Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:47.8784351Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:47.8784972Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:47.8785548Z AWS_REGION: us-east-1 2024-08-21T03:09:47.8785970Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:47.8786451Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:47.8791824Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:47.8792256Z ##[endgroup] 2024-08-21T03:09:47.8843897Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-21T03:09:47.8844686Z with: 2024-08-21T03:09:47.8845015Z name: linux-focal-rocm6.1-py3.8 2024-08-21T03:09:47.8845463Z s3-bucket: gha-artifacts 2024-08-21T03:09:47.8845820Z region: us-east-1 2024-08-21T03:09:47.8846143Z env: 2024-08-21T03:09:47.8846515Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:09:47.8846914Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:09:47.8847586Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:09:47.8848234Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:09:47.8848621Z AWS_REGION: us-east-1 2024-08-21T03:09:47.8849086Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:09:47.8849558Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:09:47.8854792Z AWS_SESSION_TOKEN: *** 2024-08-21T03:09:47.8855163Z ##[endgroup] 2024-08-21T03:09:48.2848746Z (node:1761295) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-21T03:09:48.2849587Z 2024-08-21T03:09:48.2849897Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-21T03:09:48.2850827Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-21T03:09:48.2852000Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-21T03:09:48.5671274Z Found 1 objects with prefix pytorch/pytorch/10479310961/linux-focal-rocm6.1-py3.8/ 2024-08-21T03:09:48.5673571Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-21T03:10:23.4806494Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-21T03:10:23.4821850Z Artifact download has finished successfully 2024-08-21T03:10:23.5547221Z ##[group]Run unzip -o artifacts.zip 2024-08-21T03:10:23.5547847Z unzip -o artifacts.zip 2024-08-21T03:10:23.5584212Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:23.5584917Z env: 2024-08-21T03:10:23.5585304Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:23.5585889Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:23.5586927Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:23.5587905Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:23.5588427Z AWS_REGION: us-east-1 2024-08-21T03:10:23.5588995Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:23.5589693Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:23.5599501Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:23.5599977Z ##[endgroup] 2024-08-21T03:10:23.5810130Z Archive: artifacts.zip 2024-08-21T03:10:23.5811322Z creating: dist/ 2024-08-21T03:10:24.9821972Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-21T03:10:24.9953446Z inflating: dist/.ninja_log 2024-08-21T03:10:24.9953930Z creating: build/custom_test_artifacts/ 2024-08-21T03:10:24.9954535Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-21T03:10:24.9957019Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-21T03:10:24.9957882Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-21T03:10:24.9958793Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-21T03:10:24.9959653Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-21T03:10:24.9960623Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-21T03:10:24.9961552Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-21T03:10:24.9962458Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-21T03:10:24.9963484Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-21T03:10:24.9964516Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-21T03:10:24.9965433Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-21T03:10:24.9967225Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-21T03:10:24.9968294Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-21T03:10:24.9969350Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-21T03:10:24.9970391Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-21T03:10:24.9971428Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-21T03:10:24.9972488Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-21T03:10:24.9973570Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-21T03:10:24.9974474Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-21T03:10:24.9975261Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-21T03:10:24.9976086Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-21T03:10:24.9976926Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-21T03:10:24.9977856Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-21T03:10:24.9978900Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-21T03:10:24.9979908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-21T03:10:24.9980849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-21T03:10:24.9981825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-21T03:10:24.9982805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-21T03:10:24.9983789Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-21T03:10:24.9984762Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-21T03:10:24.9985721Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-21T03:10:24.9996877Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-21T03:10:25.0128506Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-21T03:10:25.0129549Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-21T03:10:25.0130569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-21T03:10:25.0131720Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-21T03:10:25.0132797Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-21T03:10:25.0133805Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-21T03:10:25.0134840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-21T03:10:25.0135872Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-21T03:10:25.0136918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-21T03:10:25.0137945Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-21T03:10:25.0138969Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-21T03:10:25.0152173Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-21T03:10:25.0228406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-21T03:10:25.0229573Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-21T03:10:25.0230584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-21T03:10:25.0231498Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-21T03:10:25.0232351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-21T03:10:25.0233167Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-21T03:10:25.0233989Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2024-08-21T03:10:25.0234783Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-08-21T03:10:25.0235541Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-21T03:10:25.0236279Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-21T03:10:25.0237030Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-21T03:10:25.0342807Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-21T03:10:25.0399391Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-21T03:10:25.0400231Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-21T03:10:25.0400890Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-21T03:10:25.0401645Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-21T03:10:25.0403837Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-21T03:10:25.0404716Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-21T03:10:25.0405581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-21T03:10:25.0406478Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-21T03:10:25.0407351Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-21T03:10:25.0408340Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-21T03:10:25.0409557Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-21T03:10:25.0410479Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-21T03:10:25.0411368Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-21T03:10:25.0412420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-21T03:10:25.0413458Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-21T03:10:25.0414481Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-21T03:10:25.0415506Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-21T03:10:25.0416551Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-21T03:10:25.0417605Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-21T03:10:25.0418486Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-21T03:10:25.0419256Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-21T03:10:25.0420228Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-21T03:10:25.0421085Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-21T03:10:25.0422045Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-21T03:10:25.0423123Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-21T03:10:25.0424161Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-21T03:10:25.0425139Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-21T03:10:25.0426143Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-21T03:10:25.0427148Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-21T03:10:25.0428156Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-21T03:10:25.0429150Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-21T03:10:25.0430147Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-21T03:10:25.0441905Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-21T03:10:25.0500426Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-21T03:10:25.0501579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-21T03:10:25.0502579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-21T03:10:25.0503475Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-21T03:10:25.0504315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-21T03:10:25.0505117Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-21T03:10:25.0505920Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2024-08-21T03:10:25.0506685Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-08-21T03:10:25.0507425Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-21T03:10:25.0508131Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-21T03:10:25.0509083Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-21T03:10:25.0551367Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-21T03:10:25.0552150Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-21T03:10:25.0552872Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-21T03:10:25.0553737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-21T03:10:25.0555713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-21T03:10:25.0556629Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-21T03:10:25.0557547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-21T03:10:25.0558509Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-21T03:10:25.0559460Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-21T03:10:25.0560577Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-21T03:10:25.0561655Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-21T03:10:25.0562874Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-21T03:10:25.0563831Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-21T03:10:25.0564940Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-21T03:10:25.0566058Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-21T03:10:25.0567147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-21T03:10:25.0568246Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-21T03:10:25.0569354Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-21T03:10:25.0570458Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-21T03:10:25.0571401Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-21T03:10:25.0572238Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-21T03:10:25.0573122Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-21T03:10:25.0574047Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-21T03:10:25.0575084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-21T03:10:25.0576232Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-21T03:10:25.0577341Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-21T03:10:25.0578379Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-21T03:10:25.0579454Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-21T03:10:25.0580521Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-21T03:10:25.0581599Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-21T03:10:25.0582670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-21T03:10:25.0583733Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-21T03:10:25.0585031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-21T03:10:25.0693387Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-21T03:10:25.0694480Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-21T03:10:25.0695582Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-21T03:10:25.0696792Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-21T03:10:25.0697951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-21T03:10:25.0699051Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-21T03:10:25.0700185Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-21T03:10:25.0701309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-21T03:10:25.0702461Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-21T03:10:25.0703808Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-21T03:10:25.0704948Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-21T03:10:25.0717161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-21T03:10:25.0767673Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-21T03:10:25.0768865Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-21T03:10:25.0769907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-21T03:10:25.0770841Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-21T03:10:25.0771722Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-21T03:10:25.0772586Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-21T03:10:25.0773462Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2024-08-21T03:10:25.0774290Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-08-21T03:10:25.0775086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-21T03:10:25.0775846Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-21T03:10:25.0776631Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-21T03:10:25.0867086Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-21T03:10:25.0905567Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-21T03:10:25.0906146Z creating: build/lib/ 2024-08-21T03:10:25.0985977Z inflating: build/lib/libprotobuf-lite.a 2024-08-21T03:10:25.1399642Z inflating: build/lib/libprotobuf.a 2024-08-21T03:10:25.1853113Z inflating: build/lib/libprotoc.a 2024-08-21T03:10:25.1861765Z inflating: build/lib/libpthreadpool.a 2024-08-21T03:10:25.1869029Z inflating: build/lib/libcpuinfo.a 2024-08-21T03:10:25.1876058Z inflating: build/lib/libcpuinfo_internals.a 2024-08-21T03:10:25.1877024Z inflating: build/lib/libclog.a 2024-08-21T03:10:25.1894160Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-21T03:10:25.1895915Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-21T03:10:25.1912208Z inflating: build/lib/libnnpack.a 2024-08-21T03:10:25.1970943Z inflating: build/lib/libgtest.a 2024-08-21T03:10:25.1986933Z inflating: build/lib/libgmock.a 2024-08-21T03:10:25.1987862Z inflating: build/lib/libgmock_main.a 2024-08-21T03:10:25.1988748Z inflating: build/lib/libgtest_main.a 2024-08-21T03:10:25.2055489Z inflating: build/lib/libbenchmark.a 2024-08-21T03:10:25.2056444Z inflating: build/lib/libbenchmark_main.a 2024-08-21T03:10:25.2220489Z inflating: build/lib/libXNNPACK.a 2024-08-21T03:10:25.2227256Z inflating: build/lib/libittnotify.a 2024-08-21T03:10:25.2284461Z inflating: build/lib/libasmjit.a 2024-08-21T03:10:25.3431928Z inflating: build/lib/libfbgemm.a 2024-08-21T03:10:25.3456752Z inflating: build/lib/libtensorpipe_uv.a 2024-08-21T03:10:25.3978282Z inflating: build/lib/libtensorpipe.a 2024-08-21T03:10:25.4081080Z inflating: build/lib/libgloo.a 2024-08-21T03:10:25.4438094Z inflating: build/lib/libgloo_hip.a 2024-08-21T03:10:25.4476852Z inflating: build/lib/libonnx_proto.a 2024-08-21T03:10:25.5105920Z inflating: build/lib/libonnx.a 2024-08-21T03:10:26.4386885Z inflating: build/lib/libdnnl.a 2024-08-21T03:10:26.4405596Z inflating: build/lib/libfmt.a 2024-08-21T03:10:26.4697967Z inflating: build/lib/libkineto.a 2024-08-21T03:10:26.4783108Z inflating: build/lib/libc10.so 2024-08-21T03:10:26.4784070Z inflating: build/lib/libtorch_global_deps.so 2024-08-21T03:10:26.4825673Z inflating: build/lib/libc10_hip.so 2024-08-21T03:10:26.4826613Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-21T03:10:28.9447943Z inflating: build/lib/libtorch_cpu.so 2024-08-21T03:10:28.9452508Z inflating: build/lib/libunbox_lib.a 2024-08-21T03:10:28.9456172Z inflating: build/lib/libshm.so 2024-08-21T03:10:30.9877425Z inflating: build/lib/libtorch_hip.so 2024-08-21T03:10:30.9878349Z inflating: build/lib/libtorch.so 2024-08-21T03:10:30.9897452Z inflating: build/lib/libjitbackend_test.so 2024-08-21T03:10:30.9919967Z inflating: build/lib/libbackend_with_compiler.so 2024-08-21T03:10:30.9984547Z inflating: build/lib/libtorchbind_test.so 2024-08-21T03:10:31.0007874Z inflating: build/lib/libaoti_custom_ops.so 2024-08-21T03:10:31.1782516Z inflating: build/lib/libtorch_python.so 2024-08-21T03:10:31.1813279Z inflating: build/lib/libnnapi_backend.so 2024-08-21T03:10:31.1814194Z creating: build/bin/ 2024-08-21T03:10:31.1814874Z creating: build/bin/CMakeFiles/ 2024-08-21T03:10:31.1815694Z inflating: build/bin/cmake_install.cmake 2024-08-21T03:10:31.1816583Z inflating: build/bin/CTestTestfile.cmake 2024-08-21T03:10:31.2213119Z inflating: build/bin/protoc-3.13.0.0 2024-08-21T03:10:31.2610158Z inflating: build/bin/protoc 2024-08-21T03:10:31.2655337Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-21T03:10:31.2701081Z inflating: build/bin/c10_DeviceGuard_test 2024-08-21T03:10:31.2746493Z inflating: build/bin/c10_Device_test 2024-08-21T03:10:31.2799207Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-21T03:10:31.2846747Z inflating: build/bin/c10_Scalar_test 2024-08-21T03:10:31.2890380Z inflating: build/bin/c10_StreamGuard_test 2024-08-21T03:10:31.2935776Z inflating: build/bin/c10_SymInt_test 2024-08-21T03:10:31.2984321Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-21T03:10:31.3027868Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-21T03:10:31.3077569Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-21T03:10:31.3127862Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-21T03:10:31.3174988Z inflating: build/bin/c10_Bitset_test 2024-08-21T03:10:31.3238505Z inflating: build/bin/c10_cow_test 2024-08-21T03:10:31.3282983Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-21T03:10:31.3333093Z inflating: build/bin/c10_LeftRight_test 2024-08-21T03:10:31.3378294Z inflating: build/bin/c10_Half_test 2024-08-21T03:10:31.3428090Z inflating: build/bin/c10_Metaprogramming_test 2024-08-21T03:10:31.3471994Z inflating: build/bin/c10_Synchronized_test 2024-08-21T03:10:31.3521606Z inflating: build/bin/c10_ThreadLocal_test 2024-08-21T03:10:31.3567408Z inflating: build/bin/c10_TypeIndex_test 2024-08-21T03:10:31.3611013Z inflating: build/bin/c10_TypeTraits_test 2024-08-21T03:10:31.3656469Z inflating: build/bin/c10_TypeList_test 2024-08-21T03:10:31.3702508Z inflating: build/bin/c10_accumulate_test 2024-08-21T03:10:31.3751918Z inflating: build/bin/c10_bfloat16_test 2024-08-21T03:10:31.3797297Z inflating: build/bin/c10_bit_cast_test 2024-08-21T03:10:31.3847640Z inflating: build/bin/c10_complex_math_test 2024-08-21T03:10:31.3897275Z inflating: build/bin/c10_complex_test 2024-08-21T03:10:31.3944222Z inflating: build/bin/c10_exception_test 2024-08-21T03:10:31.3989046Z inflating: build/bin/c10_flags_test 2024-08-21T03:10:31.4033671Z inflating: build/bin/c10_generic_math_test 2024-08-21T03:10:31.4078868Z inflating: build/bin/c10_irange_test 2024-08-21T03:10:31.4126634Z inflating: build/bin/c10_lazy_test 2024-08-21T03:10:31.4274826Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-21T03:10:31.4325789Z inflating: build/bin/c10_logging_test 2024-08-21T03:10:31.4393110Z inflating: build/bin/c10_optional_test 2024-08-21T03:10:31.4448927Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-21T03:10:31.4496805Z inflating: build/bin/c10_registry_test 2024-08-21T03:10:31.4632754Z inflating: build/bin/c10_small_vector_test 2024-08-21T03:10:31.4678675Z inflating: build/bin/c10_ssize_test 2024-08-21T03:10:31.4725103Z inflating: build/bin/c10_string_util_test 2024-08-21T03:10:31.4777434Z inflating: build/bin/c10_string_view_test 2024-08-21T03:10:31.4822256Z inflating: build/bin/c10_tempfile_test 2024-08-21T03:10:31.4865356Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-21T03:10:31.4915311Z inflating: build/bin/c10_typeid_test 2024-08-21T03:10:31.4959378Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-08-21T03:10:31.5003051Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-08-21T03:10:31.5046831Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-08-21T03:10:31.5090206Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-08-21T03:10:31.5133925Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-21T03:10:31.5177531Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-21T03:10:31.5220101Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-08-21T03:10:31.5263721Z inflating: build/bin/c10_hip_HIPTest 2024-08-21T03:10:31.5562668Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-21T03:10:31.5872643Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-21T03:10:31.6195946Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-21T03:10:31.6243909Z inflating: build/bin/test_edge_op_registration 2024-08-21T03:10:31.6290707Z inflating: build/bin/BackoffTest 2024-08-21T03:10:31.6337984Z inflating: build/bin/FileStoreTest 2024-08-21T03:10:31.6387608Z inflating: build/bin/TCPStoreTest 2024-08-21T03:10:31.6435153Z inflating: build/bin/HashStoreTest 2024-08-21T03:10:31.6493755Z inflating: build/bin/ProcessGroupGlooTest 2024-08-21T03:10:31.6495355Z inflating: build/bin/example_allreduce 2024-08-21T03:10:31.6499953Z inflating: build/bin/torch_shm_manager 2024-08-21T03:10:31.6553380Z inflating: build/bin/static_runtime_bench 2024-08-21T03:10:31.6779330Z inflating: build/bin/static_runtime_test 2024-08-21T03:10:31.6845848Z inflating: build/bin/Dict_test 2024-08-21T03:10:31.6892200Z inflating: build/bin/Dimname_test 2024-08-21T03:10:31.6950000Z inflating: build/bin/MaybeOwned_test 2024-08-21T03:10:31.7001312Z inflating: build/bin/NamedTensor_test 2024-08-21T03:10:31.7045482Z inflating: build/bin/hip_packedtensoraccessor_test 2024-08-21T03:10:31.7097761Z inflating: build/bin/apply_utils_test 2024-08-21T03:10:31.7150735Z inflating: build/bin/atest 2024-08-21T03:10:31.7207107Z inflating: build/bin/basic 2024-08-21T03:10:31.7256245Z inflating: build/bin/broadcast_test 2024-08-21T03:10:31.7301345Z inflating: build/bin/cpu_allocator_test 2024-08-21T03:10:31.7353773Z inflating: build/bin/cpu_generator_test 2024-08-21T03:10:31.7401402Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-21T03:10:31.7484941Z inflating: build/bin/cpu_rng_test 2024-08-21T03:10:31.7529612Z inflating: build/bin/dispatch_key_set_test 2024-08-21T03:10:31.7574558Z inflating: build/bin/dlconvertor_test 2024-08-21T03:10:31.7626374Z inflating: build/bin/extension_backend_test 2024-08-21T03:10:31.7675571Z inflating: build/bin/half_test 2024-08-21T03:10:31.7761678Z inflating: build/bin/ivalue_test 2024-08-21T03:10:31.7806101Z inflating: build/bin/lazy_tensor_test 2024-08-21T03:10:31.7854609Z inflating: build/bin/math_kernel_test 2024-08-21T03:10:31.7902839Z inflating: build/bin/memory_format_test 2024-08-21T03:10:31.7950203Z inflating: build/bin/memory_overlapping_test 2024-08-21T03:10:31.7995209Z inflating: build/bin/operator_name_test 2024-08-21T03:10:31.8045144Z inflating: build/bin/native_test 2024-08-21T03:10:31.8092485Z inflating: build/bin/mobile_memory_cleanup 2024-08-21T03:10:31.8138345Z inflating: build/bin/operators_test 2024-08-21T03:10:31.8184694Z inflating: build/bin/packedtensoraccessor_test 2024-08-21T03:10:31.8244790Z inflating: build/bin/pow_test 2024-08-21T03:10:31.8295708Z inflating: build/bin/quantized_test 2024-08-21T03:10:31.8340865Z inflating: build/bin/reduce_ops_test 2024-08-21T03:10:31.8386472Z inflating: build/bin/reportMemoryUsage_test 2024-08-21T03:10:31.8436863Z inflating: build/bin/scalar_tensor_test 2024-08-21T03:10:31.8488978Z inflating: build/bin/scalar_test 2024-08-21T03:10:31.8535221Z inflating: build/bin/StorageUtils_test 2024-08-21T03:10:31.8582143Z inflating: build/bin/stride_properties_test 2024-08-21T03:10:31.8652930Z inflating: build/bin/tensor_iterator_test 2024-08-21T03:10:31.8701191Z inflating: build/bin/test_parallel 2024-08-21T03:10:31.8702797Z inflating: build/bin/thread_init_test 2024-08-21T03:10:31.8753617Z inflating: build/bin/type_ptr_test 2024-08-21T03:10:31.8807246Z inflating: build/bin/type_test 2024-08-21T03:10:31.8853925Z inflating: build/bin/undefined_tensor_test 2024-08-21T03:10:31.8854941Z inflating: build/bin/verify_api_visibility 2024-08-21T03:10:31.8901340Z inflating: build/bin/weakref_test 2024-08-21T03:10:31.8947728Z inflating: build/bin/wrapdim_test 2024-08-21T03:10:31.9009841Z inflating: build/bin/legacy_vmap_test 2024-08-21T03:10:31.9055955Z inflating: build/bin/xla_tensor_test 2024-08-21T03:10:31.9151508Z inflating: build/bin/List_test 2024-08-21T03:10:31.9205274Z inflating: build/bin/IListRef_test 2024-08-21T03:10:31.9314328Z inflating: build/bin/kernel_function_legacy_test 2024-08-21T03:10:31.9401108Z inflating: build/bin/kernel_function_test 2024-08-21T03:10:31.9494666Z inflating: build/bin/kernel_lambda_test 2024-08-21T03:10:31.9554222Z inflating: build/bin/KernelFunction_test 2024-08-21T03:10:31.9669427Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-21T03:10:31.9723984Z inflating: build/bin/kernel_stackbased_test 2024-08-21T03:10:31.9769608Z inflating: build/bin/CppSignature_test 2024-08-21T03:10:31.9856092Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-21T03:10:31.9899536Z inflating: build/bin/op_allowlist_test 2024-08-21T03:10:31.9956564Z inflating: build/bin/inline_container_test 2024-08-21T03:10:32.0006353Z inflating: build/bin/backend_fallback_test 2024-08-21T03:10:32.0280748Z inflating: build/bin/op_registration_test 2024-08-21T03:10:32.0325012Z inflating: build/bin/hip_complex_math_test 2024-08-21T03:10:32.0371710Z inflating: build/bin/hip_apply_test 2024-08-21T03:10:32.0415323Z inflating: build/bin/hip_complex_test 2024-08-21T03:10:32.0458967Z inflating: build/bin/hip_distributions_test 2024-08-21T03:10:32.0504279Z inflating: build/bin/hip_dlconvertor_test 2024-08-21T03:10:32.0547820Z inflating: build/bin/hip_generator_test 2024-08-21T03:10:32.0591310Z inflating: build/bin/hip_half_test 2024-08-21T03:10:32.0635211Z inflating: build/bin/hip_integer_divider_test 2024-08-21T03:10:32.0678686Z inflating: build/bin/hip_optional_test 2024-08-21T03:10:32.0723007Z inflating: build/bin/hip_vectorized_test 2024-08-21T03:10:32.1242460Z inflating: build/bin/test_jit 2024-08-21T03:10:32.1951671Z inflating: build/bin/test_tensorexpr 2024-08-21T03:10:32.1964687Z inflating: build/bin/tutorial_tensorexpr 2024-08-21T03:10:32.2013625Z inflating: build/bin/test_dist_autograd 2024-08-21T03:10:32.2075134Z inflating: build/bin/test_cpp_rpc 2024-08-21T03:10:32.2076610Z inflating: build/bin/parallel_benchmark 2024-08-21T03:10:32.3135704Z inflating: build/bin/test_api 2024-08-21T03:10:32.3143798Z inflating: build/bin/aot_model_compiler_test 2024-08-21T03:10:32.3203601Z inflating: build/bin/test_mobile_nnc 2024-08-21T03:10:32.3515159Z inflating: build/bin/test_lazy 2024-08-21T03:10:32.3516031Z creating: .additional_ci_files/ 2024-08-21T03:10:32.3570343Z inflating: .additional_ci_files/test-times.json 2024-08-21T03:10:32.3785300Z inflating: .additional_ci_files/test-class-times.json 2024-08-21T03:10:32.3842653Z ##[group]Run rm artifacts.zip 2024-08-21T03:10:32.3843373Z rm artifacts.zip 2024-08-21T03:10:32.3892381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:32.3893441Z env: 2024-08-21T03:10:32.3893995Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:32.3894740Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:32.3896136Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:32.3897415Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:32.3898094Z AWS_REGION: us-east-1 2024-08-21T03:10:32.3898852Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:32.3899949Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:32.3915821Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:32.3916512Z ##[endgroup] 2024-08-21T03:10:32.5133096Z ##[group]Run df -H 2024-08-21T03:10:32.5133671Z df -H 2024-08-21T03:10:32.5180663Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:32.5181569Z env: 2024-08-21T03:10:32.5182088Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:32.5182828Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:32.5184168Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:32.5185438Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:32.5186115Z AWS_REGION: us-east-1 2024-08-21T03:10:32.5186897Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:32.5187805Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:32.5200467Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:32.5201075Z ##[endgroup] 2024-08-21T03:10:32.5309878Z Filesystem Size Used Avail Use% Mounted on 2024-08-21T03:10:32.5311145Z tmpfs 14G 18M 14G 1% /run 2024-08-21T03:10:32.5312758Z /dev/mapper/ubuntu--vg-ubuntu--lv 1.9T 382G 1.5T 22% / 2024-08-21T03:10:32.5313821Z tmpfs 68G 8.2k 68G 1% /dev/shm 2024-08-21T03:10:32.5314800Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-08-21T03:10:32.5322246Z /dev/sda2 2.1G 306M 1.7G 16% /boot 2024-08-21T03:10:32.5323662Z /dev/sda1 1.2G 6.4M 1.2G 1% /boot/efi 2024-08-21T03:10:32.5324826Z tmpfs 14G 17k 14G 1% /run/user/1001 2024-08-21T03:10:32.5325800Z tmpfs 14G 4.1k 14G 1% /run/user/1000 2024-08-21T03:10:32.5388006Z Prepare all required actions 2024-08-21T03:10:32.5388850Z Getting action download info 2024-08-21T03:10:32.7180097Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-21T03:10:32.7180932Z with: 2024-08-21T03:10:32.7181407Z env: 2024-08-21T03:10:32.7181888Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:32.7182645Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:32.7183996Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:32.7185260Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:32.7185930Z AWS_REGION: us-east-1 2024-08-21T03:10:32.7186669Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:32.7187564Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:32.7199985Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:32.7200769Z ##[endgroup] 2024-08-21T03:10:32.7266075Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-21T03:10:32.7266890Z with: 2024-08-21T03:10:32.7267373Z name: td_results 2024-08-21T03:10:32.7268003Z s3-bucket: gha-artifacts 2024-08-21T03:10:32.7268621Z region: us-east-1 2024-08-21T03:10:32.7269132Z env: 2024-08-21T03:10:32.7269608Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:32.7270324Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:32.7272190Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:32.7273464Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:32.7274144Z AWS_REGION: us-east-1 2024-08-21T03:10:32.7274860Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:32.7275753Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:32.7288392Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:32.7289000Z ##[endgroup] 2024-08-21T03:10:33.1395226Z (node:1761396) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-21T03:10:33.1396537Z 2024-08-21T03:10:33.1397025Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-21T03:10:33.1398393Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-21T03:10:33.1400005Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-21T03:10:33.6847345Z Found 1 objects with prefix pytorch/pytorch/10479310961/td_results/ 2024-08-21T03:10:33.6849276Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-21T03:10:34.0694793Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-08-21T03:10:34.0706917Z Artifact download has finished successfully 2024-08-21T03:10:34.1267169Z ##[group]Run mkdir -p .additional_ci_files 2024-08-21T03:10:34.1268056Z mkdir -p .additional_ci_files 2024-08-21T03:10:34.1269069Z mv td_results.json .additional_ci_files/td_results.json 2024-08-21T03:10:34.1316199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:34.1317090Z env: 2024-08-21T03:10:34.1317607Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:34.1318391Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:34.1319732Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:34.1321117Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:34.1321826Z AWS_REGION: us-east-1 2024-08-21T03:10:34.1322595Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:34.1323505Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:34.1336823Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:34.1337405Z ##[endgroup] 2024-08-21T03:10:34.1521562Z ##[group]Run .github/scripts/parse_ref.py 2024-08-21T03:10:34.1522469Z .github/scripts/parse_ref.py 2024-08-21T03:10:34.1565694Z shell: /usr/bin/bash -e {0} 2024-08-21T03:10:34.1566342Z env: 2024-08-21T03:10:34.1566835Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:34.1567589Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:34.1568939Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:34.1570308Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:34.1571103Z AWS_REGION: us-east-1 2024-08-21T03:10:34.1571966Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:34.1573030Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:34.1586040Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:34.1586661Z ##[endgroup] 2024-08-21T03:10:34.1865732Z Prepare all required actions 2024-08-21T03:10:34.1897322Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-21T03:10:34.1897670Z with: 2024-08-21T03:10:34.1898092Z github-token: *** 2024-08-21T03:10:34.1898333Z env: 2024-08-21T03:10:34.1898546Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:34.1898866Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:34.1899430Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:34.1899966Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:34.1900258Z AWS_REGION: us-east-1 2024-08-21T03:10:34.1900548Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:34.1900920Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:34.1906049Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:34.1906306Z ##[endgroup] 2024-08-21T03:10:34.1920684Z ##[group]Run set -eux 2024-08-21T03:10:34.1920938Z set -eux 2024-08-21T03:10:34.1921399Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-21T03:10:34.1941157Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:34.1941539Z env: 2024-08-21T03:10:34.1941754Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:34.1942077Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:34.1942638Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:34.1943159Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:34.1943444Z AWS_REGION: us-east-1 2024-08-21T03:10:34.1943744Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:34.1944115Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:34.1949217Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:34.1949577Z GITHUB_TOKEN: *** 2024-08-21T03:10:34.1949815Z ##[endgroup] 2024-08-21T03:10:34.1983363Z + python3 .github/scripts/get_workflow_job_id.py 10479310961 pytorch-rocm-hw-50 2024-08-21T03:10:36.1234034Z setting job-id=29025885013 2024-08-21T03:10:36.1235529Z setting job-name=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:10:36.1658253Z Prepare all required actions 2024-08-21T03:10:36.1658771Z Getting action download info 2024-08-21T03:10:36.3597020Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-21T03:10:36.8325581Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-21T03:10:36.8326434Z with: 2024-08-21T03:10:36.8327281Z github-token: *** 2024-08-21T03:10:36.8329525Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-08-21T03:10:36.8332129Z job-name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:10:36.8333007Z env: 2024-08-21T03:10:36.8333418Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:36.8334052Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:36.8335141Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:36.8336136Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:36.8336683Z AWS_REGION: us-east-1 2024-08-21T03:10:36.8337228Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:36.8337921Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:36.8347296Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:36.8347789Z ##[endgroup] 2024-08-21T03:10:36.8412564Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-21T03:10:36.8413334Z with: 2024-08-21T03:10:36.8413737Z shell: bash 2024-08-21T03:10:36.8414166Z timeout_minutes: 10 2024-08-21T03:10:36.8414634Z max_attempts: 5 2024-08-21T03:10:36.8415079Z retry_wait_seconds: 30 2024-08-21T03:10:36.8416624Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-21T03:10:36.8418226Z polling_interval_seconds: 1 2024-08-21T03:10:36.8418773Z warning_on_retry: true 2024-08-21T03:10:36.8419274Z continue_on_error: false 2024-08-21T03:10:36.8419757Z env: 2024-08-21T03:10:36.8420149Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:36.8420730Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:36.8421768Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:36.8422747Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:36.8423295Z AWS_REGION: us-east-1 2024-08-21T03:10:36.8423864Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:36.8424563Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:36.8433964Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:36.8434639Z GITHUB_TOKEN: *** 2024-08-21T03:10:36.8435097Z ##[endgroup] 2024-08-21T03:10:36.8842437Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-21T03:10:37.1321396Z Defaulting to user installation because normal site-packages is not writeable 2024-08-21T03:10:37.2051826Z Requirement already satisfied: requests==2.27.1 in /home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-08-21T03:10:37.2055085Z Requirement already satisfied: pyyaml==6.0.1 in /home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-08-21T03:10:37.2147238Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-08-21T03:10:37.2156906Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-08-21T03:10:37.2159835Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-08-21T03:10:37.2166148Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-08-21T03:10:37.8842377Z Command completed after 1 attempt(s). 2024-08-21T03:10:37.8968271Z ##[group]Run set -x 2024-08-21T03:10:37.8969358Z set -x 2024-08-21T03:10:37.8969918Z  2024-08-21T03:10:37.8970940Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-21T03:10:37.8972356Z # in runner workspace 2024-08-21T03:10:37.8973478Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-21T03:10:37.9021968Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:37.9023012Z env: 2024-08-21T03:10:37.9023610Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:37.9024481Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:37.9025845Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:37.9027094Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:37.9027770Z AWS_REGION: us-east-1 2024-08-21T03:10:37.9028540Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:37.9029481Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:37.9042113Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:37.9042737Z ##[endgroup] 2024-08-21T03:10:37.9127941Z + python3 /home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-21T03:10:37.9378255Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-21T03:10:37.9379209Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-21T03:10:37.9380041Z echo "Job name: ${JOB_NAME}" 2024-08-21T03:10:37.9380745Z  2024-08-21T03:10:37.9381722Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-21T03:10:37.9383152Z # in runner workspace 2024-08-21T03:10:37.9384388Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-21T03:10:37.9385629Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-21T03:10:37.9386461Z  --job-name "${JOB_NAME}" \ 2024-08-21T03:10:37.9388905Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-08-21T03:10:37.9391370Z  --selected-test-configs "" \ 2024-08-21T03:10:37.9392162Z  --pr-number "${PR_NUMBER}" \ 2024-08-21T03:10:37.9392889Z  --tag "${TAG}" \ 2024-08-21T03:10:37.9393589Z  --event-name "${EVENT_NAME}" \ 2024-08-21T03:10:37.9394369Z  --schedule "${SCHEDULE}" \ 2024-08-21T03:10:37.9395125Z  --branch "${HEAD_BRANCH}" 2024-08-21T03:10:37.9438633Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:37.9439534Z env: 2024-08-21T03:10:37.9440028Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:37.9440876Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:37.9442224Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:37.9443903Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:37.9444586Z AWS_REGION: us-east-1 2024-08-21T03:10:37.9445296Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:37.9446199Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:37.9459674Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:37.9460546Z GITHUB_TOKEN: *** 2024-08-21T03:10:37.9461522Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:10:37.9462783Z PR_NUMBER: 2024-08-21T03:10:37.9463427Z TAG: ciflow/trunk/133712 2024-08-21T03:10:37.9464201Z EVENT_NAME: push 2024-08-21T03:10:37.9464843Z SCHEDULE: 2024-08-21T03:10:37.9465371Z HEAD_BRANCH: 2024-08-21T03:10:37.9465904Z ##[endgroup] 2024-08-21T03:10:37.9539351Z Workflow: trunk 2024-08-21T03:10:38.2807092Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-21T03:10:38.2808995Z Job name: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:10:38.8286257Z ##[group]Run echo "Filtered matrix:" 2024-08-21T03:10:38.8287518Z echo "Filtered matrix:" 2024-08-21T03:10:38.8289861Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-08-21T03:10:38.8292403Z  2024-08-21T03:10:38.8292996Z echo 2024-08-21T03:10:38.8293793Z echo "Is the current job unstable? False" 2024-08-21T03:10:38.8294751Z  2024-08-21T03:10:38.8295324Z echo 2024-08-21T03:10:38.8296107Z echo "Is keep-going label set? False" 2024-08-21T03:10:38.8296671Z  2024-08-21T03:10:38.8296957Z echo 2024-08-21T03:10:38.8297257Z echo "Renabled issues? " 2024-08-21T03:10:38.8338517Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:38.8339011Z env: 2024-08-21T03:10:38.8339284Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:38.8339692Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:38.8340393Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:38.8341058Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:38.8341422Z AWS_REGION: us-east-1 2024-08-21T03:10:38.8341884Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:38.8342419Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:38.8355119Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:38.8355734Z ##[endgroup] 2024-08-21T03:10:38.8429792Z Filtered matrix: 2024-08-21T03:10:38.8431990Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-08-21T03:10:38.8434044Z 2024-08-21T03:10:38.8434352Z Is the current job unstable? False 2024-08-21T03:10:38.8434911Z 2024-08-21T03:10:38.8435363Z Is keep-going label set? False 2024-08-21T03:10:38.8435826Z 2024-08-21T03:10:38.8436080Z Renabled issues? 2024-08-21T03:10:38.8525254Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-21T03:10:38.8526583Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-21T03:10:38.8570992Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:10:38.8572065Z env: 2024-08-21T03:10:38.8572661Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:38.8573538Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:38.8575147Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:38.8576642Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:38.8577374Z AWS_REGION: us-east-1 2024-08-21T03:10:38.8578082Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:38.8578962Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:38.8591941Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:38.8592953Z JOB_TIMEOUT: 300 2024-08-21T03:10:38.8593495Z ##[endgroup] 2024-08-21T03:10:38.8783913Z ##[group]Run set -x 2024-08-21T03:10:38.8784678Z set -x 2024-08-21T03:10:38.8785206Z  2024-08-21T03:10:38.8785837Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-21T03:10:38.8786798Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-21T03:10:38.8787793Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-21T03:10:38.8788703Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-08-21T03:10:38.8789431Z else 2024-08-21T03:10:38.8790048Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-21T03:10:38.8790781Z fi 2024-08-21T03:10:38.8791260Z  2024-08-21T03:10:38.8792094Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-21T03:10:38.8793423Z # TODO: Stop building test binaries as part of the build phase 2024-08-21T03:10:38.8794650Z # Used for GPU_FLAG since that doesn't play nice 2024-08-21T03:10:38.8795747Z # shellcheck disable=SC2086,SC2090 2024-08-21T03:10:38.8796580Z container_name=$(docker run \ 2024-08-21T03:10:38.8797314Z  ${GPU_FLAG:-} \ 2024-08-21T03:10:38.8797967Z  -e BUILD_ENVIRONMENT \ 2024-08-21T03:10:38.8798661Z  -e PR_NUMBER \ 2024-08-21T03:10:38.8799287Z  -e GITHUB_ACTIONS \ 2024-08-21T03:10:38.8799970Z  -e GITHUB_REPOSITORY \ 2024-08-21T03:10:38.8800813Z  -e GITHUB_WORKFLOW \ 2024-08-21T03:10:38.8801526Z  -e GITHUB_JOB \ 2024-08-21T03:10:38.8802155Z  -e GITHUB_RUN_ID \ 2024-08-21T03:10:38.8802815Z  -e GITHUB_RUN_NUMBER \ 2024-08-21T03:10:38.8803506Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-21T03:10:38.8804213Z  -e JOB_ID \ 2024-08-21T03:10:38.8804799Z  -e JOB_NAME \ 2024-08-21T03:10:38.8805403Z  -e BRANCH \ 2024-08-21T03:10:38.8805987Z  -e SHA1 \ 2024-08-21T03:10:38.8806600Z  -e AWS_DEFAULT_REGION \ 2024-08-21T03:10:38.8807318Z  -e IN_WHEEL_TEST \ 2024-08-21T03:10:38.8807986Z  -e SHARD_NUMBER \ 2024-08-21T03:10:38.8808631Z  -e TEST_CONFIG \ 2024-08-21T03:10:38.8809284Z  -e NUM_TEST_SHARDS \ 2024-08-21T03:10:38.8810100Z  -e REENABLED_ISSUES \ 2024-08-21T03:10:38.8810946Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-21T03:10:38.8811825Z  -e VERBOSE_TEST_LOGS \ 2024-08-21T03:10:38.8812650Z  -e TEST_SHOWLOCALS \ 2024-08-21T03:10:38.8813453Z  -e NO_TEST_TIMEOUT \ 2024-08-21T03:10:38.8814229Z  -e NO_TD \ 2024-08-21T03:10:38.8815054Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-21T03:10:38.8815922Z  -e SCCACHE_BUCKET \ 2024-08-21T03:10:38.8816664Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-21T03:10:38.8817531Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-21T03:10:38.8818448Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-21T03:10:38.8819287Z  -e TESTS_TO_INCLUDE \ 2024-08-21T03:10:38.8820135Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-21T03:10:38.8821176Z  --ulimit stack=10485760:83886080 \ 2024-08-21T03:10:38.8822072Z  --ulimit core=0 \ 2024-08-21T03:10:38.8822950Z  --security-opt seccomp=unconfined \ 2024-08-21T03:10:38.8823843Z  --cap-add=SYS_PTRACE \ 2024-08-21T03:10:38.8824531Z  --shm-size="8g" \ 2024-08-21T03:10:38.8825145Z  --tty \ 2024-08-21T03:10:38.8825690Z  --detach \ 2024-08-21T03:10:38.8826318Z  --name="${container_name}" \ 2024-08-21T03:10:38.8827044Z  --user jenkins \ 2024-08-21T03:10:38.8827889Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-21T03:10:38.8828876Z  -w /var/lib/jenkins/workspace \ 2024-08-21T03:10:38.8829635Z  "${DOCKER_IMAGE}" 2024-08-21T03:10:38.8830243Z ) 2024-08-21T03:10:38.8831304Z # save container name for later step 2024-08-21T03:10:38.8832682Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-08-21T03:10:38.8834518Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-08-21T03:10:38.8836885Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-08-21T03:10:38.8883814Z shell: /usr/bin/bash -e {0} 2024-08-21T03:10:38.8884450Z env: 2024-08-21T03:10:38.8884933Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:10:38.8885664Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:10:38.8887026Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:10:38.8888283Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:10:38.8888942Z AWS_REGION: us-east-1 2024-08-21T03:10:38.8889727Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:10:38.8890789Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:10:38.8904526Z AWS_SESSION_TOKEN: *** 2024-08-21T03:10:38.8905255Z BUILD_ENVIRONMENT: linux-focal-rocm6.1-py3.8 2024-08-21T03:10:38.8906047Z PR_NUMBER: 2024-08-21T03:10:38.8906634Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-21T03:10:38.8907348Z GITHUB_WORKFLOW: trunk 2024-08-21T03:10:38.8907925Z GITHUB_JOB: test 2024-08-21T03:10:38.8908482Z GITHUB_RUN_ID: 10479310961 2024-08-21T03:10:38.8909119Z GITHUB_RUN_NUMBER: 92245 2024-08-21T03:10:38.8909732Z GITHUB_RUN_ATTEMPT: 1 2024-08-21T03:10:38.8910308Z JOB_ID: 29025885013 2024-08-21T03:10:38.8911291Z JOB_NAME: linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:10:38.8912369Z BRANCH: 2024-08-21T03:10:38.8912965Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:10:38.8913776Z CONTINUE_THROUGH_ERROR: False 2024-08-21T03:10:38.8914441Z VERBOSE_TEST_LOGS: False 2024-08-21T03:10:38.8915049Z TEST_SHOWLOCALS: False 2024-08-21T03:10:38.8915653Z NO_TEST_TIMEOUT: False 2024-08-21T03:10:38.8916226Z NO_TD: False 2024-08-21T03:10:38.8916751Z TEST_CONFIG: distributed 2024-08-21T03:10:38.8917330Z SHARD_NUMBER: 1 2024-08-21T03:10:38.8917855Z NUM_TEST_SHARDS: 1 2024-08-21T03:10:38.8918413Z REENABLED_ISSUES: 2024-08-21T03:10:38.8919099Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-21T03:10:38.8921065Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:10:38.8923068Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-21T03:10:38.8924134Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-21T03:10:38.8924882Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-21T03:10:38.8926893Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-21T03:10:38.8928878Z ##[endgroup] 2024-08-21T03:10:38.9007900Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-08-21T03:10:38.9009097Z + [[ linux-focal-rocm6.1-py3.8 == *onnx* ]] 2024-08-21T03:10:38.9010045Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-21T03:10:38.9025849Z +++ nproc --ignore=2 2024-08-21T03:10:38.9059533Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_10479310961 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T03:10:41.9925902Z + container_name=ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:10:41.9926794Z + echo CONTAINER_NAME=ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:10:41.9928380Z + docker exec -t ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-08-21T03:10:52.5441755Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp38-cp38-linux_x86_64.whl 2024-08-21T03:10:52.7682514Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-21T03:10:52.7685670Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-21T03:10:52.7688517Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-21T03:10:52.7691150Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-21T03:10:52.7693768Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-21T03:10:52.7696617Z Requirement already satisfied: sympy==1.12.1 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from torch==2.5.0a0+git40ec5f6) (1.12.1) 2024-08-21T03:10:52.7712134Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from sympy==1.12.1->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-21T03:10:52.8034047Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.8/lib/python3.8/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-21T03:10:53.3279331Z Installing collected packages: torch 2024-08-21T03:11:01.3734495Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-21T03:11:01.4196747Z + export TERM=vt100 2024-08-21T03:11:01.4201181Z + TERM=vt100 2024-08-21T03:11:01.4206417Z ++ dirname .ci/pytorch/test.sh 2024-08-21T03:11:01.4227383Z + source .ci/pytorch/common.sh 2024-08-21T03:11:01.4236261Z +++ dirname .ci/pytorch/common.sh 2024-08-21T03:11:01.4252555Z ++ source .ci/pytorch/common_utils.sh 2024-08-21T03:11:01.4253215Z +++ declare -f -t trap_add 2024-08-21T03:11:01.4260086Z ++ set -ex 2024-08-21T03:11:01.4260899Z ++ [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-21T03:11:01.4261732Z ++ unset HIP_PLATFORM 2024-08-21T03:11:01.4262364Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-08-21T03:11:01.4263119Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-08-21T03:11:01.4263805Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-08-21T03:11:01.4269214Z ++ HSAKMT_DEBUG_LEVEL=4 2024-08-21T03:11:01.4269616Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-21T03:11:01.4270030Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-21T03:11:01.4270401Z ++ BUILD_TEST_LIBTORCH=0 2024-08-21T03:11:01.4270876Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-08-21T03:11:01.4271346Z + echo 'Environment variables:' 2024-08-21T03:11:01.4276321Z Environment variables: 2024-08-21T03:11:01.4277208Z + env 2024-08-21T03:11:01.4280541Z INSTALLED_DB=yes 2024-08-21T03:11:01.4282140Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:11:01.4282777Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-08-21T03:11:01.4283136Z CONTINUE_THROUGH_ERROR=False 2024-08-21T03:11:01.4283510Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-08-21T03:11:01.4283860Z HOSTNAME=ecbe6fc5f802 2024-08-21T03:11:01.4284964Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4285838Z GITHUB_ACTION=__self 2024-08-21T03:11:01.4286113Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-21T03:11:01.4286410Z GITHUB_RUN_NUMBER=92245 2024-08-21T03:11:01.4286673Z TEST_CONFIG=distributed 2024-08-21T03:11:01.4286951Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-21T03:11:01.4287292Z AWS_DEFAULT_REGION=us-east-1 2024-08-21T03:11:01.4287625Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-21T03:11:01.4287979Z GITHUB_REF_TYPE=tag 2024-08-21T03:11:01.4297105Z *** 2024-08-21T03:11:01.4297457Z GITHUB_REPOSITORY_ID=65600975 2024-08-21T03:11:01.4297756Z GITHUB_ACTIONS=true 2024-08-21T03:11:01.4298070Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4298453Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4299030Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/133712 2024-08-21T03:11:01.4299556Z VERBOSE_TEST_LOGS=False 2024-08-21T03:11:01.4299857Z GITHUB_REF=refs/tags/ciflow/trunk/133712 2024-08-21T03:11:01.4300174Z SHARD_NUMBER=1 2024-08-21T03:11:01.4300413Z GITHUB_REF_PROTECTED=false 2024-08-21T03:11:01.4300674Z HOME=/var/lib/jenkins 2024-08-21T03:11:01.4300963Z GITHUB_API_URL=https://api.github.com 2024-08-21T03:11:01.4301321Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-21T03:11:01.4301673Z LANG=C.UTF-8 2024-08-21T03:11:01.4301898Z PYTORCH_TEST_WITH_ROCM=1 2024-08-21T03:11:01.4302163Z NUM_TEST_SHARDS=1 2024-08-21T03:11:01.4302817Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4303656Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:11:01.4304119Z MAGMA_HOME=/opt/rocm/magma 2024-08-21T03:11:01.4304768Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4305374Z HSAKMT_DEBUG_LEVEL=4 2024-08-21T03:11:01.4305876Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-21T03:11:01.4306365Z GITHUB_EVENT_NAME=push 2024-08-21T03:11:01.4306624Z GITHUB_RUN_ID=10479310961 2024-08-21T03:11:01.4307328Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4308008Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-21T03:11:01.4308282Z PR_NUMBER= 2024-08-21T03:11:01.4308503Z GITHUB_RUN_ATTEMPT=1 2024-08-21T03:11:01.4308755Z ANACONDA_PYTHON_VERSION=3.8 2024-08-21T03:11:01.4309093Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-21T03:11:01.4309444Z TERM=vt100 2024-08-21T03:11:01.4309663Z INSTALLED_VISION=yes 2024-08-21T03:11:01.4309896Z BRANCH= 2024-08-21T03:11:01.4310643Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-21T03:11:01.4311831Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-08-21T03:11:01.4312400Z GITHUB_SERVER_URL=https://github.com 2024-08-21T03:11:01.4312721Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-08-21T03:11:01.4313005Z REENABLED_ISSUES= 2024-08-21T03:11:01.4313230Z SHLVL=1 2024-08-21T03:11:01.4313441Z MAX_JOBS=62 2024-08-21T03:11:01.4313663Z GITHUB_ACTOR_ID=54816060 2024-08-21T03:11:01.4314013Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4314414Z GITHUB_REF_NAME=ciflow/trunk/133712 2024-08-21T03:11:01.4314716Z ROCM_PATH=/opt/rocm 2024-08-21T03:11:01.4315160Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-21T03:11:01.4315590Z GITHUB_JOB=test 2024-08-21T03:11:01.4315836Z NO_TEST_TIMEOUT=False 2024-08-21T03:11:01.4316115Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-21T03:11:01.4316422Z LC_ALL=C.UTF-8 2024-08-21T03:11:01.4316678Z GITHUB_RETENTION_DAYS=90 2024-08-21T03:11:01.4317378Z GITHUB_ACTION_REPOSITORY= 2024-08-21T03:11:01.4318546Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-21T03:11:01.4319504Z GITHUB_BASE_REF= 2024-08-21T03:11:01.4319746Z CI=true 2024-08-21T03:11:01.4319959Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-21T03:11:01.4320937Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-21T03:11:01.4321242Z JOB_ID=29025885013 2024-08-21T03:11:01.4321475Z INSTALLED_PROTOBUF=yes 2024-08-21T03:11:01.4321724Z GITHUB_HEAD_REF= 2024-08-21T03:11:01.4321955Z GITHUB_ACTION_REF= 2024-08-21T03:11:01.4322288Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-21T03:11:01.4322639Z TEST_SHOWLOCALS=False 2024-08-21T03:11:01.4322886Z GITHUB_WORKFLOW=trunk 2024-08-21T03:11:01.4323141Z DEBIAN_FRONTEND=noninteractive 2024-08-21T03:11:01.4323835Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4324461Z NO_TD=False 2024-08-21T03:11:01.4324680Z OLDPWD=/var/lib/jenkins 2024-08-21T03:11:01.4324926Z _=/usr/bin/env 2024-08-21T03:11:01.4325309Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-21T03:11:01.4481620Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch 2024-08-21T03:11:01.4483206Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:11:01.4484684Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib 2024-08-21T03:11:01.4486229Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/test 2024-08-21T03:11:01.4487376Z + BUILD_DIR=build 2024-08-21T03:11:01.4488055Z + BUILD_RENAMED_DIR=build_renamed 2024-08-21T03:11:01.4488879Z + BUILD_BIN_DIR=build/bin 2024-08-21T03:11:01.4489620Z + SHARD_NUMBER=1 2024-08-21T03:11:01.4490322Z + NUM_TEST_SHARDS=1 2024-08-21T03:11:01.4490965Z + export VALGRIND=ON 2024-08-21T03:11:01.4491667Z + VALGRIND=ON 2024-08-21T03:11:01.4492098Z + [[ linux-focal-rocm6.1-py3.8 == *clang9* ]] 2024-08-21T03:11:01.4492465Z + [[ 0 == \1 ]] 2024-08-21T03:11:01.4492687Z + [[ False == \1 ]] 2024-08-21T03:11:01.4492997Z + [[ linux-focal-rocm6.1-py3.8 != *bazel* ]] 2024-08-21T03:11:01.4493346Z ++ realpath build/custom_test_artifacts 2024-08-21T03:11:01.4498415Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-08-21T03:11:01.4499577Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-08-21T03:11:01.4500417Z + echo 'Setting INCLUDE_CLAUSE' 2024-08-21T03:11:01.4500715Z Setting INCLUDE_CLAUSE 2024-08-21T03:11:01.4501641Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-08-21T03:11:01.4502537Z + echo 'Environment variables' 2024-08-21T03:11:01.4502824Z Environment variables 2024-08-21T03:11:01.4503064Z + env 2024-08-21T03:11:01.4508438Z INSTALLED_DB=yes 2024-08-21T03:11:01.4508949Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:11:01.4509437Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-08-21T03:11:01.4509777Z CONTINUE_THROUGH_ERROR=False 2024-08-21T03:11:01.4510134Z BUILD_ENVIRONMENT=linux-focal-rocm6.1-py3.8 2024-08-21T03:11:01.4510524Z HOSTNAME=ecbe6fc5f802 2024-08-21T03:11:01.4511174Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4511771Z GITHUB_ACTION=__self 2024-08-21T03:11:01.4512033Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-21T03:11:01.4512326Z GITHUB_RUN_NUMBER=92245 2024-08-21T03:11:01.4512583Z TEST_CONFIG=distributed 2024-08-21T03:11:01.4512848Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-21T03:11:01.4513181Z AWS_DEFAULT_REGION=us-east-1 2024-08-21T03:11:01.4513734Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-21T03:11:01.4514178Z GITHUB_REF_TYPE=tag 2024-08-21T03:11:01.4514467Z *** 2024-08-21T03:11:01.4514685Z GITHUB_REPOSITORY_ID=65600975 2024-08-21T03:11:01.4514982Z GITHUB_ACTIONS=true 2024-08-21T03:11:01.4515272Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4515650Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4516215Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/133712 2024-08-21T03:11:01.4516720Z VERBOSE_TEST_LOGS=False 2024-08-21T03:11:01.4517000Z GITHUB_REF=refs/tags/ciflow/trunk/133712 2024-08-21T03:11:01.4517313Z SHARD_NUMBER=1 2024-08-21T03:11:01.4517551Z GITHUB_REF_PROTECTED=false 2024-08-21T03:11:01.4517824Z HOME=/var/lib/jenkins 2024-08-21T03:11:01.4518103Z GITHUB_API_URL=https://api.github.com 2024-08-21T03:11:01.4518439Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-21T03:11:01.4518748Z LANG=C.UTF-8 2024-08-21T03:11:01.4518975Z PYTORCH_TEST_WITH_ROCM=1 2024-08-21T03:11:01.4519253Z NUM_TEST_SHARDS=1 2024-08-21T03:11:01.4519900Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4520825Z JOB_NAME=linux-focal-rocm6.1-py3.8 / test (distributed, 1, 1, linux.rocm.gpu) 2024-08-21T03:11:01.4521283Z MAGMA_HOME=/opt/rocm/magma 2024-08-21T03:11:01.4521922Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4522504Z HSAKMT_DEBUG_LEVEL=4 2024-08-21T03:11:01.4522988Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-21T03:11:01.4523483Z GITHUB_EVENT_NAME=push 2024-08-21T03:11:01.4523734Z GITHUB_RUN_ID=10479310961 2024-08-21T03:11:01.4524423Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4525095Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-21T03:11:01.4525367Z PR_NUMBER= 2024-08-21T03:11:01.4525592Z GITHUB_RUN_ATTEMPT=1 2024-08-21T03:11:01.4525825Z VALGRIND=ON 2024-08-21T03:11:01.4526052Z ANACONDA_PYTHON_VERSION=3.8 2024-08-21T03:11:01.4526380Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-21T03:11:01.4526717Z TERM=vt100 2024-08-21T03:11:01.4526936Z INSTALLED_VISION=yes 2024-08-21T03:11:01.4527169Z BRANCH= 2024-08-21T03:11:01.4527912Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-08-21T03:11:01.4529026Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-08-21T03:11:01.4529574Z GITHUB_SERVER_URL=https://github.com 2024-08-21T03:11:01.4529896Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-08-21T03:11:01.4530171Z REENABLED_ISSUES= 2024-08-21T03:11:01.4530391Z SHLVL=1 2024-08-21T03:11:01.4530585Z MAX_JOBS=62 2024-08-21T03:11:01.4530805Z GITHUB_ACTOR_ID=54816060 2024-08-21T03:11:01.4531159Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-21T03:11:01.4531553Z GITHUB_REF_NAME=ciflow/trunk/133712 2024-08-21T03:11:01.4531841Z ROCM_PATH=/opt/rocm 2024-08-21T03:11:01.4532254Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-21T03:11:01.4532682Z GITHUB_JOB=test 2024-08-21T03:11:01.4532905Z NO_TEST_TIMEOUT=False 2024-08-21T03:11:01.4533166Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-21T03:11:01.4533459Z LC_ALL=C.UTF-8 2024-08-21T03:11:01.4533691Z GITHUB_RETENTION_DAYS=90 2024-08-21T03:11:01.4533953Z GITHUB_ACTION_REPOSITORY= 2024-08-21T03:11:01.4534884Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-21T03:11:01.4535830Z GITHUB_BASE_REF= 2024-08-21T03:11:01.4536045Z CI=true 2024-08-21T03:11:01.4536257Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-08-21T03:11:01.4536692Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-21T03:11:01.4537100Z JOB_ID=29025885013 2024-08-21T03:11:01.4537336Z INSTALLED_PROTOBUF=yes 2024-08-21T03:11:01.4537582Z GITHUB_HEAD_REF= 2024-08-21T03:11:01.4537806Z GITHUB_ACTION_REF= 2024-08-21T03:11:01.4538128Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-21T03:11:01.4538499Z TEST_SHOWLOCALS=False 2024-08-21T03:11:01.4538750Z GITHUB_WORKFLOW=trunk 2024-08-21T03:11:01.4539005Z DEBIAN_FRONTEND=noninteractive 2024-08-21T03:11:01.4539692Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_570e791d-9f15-48f4-bf92-4925bd5790ad 2024-08-21T03:11:01.4540298Z NO_TD=False 2024-08-21T03:11:01.4540517Z OLDPWD=/var/lib/jenkins 2024-08-21T03:11:01.4540769Z _=/usr/bin/env 2024-08-21T03:11:01.4541022Z + echo 'Testing pytorch' 2024-08-21T03:11:01.4541272Z Testing pytorch 2024-08-21T03:11:01.4541515Z + export LANG=C.UTF-8 2024-08-21T03:11:01.4541763Z + LANG=C.UTF-8 2024-08-21T03:11:01.4541981Z + PR_NUMBER= 2024-08-21T03:11:01.4542226Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-08-21T03:11:01.4542575Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-21T03:11:01.4542967Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-21T03:11:01.4543298Z + export HIP_VISIBLE_DEVICES=0,1 2024-08-21T03:11:01.4543589Z + HIP_VISIBLE_DEVICES=0,1 2024-08-21T03:11:01.4543849Z + [[ distributed == \s\l\o\w ]] 2024-08-21T03:11:01.4544228Z + [[ linux-focal-rocm6.1-py3.8 == *slow-gradcheck* ]] 2024-08-21T03:11:01.4544644Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-21T03:11:01.4545009Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-21T03:11:01.4545369Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-21T03:11:01.4545710Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-21T03:11:01.4546031Z + [[ distributed == *crossref* ]] 2024-08-21T03:11:01.4546375Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-21T03:11:01.4546683Z + export VALGRIND=OFF 2024-08-21T03:11:01.4546921Z + VALGRIND=OFF 2024-08-21T03:11:01.4547147Z + rocminfo 2024-08-21T03:11:01.4743722Z ROCk module version 6.7.0 is loaded 2024-08-21T03:11:01.5362413Z ===================== 2024-08-21T03:11:01.5363027Z HSA System Attributes 2024-08-21T03:11:01.5363580Z ===================== 2024-08-21T03:11:01.5364116Z Runtime Version: 1.13 2024-08-21T03:11:01.5364678Z Runtime Ext Version: 1.4 2024-08-21T03:11:01.5365263Z System Timestamp Freq.: 1000.000000MHz 2024-08-21T03:11:01.5366242Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-08-21T03:11:01.5367469Z Machine Model: LARGE 2024-08-21T03:11:01.5368510Z System Endianness: LITTLE 2024-08-21T03:11:01.5369381Z Mwaitx: DISABLED 2024-08-21T03:11:01.5370060Z DMAbuf Support: YES 2024-08-21T03:11:01.5370613Z 2024-08-21T03:11:01.5370813Z ========== 2024-08-21T03:11:01.5371420Z HSA Agents 2024-08-21T03:11:01.5372019Z ========== 2024-08-21T03:11:01.5372332Z ******* 2024-08-21T03:11:01.5372607Z Agent 1 2024-08-21T03:11:01.5372850Z ******* 2024-08-21T03:11:01.5373260Z Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.5373726Z Uuid: CPU-XX 2024-08-21T03:11:01.5374199Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.5374640Z Vendor Name: CPU 2024-08-21T03:11:01.5375049Z Feature: None specified 2024-08-21T03:11:01.5375451Z Profile: FULL_PROFILE 2024-08-21T03:11:01.5375877Z Float Round Mode: NEAR 2024-08-21T03:11:01.5376315Z Max Queue Number: 0(0x0) 2024-08-21T03:11:01.5376734Z Queue Min Size: 0(0x0) 2024-08-21T03:11:01.5381998Z Queue Max Size: 0(0x0) 2024-08-21T03:11:01.5382431Z Queue Type: MULTI 2024-08-21T03:11:01.5382824Z Node: 0 2024-08-21T03:11:01.5383212Z Device Type: CPU 2024-08-21T03:11:01.5383566Z Cache Info: 2024-08-21T03:11:01.5383889Z L1: 32768(0x8000) KB 2024-08-21T03:11:01.5384291Z Chip ID: 0(0x0) 2024-08-21T03:11:01.5384693Z ASIC Revision: 0(0x0) 2024-08-21T03:11:01.5385107Z Cacheline Size: 64(0x40) 2024-08-21T03:11:01.5385524Z Max Clock Freq. (MHz): 2600 2024-08-21T03:11:01.5385914Z BDFID: 0 2024-08-21T03:11:01.5386305Z Internal Node ID: 0 2024-08-21T03:11:01.5386719Z Compute Unit: 32 2024-08-21T03:11:01.5387113Z SIMDs per CU: 0 2024-08-21T03:11:01.5387522Z Shader Engines: 0 2024-08-21T03:11:01.5387939Z Shader Arrs. per Eng.: 0 2024-08-21T03:11:01.5388374Z WatchPts on Addr. Ranges:1 2024-08-21T03:11:01.5388738Z Features: None 2024-08-21T03:11:01.5389018Z Pool Info: 2024-08-21T03:11:01.5389280Z Pool 1 2024-08-21T03:11:01.5389633Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:11:01.5390058Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:11:01.5390477Z Allocatable: TRUE 2024-08-21T03:11:01.5390907Z Alloc Granule: 4KB 2024-08-21T03:11:01.5391369Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5391827Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5392264Z Accessible by all: TRUE 2024-08-21T03:11:01.5392640Z Pool 2 2024-08-21T03:11:01.5392986Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-21T03:11:01.5393408Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:11:01.5393815Z Allocatable: TRUE 2024-08-21T03:11:01.5394226Z Alloc Granule: 4KB 2024-08-21T03:11:01.5394681Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5395120Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5395548Z Accessible by all: TRUE 2024-08-21T03:11:01.5395922Z Pool 3 2024-08-21T03:11:01.5396261Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:11:01.5396681Z Size: 65839404(0x3eca12c) KB 2024-08-21T03:11:01.5397100Z Allocatable: TRUE 2024-08-21T03:11:01.5397530Z Alloc Granule: 4KB 2024-08-21T03:11:01.5397989Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5398439Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5398892Z Accessible by all: TRUE 2024-08-21T03:11:01.5399266Z ISA Info: 2024-08-21T03:11:01.5399529Z ******* 2024-08-21T03:11:01.5399777Z Agent 2 2024-08-21T03:11:01.5400019Z ******* 2024-08-21T03:11:01.5400453Z Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.5401274Z Uuid: CPU-XX 2024-08-21T03:11:01.5401753Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.5402184Z Vendor Name: CPU 2024-08-21T03:11:01.5402594Z Feature: None specified 2024-08-21T03:11:01.5402995Z Profile: FULL_PROFILE 2024-08-21T03:11:01.5403409Z Float Round Mode: NEAR 2024-08-21T03:11:01.5403829Z Max Queue Number: 0(0x0) 2024-08-21T03:11:01.5404237Z Queue Min Size: 0(0x0) 2024-08-21T03:11:01.5404639Z Queue Max Size: 0(0x0) 2024-08-21T03:11:01.5405036Z Queue Type: MULTI 2024-08-21T03:11:01.5405419Z Node: 1 2024-08-21T03:11:01.5405816Z Device Type: CPU 2024-08-21T03:11:01.5406156Z Cache Info: 2024-08-21T03:11:01.5406470Z L1: 32768(0x8000) KB 2024-08-21T03:11:01.5406866Z Chip ID: 0(0x0) 2024-08-21T03:11:01.5407264Z ASIC Revision: 0(0x0) 2024-08-21T03:11:01.5407665Z Cacheline Size: 64(0x40) 2024-08-21T03:11:01.5408080Z Max Clock Freq. (MHz): 2600 2024-08-21T03:11:01.5408469Z BDFID: 0 2024-08-21T03:11:01.5408863Z Internal Node ID: 1 2024-08-21T03:11:01.5409270Z Compute Unit: 32 2024-08-21T03:11:01.5409667Z SIMDs per CU: 0 2024-08-21T03:11:01.5410080Z Shader Engines: 0 2024-08-21T03:11:01.5410491Z Shader Arrs. per Eng.: 0 2024-08-21T03:11:01.5410909Z WatchPts on Addr. Ranges:1 2024-08-21T03:11:01.5411266Z Features: None 2024-08-21T03:11:01.5411534Z Pool Info: 2024-08-21T03:11:01.5411790Z Pool 1 2024-08-21T03:11:01.5412133Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:11:01.5412553Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:11:01.5412967Z Allocatable: TRUE 2024-08-21T03:11:01.5413389Z Alloc Granule: 4KB 2024-08-21T03:11:01.5413833Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5414276Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5414717Z Accessible by all: TRUE 2024-08-21T03:11:01.5415079Z Pool 2 2024-08-21T03:11:01.5415417Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-08-21T03:11:01.5415839Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:11:01.5416246Z Allocatable: TRUE 2024-08-21T03:11:01.5416663Z Alloc Granule: 4KB 2024-08-21T03:11:01.5417092Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5417526Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5417944Z Accessible by all: TRUE 2024-08-21T03:11:01.5418295Z Pool 3 2024-08-21T03:11:01.5418621Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:11:01.5419144Z Size: 65997884(0x3ef0c3c) KB 2024-08-21T03:11:01.5419638Z Allocatable: TRUE 2024-08-21T03:11:01.5420049Z Alloc Granule: 4KB 2024-08-21T03:11:01.5420485Z Alloc Recommended Granule:4KB 2024-08-21T03:11:01.5420913Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5421328Z Accessible by all: TRUE 2024-08-21T03:11:01.5421677Z ISA Info: 2024-08-21T03:11:01.5421924Z ******* 2024-08-21T03:11:01.5422167Z Agent 3 2024-08-21T03:11:01.5422410Z ******* 2024-08-21T03:11:01.5422710Z Name: gfx90a 2024-08-21T03:11:01.5423156Z Uuid: GPU-09623e4b9a2a2fde 2024-08-21T03:11:01.5423574Z Marketing Name: AMD Instinct MI210 2024-08-21T03:11:01.5424010Z Vendor Name: AMD 2024-08-21T03:11:01.5424411Z Feature: KERNEL_DISPATCH 2024-08-21T03:11:01.5424827Z Profile: BASE_PROFILE 2024-08-21T03:11:01.5425238Z Float Round Mode: NEAR 2024-08-21T03:11:01.5425657Z Max Queue Number: 128(0x80) 2024-08-21T03:11:01.5426073Z Queue Min Size: 64(0x40) 2024-08-21T03:11:01.5426486Z Queue Max Size: 131072(0x20000) 2024-08-21T03:11:01.5426889Z Queue Type: MULTI 2024-08-21T03:11:01.5427270Z Node: 2 2024-08-21T03:11:01.5427666Z Device Type: GPU 2024-08-21T03:11:01.5428011Z Cache Info: 2024-08-21T03:11:01.5428337Z L1: 16(0x10) KB 2024-08-21T03:11:01.5428732Z L2: 8192(0x2000) KB 2024-08-21T03:11:01.5429135Z Chip ID: 29711(0x740f) 2024-08-21T03:11:01.5429543Z ASIC Revision: 1(0x1) 2024-08-21T03:11:01.5429970Z Cacheline Size: 64(0x40) 2024-08-21T03:11:01.5430388Z Max Clock Freq. (MHz): 1700 2024-08-21T03:11:01.5430788Z BDFID: 768 2024-08-21T03:11:01.5431188Z Internal Node ID: 2 2024-08-21T03:11:01.5431600Z Compute Unit: 104 2024-08-21T03:11:01.5431995Z SIMDs per CU: 4 2024-08-21T03:11:01.5432399Z Shader Engines: 8 2024-08-21T03:11:01.5432896Z Shader Arrs. per Eng.: 1 2024-08-21T03:11:01.5433330Z WatchPts on Addr. Ranges:4 2024-08-21T03:11:01.5433768Z Coherent Host Access: FALSE 2024-08-21T03:11:01.5434158Z Features: KERNEL_DISPATCH 2024-08-21T03:11:01.5434555Z Fast F16 Operation: TRUE 2024-08-21T03:11:01.5434977Z Wavefront Size: 64(0x40) 2024-08-21T03:11:01.5435392Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:11:01.5435761Z Workgroup Max Size per Dimension: 2024-08-21T03:11:01.5436102Z x 1024(0x400) 2024-08-21T03:11:01.5436471Z y 1024(0x400) 2024-08-21T03:11:01.5436847Z z 1024(0x400) 2024-08-21T03:11:01.5437258Z Max Waves Per CU: 32(0x20) 2024-08-21T03:11:01.5437957Z Max Work-item Per CU: 2048(0x800) 2024-08-21T03:11:01.5438401Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:11:01.5438764Z Grid Max Size per Dimension: 2024-08-21T03:11:01.5439075Z x 4294967295(0xffffffff) 2024-08-21T03:11:01.5439445Z y 4294967295(0xffffffff) 2024-08-21T03:11:01.5439813Z z 4294967295(0xffffffff) 2024-08-21T03:11:01.5440302Z Max fbarriers/Workgrp: 32 2024-08-21T03:11:01.5440822Z Packet Processor uCode:: 82 2024-08-21T03:11:01.5441247Z SDMA engine uCode:: 8 2024-08-21T03:11:01.5441662Z IOMMU Support:: None 2024-08-21T03:11:01.5442015Z Pool Info: 2024-08-21T03:11:01.5442280Z Pool 1 2024-08-21T03:11:01.5442638Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:11:01.5443067Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5443486Z Allocatable: TRUE 2024-08-21T03:11:01.5443915Z Alloc Granule: 4KB 2024-08-21T03:11:01.5444366Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5444813Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5445246Z Accessible by all: FALSE 2024-08-21T03:11:01.5445613Z Pool 2 2024-08-21T03:11:01.5445967Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-21T03:11:01.5446393Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5446804Z Allocatable: TRUE 2024-08-21T03:11:01.5447228Z Alloc Granule: 4KB 2024-08-21T03:11:01.5447672Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5448116Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5448546Z Accessible by all: FALSE 2024-08-21T03:11:01.5448901Z Pool 3 2024-08-21T03:11:01.5449231Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:11:01.5449648Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5450052Z Allocatable: TRUE 2024-08-21T03:11:01.5450457Z Alloc Granule: 4KB 2024-08-21T03:11:01.5450898Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5451331Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5451766Z Accessible by all: FALSE 2024-08-21T03:11:01.5452116Z Pool 4 2024-08-21T03:11:01.5452431Z Segment: GROUP 2024-08-21T03:11:01.5452830Z Size: 64(0x40) KB 2024-08-21T03:11:01.5453226Z Allocatable: FALSE 2024-08-21T03:11:01.5453638Z Alloc Granule: 0KB 2024-08-21T03:11:01.5454085Z Alloc Recommended Granule:0KB 2024-08-21T03:11:01.5454526Z Alloc Alignment: 0KB 2024-08-21T03:11:01.5454949Z Accessible by all: FALSE 2024-08-21T03:11:01.5455315Z ISA Info: 2024-08-21T03:11:01.5455581Z ISA 1 2024-08-21T03:11:01.5455984Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-21T03:11:01.5456706Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-21T03:11:01.5457152Z Profiles: HSA_PROFILE_BASE 2024-08-21T03:11:01.5457598Z Default Rounding Mode: NEAR 2024-08-21T03:11:01.5458042Z Default Rounding Mode: NEAR 2024-08-21T03:11:01.5458464Z Fast f16: TRUE 2024-08-21T03:11:01.5458880Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:11:01.5459261Z Workgroup Max Size per Dimension: 2024-08-21T03:11:01.5459620Z x 1024(0x400) 2024-08-21T03:11:01.5460004Z y 1024(0x400) 2024-08-21T03:11:01.5460381Z z 1024(0x400) 2024-08-21T03:11:01.5460790Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:11:01.5461186Z Grid Max Size per Dimension: 2024-08-21T03:11:01.5461516Z x 4294967295(0xffffffff) 2024-08-21T03:11:01.5461886Z y 4294967295(0xffffffff) 2024-08-21T03:11:01.5462266Z z 4294967295(0xffffffff) 2024-08-21T03:11:01.5462670Z FBarrier Max Size: 32 2024-08-21T03:11:01.5463023Z ******* 2024-08-21T03:11:01.5463267Z Agent 4 2024-08-21T03:11:01.5463510Z ******* 2024-08-21T03:11:01.5463804Z Name: gfx90a 2024-08-21T03:11:01.5464260Z Uuid: GPU-17f2a9b321b37cb8 2024-08-21T03:11:01.5464684Z Marketing Name: AMD Instinct MI210 2024-08-21T03:11:01.5465094Z Vendor Name: AMD 2024-08-21T03:11:01.5465507Z Feature: KERNEL_DISPATCH 2024-08-21T03:11:01.5465909Z Profile: BASE_PROFILE 2024-08-21T03:11:01.5466315Z Float Round Mode: NEAR 2024-08-21T03:11:01.5466734Z Max Queue Number: 128(0x80) 2024-08-21T03:11:01.5467142Z Queue Min Size: 64(0x40) 2024-08-21T03:11:01.5467551Z Queue Max Size: 131072(0x20000) 2024-08-21T03:11:01.5467955Z Queue Type: MULTI 2024-08-21T03:11:01.5468335Z Node: 3 2024-08-21T03:11:01.5468714Z Device Type: GPU 2024-08-21T03:11:01.5469056Z Cache Info: 2024-08-21T03:11:01.5469367Z L1: 16(0x10) KB 2024-08-21T03:11:01.5469748Z L2: 8192(0x2000) KB 2024-08-21T03:11:01.5470149Z Chip ID: 29711(0x740f) 2024-08-21T03:11:01.5470564Z ASIC Revision: 1(0x1) 2024-08-21T03:11:01.5470970Z Cacheline Size: 64(0x40) 2024-08-21T03:11:01.5471386Z Max Clock Freq. (MHz): 1700 2024-08-21T03:11:01.5471773Z BDFID: 33536 2024-08-21T03:11:01.5472160Z Internal Node ID: 3 2024-08-21T03:11:01.5472563Z Compute Unit: 104 2024-08-21T03:11:01.5472957Z SIMDs per CU: 4 2024-08-21T03:11:01.5473352Z Shader Engines: 8 2024-08-21T03:11:01.5473763Z Shader Arrs. per Eng.: 1 2024-08-21T03:11:01.5474187Z WatchPts on Addr. Ranges:4 2024-08-21T03:11:01.5474839Z Coherent Host Access: FALSE 2024-08-21T03:11:01.5475226Z Features: KERNEL_DISPATCH 2024-08-21T03:11:01.5475613Z Fast F16 Operation: TRUE 2024-08-21T03:11:01.5476050Z Wavefront Size: 64(0x40) 2024-08-21T03:11:01.5476468Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:11:01.5476848Z Workgroup Max Size per Dimension: 2024-08-21T03:11:01.5477190Z x 1024(0x400) 2024-08-21T03:11:01.5477551Z y 1024(0x400) 2024-08-21T03:11:01.5477906Z z 1024(0x400) 2024-08-21T03:11:01.5478294Z Max Waves Per CU: 32(0x20) 2024-08-21T03:11:01.5478763Z Max Work-item Per CU: 2048(0x800) 2024-08-21T03:11:01.5479185Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:11:01.5479568Z Grid Max Size per Dimension: 2024-08-21T03:11:01.5479877Z x 4294967295(0xffffffff) 2024-08-21T03:11:01.5480400Z y 4294967295(0xffffffff) 2024-08-21T03:11:01.5480774Z z 4294967295(0xffffffff) 2024-08-21T03:11:01.5481176Z Max fbarriers/Workgrp: 32 2024-08-21T03:11:01.5481611Z Packet Processor uCode:: 82 2024-08-21T03:11:01.5482034Z SDMA engine uCode:: 8 2024-08-21T03:11:01.5482450Z IOMMU Support:: None 2024-08-21T03:11:01.5482795Z Pool Info: 2024-08-21T03:11:01.5483057Z Pool 1 2024-08-21T03:11:01.5483391Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-08-21T03:11:01.5483845Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5484263Z Allocatable: TRUE 2024-08-21T03:11:01.5484696Z Alloc Granule: 4KB 2024-08-21T03:11:01.5485138Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5485588Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5486023Z Accessible by all: FALSE 2024-08-21T03:11:01.5486391Z Pool 2 2024-08-21T03:11:01.5486733Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-08-21T03:11:01.5487152Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5487554Z Allocatable: TRUE 2024-08-21T03:11:01.5487969Z Alloc Granule: 4KB 2024-08-21T03:11:01.5488401Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5488851Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5489281Z Accessible by all: FALSE 2024-08-21T03:11:01.5489634Z Pool 3 2024-08-21T03:11:01.5489960Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-08-21T03:11:01.5490383Z Size: 67092480(0x3ffc000) KB 2024-08-21T03:11:01.5490795Z Allocatable: TRUE 2024-08-21T03:11:01.5491214Z Alloc Granule: 4KB 2024-08-21T03:11:01.5491653Z Alloc Recommended Granule:2048KB 2024-08-21T03:11:01.5492100Z Alloc Alignment: 4KB 2024-08-21T03:11:01.5492528Z Accessible by all: FALSE 2024-08-21T03:11:01.5492886Z Pool 4 2024-08-21T03:11:01.5493470Z Segment: GROUP 2024-08-21T03:11:01.5493880Z Size: 64(0x40) KB 2024-08-21T03:11:01.5494287Z Allocatable: FALSE 2024-08-21T03:11:01.5494696Z Alloc Granule: 0KB 2024-08-21T03:11:01.5495125Z Alloc Recommended Granule:0KB 2024-08-21T03:11:01.5495563Z Alloc Alignment: 0KB 2024-08-21T03:11:01.5495978Z Accessible by all: FALSE 2024-08-21T03:11:01.5496330Z ISA Info: 2024-08-21T03:11:01.5496588Z ISA 1 2024-08-21T03:11:01.5496984Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-08-21T03:11:01.5497433Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-08-21T03:11:01.5497873Z Profiles: HSA_PROFILE_BASE 2024-08-21T03:11:01.5498304Z Default Rounding Mode: NEAR 2024-08-21T03:11:01.5498745Z Default Rounding Mode: NEAR 2024-08-21T03:11:01.5499155Z Fast f16: TRUE 2024-08-21T03:11:01.5499567Z Workgroup Max Size: 1024(0x400) 2024-08-21T03:11:01.5499943Z Workgroup Max Size per Dimension: 2024-08-21T03:11:01.5500292Z x 1024(0x400) 2024-08-21T03:11:01.5500664Z y 1024(0x400) 2024-08-21T03:11:01.5501033Z z 1024(0x400) 2024-08-21T03:11:01.5501447Z Grid Max Size: 4294967295(0xffffffff) 2024-08-21T03:11:01.5501810Z Grid Max Size per Dimension: 2024-08-21T03:11:01.5502130Z x 4294967295(0xffffffff) 2024-08-21T03:11:01.5502507Z y 4294967295(0xffffffff) 2024-08-21T03:11:01.5502882Z z 4294967295(0xffffffff) 2024-08-21T03:11:01.5503284Z FBarrier Max Size: 32 2024-08-21T03:11:01.5503640Z *** Done *** 2024-08-21T03:11:01.5535508Z + rocminfo 2024-08-21T03:11:01.5536923Z + grep -E 'Name:.*\sgfx|Marketing' 2024-08-21T03:11:01.6210625Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.6212007Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-08-21T03:11:01.6212814Z Name: gfx90a 2024-08-21T03:11:01.6213368Z Marketing Name: AMD Instinct MI210 2024-08-21T03:11:01.6213868Z Name: gfx90a 2024-08-21T03:11:01.6214392Z Marketing Name: AMD Instinct MI210 2024-08-21T03:11:01.6379469Z + [[ linux-focal-rocm6.1-py3.8 == *xpu* ]] 2024-08-21T03:11:01.6380808Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-08-21T03:11:01.6381752Z + pip_install --user ninja==1.10.2 2024-08-21T03:11:01.6382704Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-21T03:11:01.9697876Z Collecting ninja==1.10.2 2024-08-21T03:11:01.9957056Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-21T03:11:02.0059604Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-21T03:11:02.2650260Z Installing collected packages: ninja 2024-08-21T03:11:02.2726581Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-21T03:11:02.2729361Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-21T03:11:02.2756010Z Successfully installed ninja-1.10.2 2024-08-21T03:11:02.3176013Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-21T03:11:02.3181115Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-21T03:11:02.3184262Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-08-21T03:11:02.3185057Z + install_tlparse 2024-08-21T03:11:02.3185729Z + pip_install --user tlparse==0.3.25 2024-08-21T03:11:02.3186721Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-21T03:11:02.6487061Z Collecting tlparse==0.3.25 2024-08-21T03:11:02.6633261Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-21T03:11:02.6934776Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-21T03:11:03.0050550Z Installing collected packages: tlparse 2024-08-21T03:11:03.0344090Z Successfully installed tlparse-0.3.25 2024-08-21T03:11:03.0774913Z ++ python -m site --user-base 2024-08-21T03:11:03.1017895Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.8/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-21T03:11:03.1021116Z + [[ linux-focal-rocm6.1-py3.8 == *asan* ]] 2024-08-21T03:11:03.1022090Z + [[ linux-focal-rocm6.1-py3.8 == *-debug* ]] 2024-08-21T03:11:03.1023035Z + [[ linux-focal-rocm6.1-py3.8 != *-bazel-* ]] 2024-08-21T03:11:03.1024460Z + echo 'We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass' 2024-08-21T03:11:03.1026363Z We are not in debug mode: linux-focal-rocm6.1-py3.8. Expect the assertion to pass 2024-08-21T03:11:03.1027487Z + cd test 2024-08-21T03:11:03.1028419Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-21T03:11:07.5978702Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-21T03:11:07.5979161Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-21T03:11:07.5982131Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-21T03:11:07.5982503Z + [[ distributed == *pr_time_benchmarks* ]] 2024-08-21T03:11:07.5982881Z + [[ distributed == *dynamo_eager* ]] 2024-08-21T03:11:07.5983219Z + [[ distributed == *aot_eager* ]] 2024-08-21T03:11:07.5983546Z + [[ distributed == *aot_inductor* ]] 2024-08-21T03:11:07.5983860Z + [[ distributed == *inductor* ]] 2024-08-21T03:11:07.5984168Z + [[ distributed == *dynamic* ]] 2024-08-21T03:11:07.5984469Z + [[ distributed == *cpu* ]] 2024-08-21T03:11:07.5985005Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-21T03:11:07.6014557Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-08-21T03:11:07.6015074Z + [[ linux-focal-rocm6.1-py3.8 == *-bazel-* ]] 2024-08-21T03:11:07.6017435Z + cd test 2024-08-21T03:11:07.6017917Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-21T03:11:10.2580028Z PyTorch built with: 2024-08-21T03:11:10.2581046Z - GCC 9.4 2024-08-21T03:11:10.2581700Z - C++ Version: 201703 2024-08-21T03:11:10.2583331Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-21T03:11:10.2585277Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-21T03:11:10.2586456Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-21T03:11:10.2587380Z - LAPACK is enabled (usually provided by MKL) 2024-08-21T03:11:10.2588240Z - NNPACK is enabled 2024-08-21T03:11:10.2588919Z - CPU capability usage: AVX2 2024-08-21T03:11:10.2589647Z - HIP Runtime 6.1.40091 2024-08-21T03:11:10.2590298Z - MIOpen 3.1.0 2024-08-21T03:11:10.2590877Z - Magma 2.7.2 2024-08-21T03:11:10.2605465Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-21T03:11:10.2616340Z 2024-08-21T03:11:10.7211419Z + cd test 2024-08-21T03:11:10.7212831Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-21T03:11:11.8543945Z ATen/Parallel: 2024-08-21T03:11:11.8544716Z at::get_num_threads() : 64 2024-08-21T03:11:11.8545444Z at::get_num_interop_threads() : 64 2024-08-21T03:11:11.8546243Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-21T03:11:11.8546968Z omp_get_max_threads() : 64 2024-08-21T03:11:11.8548772Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-21T03:11:11.8550248Z mkl_get_max_threads() : 64 2024-08-21T03:11:11.8551607Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-21T03:11:11.8552946Z std::thread::hardware_concurrency() : 64 2024-08-21T03:11:11.8553851Z Environment variables: 2024-08-21T03:11:11.8554648Z OMP_NUM_THREADS : [not set] 2024-08-21T03:11:11.8555441Z MKL_NUM_THREADS : [not set] 2024-08-21T03:11:11.8555910Z ATen parallel backend: OpenMP 2024-08-21T03:11:11.8556191Z 2024-08-21T03:11:12.2902076Z + [[ linux-focal-rocm6.1-py3.8 == *aarch64* ]] 2024-08-21T03:11:12.2902589Z + [[ distributed == *backward* ]] 2024-08-21T03:11:12.2902967Z + [[ distributed == *xla* ]] 2024-08-21T03:11:12.2903342Z + [[ distributed == *executorch* ]] 2024-08-21T03:11:12.2903754Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-21T03:11:12.2904256Z + [[ linux-focal-rocm6.1-py3.8 == *libtorch* ]] 2024-08-21T03:11:12.2904709Z + [[ distributed == distributed ]] 2024-08-21T03:11:12.2905071Z + test_distributed 2024-08-21T03:11:12.2905452Z + echo 'Testing distributed python tests' 2024-08-21T03:11:12.2905875Z Testing distributed python tests 2024-08-21T03:11:12.2907350Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-08-21T03:11:12.3845762Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-21T03:11:12.3847525Z import pkg_resources 2024-08-21T03:11:16.5823412Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-21T03:11:16.9919882Z Ignoring disabled issues: [''] 2024-08-21T03:11:17.0137950Z Found test times from artifacts 2024-08-21T03:11:17.0647256Z Found test times from artifacts 2024-08-21T03:11:17.0658752Z Running all tests 2024-08-21T03:11:17.0660501Z Running parallel tests on 2 processes 2024-08-21T03:11:17.0661362Z Name: tests to run (est. time: 25.91min) 2024-08-21T03:11:17.0662158Z Serial tests (4): 2024-08-21T03:11:17.0662813Z distributed/test_c10d_common 1/1 2024-08-21T03:11:17.0664353Z distributed/test_c10d_nccl 1/3 2024-08-21T03:11:17.0665464Z distributed/test_c10d_nccl 2/3 2024-08-21T03:11:17.0666180Z distributed/test_c10d_nccl 3/3 2024-08-21T03:11:17.0666868Z Parallel tests (0): 2024-08-21T03:11:17.0667482Z Name: excluded (est. time: 0.0min) 2024-08-21T03:11:17.0668141Z Serial tests (0): 2024-08-21T03:11:17.0668706Z Parallel tests (0): 2024-08-21T03:11:17.0923576Z Running distributed/test_c10d_common 1/1 ... [2024-08-21 03:11:17.091800] 2024-08-21T03:11:17.0924270Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:11:17.0929330Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-21 03:11:17.092516] 2024-08-21T03:14:14.7855661Z 2024-08-21T03:14:14.7858340Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_80a1a09aaa611f1f_.log 2024-08-21T03:14:14.7880765Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-08-21T03:14:14.7892586Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-08-21T03:14:14.7893700Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-08-21T03:14:14.7894914Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-08-21T03:14:14.7896133Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-08-21T03:14:14.7897343Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-08-21T03:14:14.7898411Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-08-21T03:14:14.7900147Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-08-21T03:14:14.7901363Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-08-21T03:14:14.7902526Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-08-21T03:14:14.7903834Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-08-21T03:14:14.7906564Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-08-21T03:14:14.7909093Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-08-21T03:14:14.7911366Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-08-21T03:14:14.7913906Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-08-21T03:14:14.7916767Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-08-21T03:14:14.7919224Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-08-21T03:14:14.7921325Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-08-21T03:14:14.7923216Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-08-21T03:14:14.7925095Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-08-21T03:14:14.7926979Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-08-21T03:14:14.7929000Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-08-21T03:14:14.7931007Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-08-21T03:14:14.7932922Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-08-21T03:14:14.7934091Z 2024-08-21T03:14:14.7934740Z Running distributed/test_c10d_nccl 1/3 ... [2024-08-21 03:14:14.785892] 2024-08-21T03:14:14.7935287Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:14:14.7936866Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-21 03:14:14.786230] 2024-08-21T03:28:37.1754329Z 2024-08-21T03:28:37.1761982Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_f4174f76fe572400_.log 2024-08-21T03:28:37.1795392Z Running 70 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-08-21T03:28:37.1830324Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-08-21T03:28:37.1831929Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-08-21T03:28:37.1833745Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-08-21T03:28:37.1835416Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-08-21T03:28:37.1837081Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-08-21T03:28:37.1838734Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg 2024-08-21T03:28:37.1839976Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-08-21T03:28:37.1841174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization 2024-08-21T03:28:37.1842408Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2024-08-21T03:28:37.1843605Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-08-21T03:28:37.1844766Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-08-21T03:28:37.1845935Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-08-21T03:28:37.1847040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-08-21T03:28:37.1848195Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-08-21T03:28:37.1849579Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-08-21T03:28:37.1850890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-08-21T03:28:37.1852101Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-08-21T03:28:37.1853437Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-08-21T03:28:37.1854811Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-08-21T03:28:37.1856199Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-08-21T03:28:37.1857432Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-08-21T03:28:37.1858600Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-08-21T03:28:37.1859825Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-08-21T03:28:37.1861115Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-08-21T03:28:37.1862415Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-08-21T03:28:37.1863776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-08-21T03:28:37.1865194Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-08-21T03:28:37.1866402Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-08-21T03:28:37.1867515Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-08-21T03:28:37.1869208Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-08-21T03:28:37.1871837Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-08-21T03:28:37.1874310Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-08-21T03:28:37.1876591Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-08-21T03:28:37.1879098Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-08-21T03:28:37.1881952Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-08-21T03:28:37.1884492Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-08-21T03:28:37.1886658Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-08-21T03:28:37.1889013Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-08-21T03:28:37.1891755Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-08-21T03:28:37.1893610Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-08-21T03:28:37.1895073Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-08-21T03:28:37.1896120Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-08-21T03:28:37.1897149Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-08-21T03:28:37.1898282Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-08-21T03:28:37.1900552Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-08-21T03:28:37.1902823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-08-21T03:28:37.1905025Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-08-21T03:28:37.1907078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-08-21T03:28:37.1909071Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-08-21T03:28:37.1911149Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-08-21T03:28:37.1913283Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-08-21T03:28:37.1915336Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-08-21T03:28:37.1917635Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-08-21T03:28:37.1920427Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-08-21T03:28:37.1923302Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-08-21T03:28:37.1925688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-08-21T03:28:37.1927610Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-08-21T03:28:37.1929360Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-08-21T03:28:37.1930310Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-08-21T03:28:37.1931294Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-08-21T03:28:37.1932280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-08-21T03:28:37.1933372Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-08-21T03:28:37.1934424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-08-21T03:28:37.1935522Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-08-21T03:28:37.1936849Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-08-21T03:28:37.1938164Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-08-21T03:28:37.1940628Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-08-21T03:28:37.1943790Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-08-21T03:28:37.1946092Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-08-21T03:28:37.1948369Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-08-21T03:28:37.1949747Z 2024-08-21T03:28:37.1950429Z Running distributed/test_c10d_nccl 2/3 ... [2024-08-21 03:28:37.177409] 2024-08-21T03:28:37.1951469Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:28:37.1954545Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-21 03:28:37.177979] 2024-08-21T03:39:22.2276211Z 2024-08-21T03:39:22.2283523Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_23cbdfdea1a62acf_.log 2024-08-21T03:39:22.2320103Z Running 61 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-08-21T03:39:22.2352421Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-08-21T03:39:22.2354074Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-08-21T03:39:22.2355825Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-08-21T03:39:22.2357548Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-08-21T03:39:22.2359357Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2024-08-21T03:39:22.2361453Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-08-21T03:39:22.2363399Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-08-21T03:39:22.2365153Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-08-21T03:39:22.2366911Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-08-21T03:39:22.2368859Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-08-21T03:39:22.2371176Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-08-21T03:39:22.2373362Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-08-21T03:39:22.2374718Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-08-21T03:39:22.2375985Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-08-21T03:39:22.2377269Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-08-21T03:39:22.2378667Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-08-21T03:39:22.2381075Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-08-21T03:39:22.2383174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-08-21T03:39:22.2385198Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-08-21T03:39:22.2387107Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-08-21T03:39:22.2388959Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-08-21T03:39:22.2391041Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-08-21T03:39:22.2393297Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-08-21T03:39:22.2395190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-08-21T03:39:22.2396979Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-08-21T03:39:22.2398934Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-08-21T03:39:22.2401337Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-08-21T03:39:22.2403455Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-08-21T03:39:22.2405554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-08-21T03:39:22.2407424Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-08-21T03:39:22.2409977Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-08-21T03:39:22.2412103Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-08-21T03:39:22.2413790Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-08-21T03:39:22.2414941Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-08-21T03:39:22.2416045Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-08-21T03:39:22.2417113Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-08-21T03:39:22.2418119Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-08-21T03:39:22.2419364Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-08-21T03:39:22.2420767Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-08-21T03:39:22.2422162Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-08-21T03:39:22.2423681Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-08-21T03:39:22.2425181Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-08-21T03:39:22.2426733Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-08-21T03:39:22.2428292Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-08-21T03:39:22.2430006Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-08-21T03:39:22.2432311Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-08-21T03:39:22.2434734Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-08-21T03:39:22.2436992Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-08-21T03:39:22.2439067Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-08-21T03:39:22.2440467Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-08-21T03:39:22.2441561Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-08-21T03:39:22.2442793Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-08-21T03:39:22.2444122Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-08-21T03:39:22.2445438Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-08-21T03:39:22.2446735Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-08-21T03:39:22.2447984Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-08-21T03:39:22.2449252Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-08-21T03:39:22.2450860Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-08-21T03:39:22.2452164Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-08-21T03:39:22.2453450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-08-21T03:39:22.2454703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-08-21T03:39:22.2455387Z 2024-08-21T03:39:22.2455738Z Running distributed/test_c10d_nccl 3/3 ... [2024-08-21 03:39:22.229654] 2024-08-21T03:39:22.2456264Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:39:22.2457823Z Executing ['/opt/conda/envs/py_3.8/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-08-21 03:39:22.230091] 2024-08-21T03:50:39.9653828Z 2024-08-21T03:50:39.9656070Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_b8715ab436dcc27a_.log 2024-08-21T03:50:39.9692090Z Running 56 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-08-21T03:50:39.9713006Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-08-21T03:50:39.9713918Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-08-21T03:50:39.9714764Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-08-21T03:50:39.9715693Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-08-21T03:50:39.9716625Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-08-21T03:50:39.9717594Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-08-21T03:50:39.9718611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-08-21T03:50:39.9719861Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-08-21T03:50:39.9720917Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-08-21T03:50:39.9721874Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-08-21T03:50:39.9722905Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-08-21T03:50:39.9724002Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-08-21T03:50:39.9725150Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-08-21T03:50:39.9726264Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-08-21T03:50:39.9727309Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-08-21T03:50:39.9728393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-08-21T03:50:39.9729501Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-08-21T03:50:39.9730563Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-08-21T03:50:39.9731511Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-08-21T03:50:39.9732408Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-08-21T03:50:39.9733410Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-08-21T03:50:39.9734456Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-08-21T03:50:39.9735486Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-08-21T03:50:39.9736554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-08-21T03:50:39.9737560Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-08-21T03:50:39.9738546Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-08-21T03:50:39.9739511Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-08-21T03:50:39.9740423Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-08-21T03:50:39.9741276Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-08-21T03:50:39.9742075Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-08-21T03:50:39.9742919Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-08-21T03:50:39.9743796Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-08-21T03:50:39.9744899Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-08-21T03:50:39.9745794Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-08-21T03:50:39.9746674Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-08-21T03:50:39.9747470Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-08-21T03:50:39.9748232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-08-21T03:50:39.9749174Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-08-21T03:50:39.9750303Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-08-21T03:50:39.9751434Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-08-21T03:50:39.9752569Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-08-21T03:50:39.9753608Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-08-21T03:50:39.9754497Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-08-21T03:50:39.9755353Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-08-21T03:50:39.9756154Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-08-21T03:50:39.9757022Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0 2024-08-21T03:50:39.9757951Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1 2024-08-21T03:50:39.9758943Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-08-21T03:50:39.9759975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-08-21T03:50:39.9761047Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-08-21T03:50:39.9762048Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-08-21T03:50:39.9763087Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-08-21T03:50:39.9764140Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-08-21T03:50:39.9765173Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-08-21T03:50:39.9766146Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-08-21T03:50:39.9767055Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-08-21T03:50:39.9767578Z 2024-08-21T03:50:40.5779499Z Running test batch 'tests to run' cost 2363.51 seconds 2024-08-21T03:50:41.2744448Z 2024-08-21T03:50:41.2744909Z real 39m28.984s 2024-08-21T03:50:41.2745288Z user 54m2.918s 2024-08-21T03:50:41.2749153Z sys 57m10.719s 2024-08-21T03:50:41.2749674Z + assert_git_not_dirty 2024-08-21T03:50:41.2750328Z + [[ linux-focal-rocm6.1-py3.8 != *rocm* ]] 2024-08-21T03:50:41.2750818Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-21T03:50:41.2751296Z + [[ linux-focal-rocm6.1-py3.8 == *rocm* ]] 2024-08-21T03:50:41.2751687Z + [[ 1 == 1 ]] 2024-08-21T03:50:41.2752022Z + echo 'Testing distributed C++ tests' 2024-08-21T03:50:41.2752409Z Testing distributed C++ tests 2024-08-21T03:50:41.2755593Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:50:41.2780150Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:50:41.2796634Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:50:41.2797877Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:50:41.2799289Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-08-21T03:50:41.3629739Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-21T03:50:41.3631509Z import pkg_resources 2024-08-21T03:50:45.0764153Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-21T03:50:45.0844258Z Found test times from artifacts 2024-08-21T03:50:45.1195453Z Found test times from artifacts 2024-08-21T03:50:45.1207931Z Running all tests 2024-08-21T03:50:45.1216824Z Running parallel tests on 2 processes 2024-08-21T03:50:45.1217736Z Name: tests to run (est. time: 0.0min) 2024-08-21T03:50:45.1218471Z Serial tests (1): 2024-08-21T03:50:45.1219077Z cpp/FileStoreTest 1/1 2024-08-21T03:50:45.1219734Z Parallel tests (0): 2024-08-21T03:50:45.1220350Z Name: excluded (est. time: 0.0min) 2024-08-21T03:50:45.1221112Z Serial tests (0): 2024-08-21T03:50:45.1221758Z Parallel tests (0): 2024-08-21T03:50:45.1462024Z Running cpp/FileStoreTest 1/1 ... [2024-08-21 03:50:45.145609] 2024-08-21T03:50:45.1463104Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:50:45.1477845Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b8acf0e04b139b84.xml', '-x', '--reruns=2'] ... [2024-08-21 03:50:45.147041] 2024-08-21T03:50:47.6802262Z 2024-08-21T03:50:47.6804157Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_3da961228a0815ca_.log 2024-08-21T03:50:47.6804984Z 2024-08-21T03:50:48.7144954Z Running test batch 'tests to run' cost 3.59 seconds 2024-08-21T03:50:49.3420315Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-08-21T03:50:49.4262840Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-21T03:50:49.4264635Z import pkg_resources 2024-08-21T03:50:53.2036031Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-21T03:50:53.2115778Z Found test times from artifacts 2024-08-21T03:50:53.2470805Z Found test times from artifacts 2024-08-21T03:50:53.2484414Z Running all tests 2024-08-21T03:50:53.2485671Z Running parallel tests on 2 processes 2024-08-21T03:50:53.2496169Z Name: tests to run (est. time: 0.0min) 2024-08-21T03:50:53.2496567Z Serial tests (1): 2024-08-21T03:50:53.2496893Z cpp/HashStoreTest 1/1 2024-08-21T03:50:53.2497218Z Parallel tests (0): 2024-08-21T03:50:53.2497538Z Name: excluded (est. time: 0.0min) 2024-08-21T03:50:53.2497903Z Serial tests (0): 2024-08-21T03:50:53.2498186Z Parallel tests (0): 2024-08-21T03:50:53.2730330Z Running cpp/HashStoreTest 1/1 ... [2024-08-21 03:50:53.272457] 2024-08-21T03:50:53.2731448Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:50:53.2743569Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-581d5f21d6a027bb.xml', '-x', '--reruns=2'] ... [2024-08-21 03:50:53.273660] 2024-08-21T03:50:55.6576722Z 2024-08-21T03:50:55.6579600Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_c2d82a303c75e90b_.log 2024-08-21T03:50:55.6581270Z 2024-08-21T03:50:56.8664074Z Running test batch 'tests to run' cost 3.62 seconds 2024-08-21T03:50:57.4951305Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-08-21T03:50:57.5782711Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-21T03:50:57.5784453Z import pkg_resources 2024-08-21T03:51:01.3050103Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-21T03:51:01.3130430Z Found test times from artifacts 2024-08-21T03:51:01.3481005Z Found test times from artifacts 2024-08-21T03:51:01.3494093Z Running all tests 2024-08-21T03:51:01.3497437Z Running parallel tests on 2 processes 2024-08-21T03:51:01.3498285Z Name: tests to run (est. time: 0.0min) 2024-08-21T03:51:01.3499020Z Serial tests (1): 2024-08-21T03:51:01.3499668Z cpp/TCPStoreTest 1/1 2024-08-21T03:51:01.3500318Z Parallel tests (0): 2024-08-21T03:51:01.3500939Z Name: excluded (est. time: 0.0min) 2024-08-21T03:51:01.3501662Z Serial tests (0): 2024-08-21T03:51:01.3502213Z Parallel tests (0): 2024-08-21T03:51:01.3748318Z Running cpp/TCPStoreTest 1/1 ... [2024-08-21 03:51:01.374222] 2024-08-21T03:51:01.3749409Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:51:01.3760983Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7a78eb3223299354.xml', '-x', '--reruns=2'] ... [2024-08-21 03:51:01.375385] 2024-08-21T03:51:15.2559404Z 2024-08-21T03:51:15.2561868Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_6248d13f137dad6d_.log 2024-08-21T03:51:15.2563538Z 2024-08-21T03:51:15.9145735Z Running test batch 'tests to run' cost 14.56 seconds 2024-08-21T03:51:16.5536971Z + [[ linux-focal-rocm6.1-py3.8 == *cuda* ]] 2024-08-21T03:51:16.5537852Z + [[ 1 == 1 ]] 2024-08-21T03:51:16.5538451Z + test_rpc 2024-08-21T03:51:16.5539079Z + echo 'Testing RPC C++ tests' 2024-08-21T03:51:16.5539782Z Testing RPC C++ tests 2024-08-21T03:51:16.5546078Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:51:16.5579375Z + ln -sf /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:51:16.5600037Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin 2024-08-21T03:51:16.5601839Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-08-21T03:51:16.6419064Z test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-08-21T03:51:16.6420857Z import pkg_resources 2024-08-21T03:51:20.3454107Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-08-21T03:51:20.3533984Z Found test times from artifacts 2024-08-21T03:51:20.3886283Z Found test times from artifacts 2024-08-21T03:51:20.3899388Z Running all tests 2024-08-21T03:51:20.3902269Z Running parallel tests on 2 processes 2024-08-21T03:51:20.3903102Z Name: tests to run (est. time: 0.0min) 2024-08-21T03:51:20.3903864Z Serial tests (1): 2024-08-21T03:51:20.3904536Z cpp/test_cpp_rpc 1/1 2024-08-21T03:51:20.3905215Z Parallel tests (0): 2024-08-21T03:51:20.3905850Z Name: excluded (est. time: 0.0min) 2024-08-21T03:51:20.3906527Z Serial tests (0): 2024-08-21T03:51:20.3907287Z Parallel tests (0): 2024-08-21T03:51:20.4143451Z Running cpp/test_cpp_rpc 1/1 ... [2024-08-21 03:51:20.413768] 2024-08-21T03:51:20.4144594Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-08-21T03:51:20.4156802Z Executing ['pytest', '/opt/conda/envs/py_3.8/lib/python3.8/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-117e917cf79e19c2.xml', '-x', '--reruns=2'] ... [2024-08-21 03:51:20.414918] 2024-08-21T03:51:28.8615759Z 2024-08-21T03:51:28.8616796Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_bcd19c6e6d88261d_.log 2024-08-21T03:51:28.8621160Z 2024-08-21T03:51:29.4874807Z Running test batch 'tests to run' cost 9.1 seconds 2024-08-21T03:51:30.1252585Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-08-21T03:51:30.1253704Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-08-21T03:51:30.1254960Z docker exec -t "ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-08-21T03:51:30.1297813Z shell: /usr/bin/bash -e {0} 2024-08-21T03:51:30.1298152Z env: 2024-08-21T03:51:30.1298424Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:30.1298813Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:30.1299512Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:30.1300188Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:30.1300551Z AWS_REGION: us-east-1 2024-08-21T03:51:30.1300962Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:30.1301502Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:30.1314381Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:30.1315419Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:30.1316526Z ##[endgroup] 2024-08-21T03:51:30.1877695Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T03:51:30.2113163Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T03:51:30.2114095Z cat test/**/*_toprint.log || true 2024-08-21T03:51:30.2155159Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:51:30.2155996Z env: 2024-08-21T03:51:30.2156466Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:30.2157173Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:30.2158433Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:30.2159614Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:30.2160348Z AWS_REGION: us-east-1 2024-08-21T03:51:30.2161526Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:30.2162404Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:30.2174943Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:30.2175779Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:30.2176426Z ##[endgroup] 2024-08-21T03:51:30.2310114Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T03:51:30.2367899Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T03:51:30.2368807Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T03:51:30.2411909Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:51:30.2412992Z env: 2024-08-21T03:51:30.2413593Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:30.2414486Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:30.2416082Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:30.2416888Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:30.2417290Z AWS_REGION: us-east-1 2024-08-21T03:51:30.2417727Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:30.2418219Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:30.2424639Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:30.2425181Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:30.2425778Z MONITOR_SCRIPT_PID: 1761261 2024-08-21T03:51:30.2426111Z ##[endgroup] 2024-08-21T03:51:30.2763481Z Prepare all required actions 2024-08-21T03:51:30.2764340Z Getting action download info 2024-08-21T03:51:30.7968938Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-21T03:51:31.4034981Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T03:51:31.4035341Z with: 2024-08-21T03:51:31.4035547Z use-gha: true 2024-08-21T03:51:31.4035876Z file-suffix: test-distributed-1-1-linux.rocm.gpu_29025885013 2024-08-21T03:51:31.4036277Z s3-bucket: gha-artifacts 2024-08-21T03:51:31.4036527Z env: 2024-08-21T03:51:31.4036734Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:31.4037050Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:31.4037603Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:31.4038139Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:31.4038416Z AWS_REGION: us-east-1 2024-08-21T03:51:31.4038748Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:31.4039126Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:31.4044272Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:31.4044695Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:31.4045146Z ##[endgroup] 2024-08-21T03:51:31.4111442Z ##[group]Run actions/upload-artifact@v3 2024-08-21T03:51:31.4111764Z with: 2024-08-21T03:51:31.4112188Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip 2024-08-21T03:51:31.4112708Z retention-days: 14 2024-08-21T03:51:31.4112969Z if-no-files-found: warn 2024-08-21T03:51:31.4113247Z path: test/**/*.json 2024-08-21T03:51:31.4113501Z env: 2024-08-21T03:51:31.4113720Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:31.4114034Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:31.4114598Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:31.4115129Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:31.4115432Z AWS_REGION: us-east-1 2024-08-21T03:51:31.4115752Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:31.4116134Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:31.4121302Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:31.4121741Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:31.4122217Z ##[endgroup] 2024-08-21T03:51:31.8107539Z With the provided path, there will be 10 files uploaded 2024-08-21T03:51:31.8109180Z Starting artifact upload 2024-08-21T03:51:31.8110957Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-21T03:51:31.8112519Z Artifact name is valid! 2024-08-21T03:51:32.1514063Z Container for artifact "test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip" successfully created. Starting upload of file(s) 2024-08-21T03:51:33.0786686Z Total size of all the files uploaded is 41131 bytes 2024-08-21T03:51:33.0788008Z File upload process has finished. Finalizing the artifact upload 2024-08-21T03:51:33.1593612Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-21T03:51:33.1594478Z 2024-08-21T03:51:33.1595093Z The raw size of all the files that were specified for upload is 361624 bytes 2024-08-21T03:51:33.1597282Z The size of all the files that were uploaded is 41131 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-21T03:51:33.1598808Z 2024-08-21T03:51:33.1601008Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-21T03:51:33.1602702Z 2024-08-21T03:51:33.1603870Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip has been successfully uploaded! 2024-08-21T03:51:33.1829540Z ##[group]Run actions/upload-artifact@v3 2024-08-21T03:51:33.1830830Z with: 2024-08-21T03:51:33.1831885Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip 2024-08-21T03:51:33.1833142Z retention-days: 14 2024-08-21T03:51:33.1833749Z if-no-files-found: ignore 2024-08-21T03:51:33.1834427Z path: test/**/*.xml test/**/*.csv 2024-08-21T03:51:33.1835124Z env: 2024-08-21T03:51:33.1835613Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:33.1836351Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:33.1837692Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:33.1838962Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:33.1839630Z AWS_REGION: us-east-1 2024-08-21T03:51:33.1840654Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:33.1841565Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:33.1854620Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:33.1855840Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:33.1857162Z ##[endgroup] 2024-08-21T03:51:33.6315100Z With the provided path, there will be 214 files uploaded 2024-08-21T03:51:33.6316513Z Starting artifact upload 2024-08-21T03:51:33.6319104Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-21T03:51:33.6321566Z Artifact name is valid! 2024-08-21T03:51:33.9724211Z Container for artifact "test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip" successfully created. Starting upload of file(s) 2024-08-21T03:51:43.9782704Z Total file count: 214 ---- Processed file #162 (75.7%) 2024-08-21T03:51:47.0936939Z Total size of all the files uploaded is 52758 bytes 2024-08-21T03:51:47.0938260Z File upload process has finished. Finalizing the artifact upload 2024-08-21T03:51:47.2176547Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-21T03:51:47.2177439Z 2024-08-21T03:51:47.2178064Z The raw size of all the files that were specified for upload is 80740 bytes 2024-08-21T03:51:47.2180188Z The size of all the files that were uploaded is 52758 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-21T03:51:47.2181690Z 2024-08-21T03:51:47.2183517Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-21T03:51:47.2185303Z 2024-08-21T03:51:47.2186565Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip has been successfully uploaded! 2024-08-21T03:51:47.2429555Z ##[group]Run actions/upload-artifact@v3 2024-08-21T03:51:47.2430312Z with: 2024-08-21T03:51:47.2431239Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip 2024-08-21T03:51:47.2432365Z retention-days: 14 2024-08-21T03:51:47.2432969Z if-no-files-found: ignore 2024-08-21T03:51:47.2433681Z path: usage_log.txt test/**/*.log 2024-08-21T03:51:47.2434401Z env: 2024-08-21T03:51:47.2434903Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:47.2435652Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:47.2437040Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:47.2438318Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:47.2438994Z AWS_REGION: us-east-1 2024-08-21T03:51:47.2439730Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:47.2440806Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:47.2453406Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:47.2454652Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:47.2455961Z ##[endgroup] 2024-08-21T03:51:47.6817882Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-08-21T03:51:47.6821800Z The least common ancestor is /home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-08-21T03:51:47.6823628Z With the provided path, there will be 9 files uploaded 2024-08-21T03:51:47.6825006Z Starting artifact upload 2024-08-21T03:51:47.6827633Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-08-21T03:51:47.6830001Z Artifact name is valid! 2024-08-21T03:51:48.0216245Z Container for artifact "logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip" successfully created. Starting upload of file(s) 2024-08-21T03:51:48.7600517Z Total size of all the files uploaded is 43189 bytes 2024-08-21T03:51:48.7601812Z File upload process has finished. Finalizing the artifact upload 2024-08-21T03:51:48.8416073Z Artifact has been finalized. All files have been successfully uploaded! 2024-08-21T03:51:48.8416997Z 2024-08-21T03:51:48.8417627Z The raw size of all the files that were specified for upload is 676389 bytes 2024-08-21T03:51:48.8419762Z The size of all the files that were uploaded is 43189 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-08-21T03:51:48.8421301Z 2024-08-21T03:51:48.8423136Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-08-21T03:51:48.8424904Z 2024-08-21T03:51:48.8426076Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_29025885013.zip has been successfully uploaded! 2024-08-21T03:51:48.8623756Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T03:51:48.8624780Z # shellcheck disable=SC2156 2024-08-21T03:51:48.8626430Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T03:51:48.8670828Z shell: /usr/bin/bash -e {0} 2024-08-21T03:51:48.8671468Z env: 2024-08-21T03:51:48.8671974Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:48.8672725Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:48.8674056Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:48.8675317Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:48.8676006Z AWS_REGION: us-east-1 2024-08-21T03:51:48.8676739Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:48.8677641Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:48.8690306Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:48.8691372Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:48.8692917Z ##[endgroup] 2024-08-21T03:51:49.1537236Z Prepare all required actions 2024-08-21T03:51:49.1538256Z Getting action download info 2024-08-21T03:51:49.1604981Z ##[group]Run ./.github/actions/teardown-rocm 2024-08-21T03:51:49.1605758Z env: 2024-08-21T03:51:49.1606265Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:49.1607028Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:49.1608395Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:49.1609688Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:49.1610365Z AWS_REGION: us-east-1 2024-08-21T03:51:49.1611097Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:49.1612003Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:49.1625584Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:49.1626804Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:49.1628013Z ##[endgroup] 2024-08-21T03:51:49.1662874Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T03:51:49.1664274Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T03:51:49.1665473Z # shellcheck disable=SC2046 2024-08-21T03:51:49.1666444Z docker stop $(docker ps -q) || true 2024-08-21T03:51:49.1667455Z # Prune all stopped containers. 2024-08-21T03:51:49.1668287Z docker container prune -f 2024-08-21T03:51:49.1717448Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:51:49.1718336Z env: 2024-08-21T03:51:49.1718840Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:51:49.1719578Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:51:49.1721097Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:51:49.1722357Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:51:49.1723025Z AWS_REGION: us-east-1 2024-08-21T03:51:49.1723730Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:51:49.1724618Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:51:49.1737749Z AWS_SESSION_TOKEN: *** 2024-08-21T03:51:49.1738948Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:51:49.1740104Z ##[endgroup] 2024-08-21T03:51:49.6850960Z ecbe6fc5f802 2024-08-21T03:52:06.5796622Z Deleted Containers: 2024-08-21T03:52:06.5797202Z ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:52:06.5797607Z 2024-08-21T03:52:06.5797753Z Total reclaimed space: 6.733GB 2024-08-21T03:52:06.5854454Z Prepare all required actions 2024-08-21T03:52:06.5884358Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-08-21T03:52:06.5884706Z with: 2024-08-21T03:52:06.5884930Z diskspace-cutoff: 70 2024-08-21T03:52:06.5885176Z env: 2024-08-21T03:52:06.5885391Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:52:06.5885713Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:52:06.5886283Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:52:06.5886854Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:52:06.5887154Z AWS_REGION: us-east-1 2024-08-21T03:52:06.5887482Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:52:06.5887877Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:52:06.5893009Z AWS_SESSION_TOKEN: *** 2024-08-21T03:52:06.5893445Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:52:06.5893910Z ##[endgroup] 2024-08-21T03:52:06.5908734Z ##[group]Run set -ex 2024-08-21T03:52:06.5909002Z set -ex 2024-08-21T03:52:06.5909244Z diskspace_cutoff=70 2024-08-21T03:52:06.5909623Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-08-21T03:52:06.5910256Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-21T03:52:06.5911198Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-08-21T03:52:06.5912424Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-08-21T03:52:06.5912828Z  docker system prune -af 2024-08-21T03:52:06.5913379Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-08-21T03:52:06.5914010Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-08-21T03:52:06.5914664Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-08-21T03:52:06.5915218Z  echo "$msg" 2024-08-21T03:52:06.5915500Z  exit 1 2024-08-21T03:52:06.5915752Z  else 2024-08-21T03:52:06.5916055Z  difference=$((diskspace - diskspace_new)) 2024-08-21T03:52:06.5916492Z  echo "Diskspace saved: $difference percent" 2024-08-21T03:52:06.5916852Z  fi 2024-08-21T03:52:06.5917078Z fi 2024-08-21T03:52:06.5938769Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T03:52:06.5939231Z env: 2024-08-21T03:52:06.5939454Z GIT_DEFAULT_BRANCH: main 2024-08-21T03:52:06.5939773Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-08-21T03:52:06.5940333Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-08-21T03:52:06.5940863Z AWS_DEFAULT_REGION: us-east-1 2024-08-21T03:52:06.5941156Z AWS_REGION: us-east-1 2024-08-21T03:52:06.5941456Z AWS_ACCESS_KEY_ID: *** 2024-08-21T03:52:06.5941830Z AWS_SECRET_ACCESS_KEY: *** 2024-08-21T03:52:06.5946943Z AWS_SESSION_TOKEN: *** 2024-08-21T03:52:06.5947369Z CONTAINER_NAME: ecbe6fc5f802e9b519af40c915898958543d07834842c29576cec784d6db9bad 2024-08-21T03:52:06.5947826Z ##[endgroup] 2024-08-21T03:52:06.5990035Z + diskspace_cutoff=70 2024-08-21T03:52:06.5997456Z ++ docker info -f '{{.DockerRootDir}}' 2024-08-21T03:52:06.6560979Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-08-21T03:52:06.6566848Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-08-21T03:52:06.6567688Z ++ sed -n 2p 2024-08-21T03:52:06.6569225Z ++ sed s/%// 2024-08-21T03:52:06.6570413Z ++ sed 's/ //' 2024-08-21T03:52:06.6587960Z + diskspace=22 2024-08-21T03:52:06.6588863Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-08-21T03:52:06.6589725Z + [[ 22 -ge 70 ]] 2024-08-21T03:52:06.6651821Z Post job cleanup. 2024-08-21T03:52:06.6711886Z Post job cleanup. 2024-08-21T03:52:06.7039157Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-21T03:52:06.7364029Z Post job cleanup. 2024-08-21T03:52:06.8590182Z Post job cleanup. 2024-08-21T03:52:06.8696371Z Post job cleanup. 2024-08-21T03:52:06.8837018Z Post job cleanup. 2024-08-21T03:52:06.9473789Z [command]/usr/bin/git version 2024-08-21T03:52:06.9509899Z git version 2.34.1 2024-08-21T03:52:06.9547307Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/701d18cd-c55e-48d7-b66a-d5fad3d3fa1c' before making global git config changes 2024-08-21T03:52:06.9549727Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T03:52:06.9551768Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-08-21T03:52:06.9576919Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T03:52:06.9599572Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T03:52:06.9895228Z Entering 'android/libs/fbjni' 2024-08-21T03:52:06.9948231Z Entering 'third_party/FP16' 2024-08-21T03:52:07.0009888Z Entering 'third_party/FXdiv' 2024-08-21T03:52:07.0060756Z Entering 'third_party/NNPACK' 2024-08-21T03:52:07.0104754Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:52:07.0149229Z Entering 'third_party/XNNPACK' 2024-08-21T03:52:07.0211655Z Entering 'third_party/benchmark' 2024-08-21T03:52:07.0257921Z Entering 'third_party/cpp-httplib' 2024-08-21T03:52:07.0304441Z Entering 'third_party/cpuinfo' 2024-08-21T03:52:07.0345678Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:52:07.0386857Z Entering 'third_party/cutlass' 2024-08-21T03:52:07.0434736Z Entering 'third_party/eigen' 2024-08-21T03:52:07.0486711Z Entering 'third_party/fbgemm' 2024-08-21T03:52:07.0527466Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:52:07.0571099Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:52:07.0609516Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:52:07.0652859Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:52:07.0689858Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:52:07.0728524Z Entering 'third_party/flatbuffers' 2024-08-21T03:52:07.0788741Z Entering 'third_party/fmt' 2024-08-21T03:52:07.0830092Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:52:07.0878408Z Entering 'third_party/gloo' 2024-08-21T03:52:07.0918305Z Entering 'third_party/googletest' 2024-08-21T03:52:07.0960625Z Entering 'third_party/ideep' 2024-08-21T03:52:07.1005634Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:52:07.1049958Z Entering 'third_party/ittapi' 2024-08-21T03:52:07.1090063Z Entering 'third_party/kineto' 2024-08-21T03:52:07.1128818Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:52:07.1172341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:52:07.1219117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:52:07.1267992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:52:07.1308061Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:52:07.1351873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:52:07.1401832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:52:07.1445853Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:52:07.1490177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:52:07.1530551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:52:07.1570645Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:52:07.1608158Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:52:07.1652494Z Entering 'third_party/mimalloc' 2024-08-21T03:52:07.1697807Z Entering 'third_party/nccl/nccl' 2024-08-21T03:52:07.1739410Z Entering 'third_party/nlohmann' 2024-08-21T03:52:07.1780133Z Entering 'third_party/onnx' 2024-08-21T03:52:07.1834610Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:52:07.1879037Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:52:07.1925966Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:52:07.1972647Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:52:07.2015170Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:52:07.2056762Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:52:07.2101499Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:52:07.2142160Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:52:07.2179472Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:52:07.2216939Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:52:07.2252007Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:52:07.2299384Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:52:07.2345702Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:52:07.2409177Z Entering 'third_party/pocketfft' 2024-08-21T03:52:07.2450351Z Entering 'third_party/protobuf' 2024-08-21T03:52:07.2490052Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:52:07.2530056Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:52:07.2569837Z Entering 'third_party/psimd' 2024-08-21T03:52:07.2615297Z Entering 'third_party/pthreadpool' 2024-08-21T03:52:07.2660569Z Entering 'third_party/pybind11' 2024-08-21T03:52:07.2708526Z Entering 'third_party/python-peachpy' 2024-08-21T03:52:07.2755581Z Entering 'third_party/sleef' 2024-08-21T03:52:07.2794915Z Entering 'third_party/tensorpipe' 2024-08-21T03:52:07.2838874Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:52:07.2890742Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:52:07.2931556Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:52:07.2984536Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:52:07.3026522Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:52:07.3097285Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T03:52:07.3114905Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3128556Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T03:52:07.3155729Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T03:52:07.3421189Z Entering 'android/libs/fbjni' 2024-08-21T03:52:07.3448244Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3479484Z Entering 'third_party/FP16' 2024-08-21T03:52:07.3502363Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3537608Z Entering 'third_party/FXdiv' 2024-08-21T03:52:07.3570238Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3600633Z Entering 'third_party/NNPACK' 2024-08-21T03:52:07.3627659Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3663860Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T03:52:07.3686799Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3724160Z Entering 'third_party/XNNPACK' 2024-08-21T03:52:07.3751829Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3812225Z Entering 'third_party/benchmark' 2024-08-21T03:52:07.3834962Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3872051Z Entering 'third_party/cpp-httplib' 2024-08-21T03:52:07.3894453Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3926855Z Entering 'third_party/cpuinfo' 2024-08-21T03:52:07.3949058Z http.https://github.com/.extraheader 2024-08-21T03:52:07.3986589Z Entering 'third_party/cudnn_frontend' 2024-08-21T03:52:07.4006145Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4038898Z Entering 'third_party/cutlass' 2024-08-21T03:52:07.4059730Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4102940Z Entering 'third_party/eigen' 2024-08-21T03:52:07.4128340Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4165946Z Entering 'third_party/fbgemm' 2024-08-21T03:52:07.4185714Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4223120Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T03:52:07.4251739Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4285401Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T03:52:07.4305963Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4342283Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T03:52:07.4366417Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4400088Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T03:52:07.4424942Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4465485Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T03:52:07.4488627Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4530088Z Entering 'third_party/flatbuffers' 2024-08-21T03:52:07.4551023Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4596046Z Entering 'third_party/fmt' 2024-08-21T03:52:07.4616942Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4650626Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T03:52:07.4679427Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4713041Z Entering 'third_party/gloo' 2024-08-21T03:52:07.4733341Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4764981Z Entering 'third_party/googletest' 2024-08-21T03:52:07.4788059Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4824564Z Entering 'third_party/ideep' 2024-08-21T03:52:07.4846780Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4878800Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T03:52:07.4900812Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4941448Z Entering 'third_party/ittapi' 2024-08-21T03:52:07.4964445Z http.https://github.com/.extraheader 2024-08-21T03:52:07.4996693Z Entering 'third_party/kineto' 2024-08-21T03:52:07.5020883Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5056815Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T03:52:07.5082603Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5118495Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T03:52:07.5141904Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5181168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T03:52:07.5208682Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5240966Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T03:52:07.5262283Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5291192Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T03:52:07.5310610Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5339185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T03:52:07.5363525Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5396833Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T03:52:07.5418952Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5457214Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T03:52:07.5478991Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5514050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T03:52:07.5533422Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5571463Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T03:52:07.5592280Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5623862Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T03:52:07.5643969Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5675244Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T03:52:07.5698510Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5740343Z Entering 'third_party/mimalloc' 2024-08-21T03:52:07.5769983Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5805183Z Entering 'third_party/nccl/nccl' 2024-08-21T03:52:07.5829946Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5864182Z Entering 'third_party/nlohmann' 2024-08-21T03:52:07.5887853Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5919370Z Entering 'third_party/onnx' 2024-08-21T03:52:07.5943687Z http.https://github.com/.extraheader 2024-08-21T03:52:07.5987890Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T03:52:07.6016073Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6050466Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T03:52:07.6070335Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6102964Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T03:52:07.6124231Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6157690Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T03:52:07.6180815Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6216617Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T03:52:07.6238713Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6277567Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T03:52:07.6299858Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6328115Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T03:52:07.6349415Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6412760Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T03:52:07.6433173Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6462050Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T03:52:07.6481954Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6692239Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T03:52:07.6714781Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6748424Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T03:52:07.6769503Z http.https://github.com/.extraheader 2024-08-21T03:52:07.6800759Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T03:52:07.6823820Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7014214Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T03:52:07.7033850Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7086070Z Entering 'third_party/pocketfft' 2024-08-21T03:52:07.7104952Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7137214Z Entering 'third_party/protobuf' 2024-08-21T03:52:07.7158015Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7206104Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T03:52:07.7227700Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7262599Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T03:52:07.7285782Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7315869Z Entering 'third_party/psimd' 2024-08-21T03:52:07.7339349Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7371053Z Entering 'third_party/pthreadpool' 2024-08-21T03:52:07.7391760Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7421327Z Entering 'third_party/pybind11' 2024-08-21T03:52:07.7441765Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7476063Z Entering 'third_party/python-peachpy' 2024-08-21T03:52:07.7499579Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7531984Z Entering 'third_party/sleef' 2024-08-21T03:52:07.7552880Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7582644Z Entering 'third_party/tensorpipe' 2024-08-21T03:52:07.7602967Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7642675Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T03:52:07.7665599Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7701099Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T03:52:07.7719844Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7760589Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T03:52:07.7785608Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7821281Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T03:52:07.7846630Z http.https://github.com/.extraheader 2024-08-21T03:52:07.7880865Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T03:52:07.7904727Z http.https://github.com/.extraheader 2024-08-21T03:52:07.8189850Z Cleaning up orphan processes