2024-10-08T20:53:40.1307141Z Current runner version: '2.320.0' 2024-10-08T20:53:40.1312529Z Runner name: 'pytorch-rocm-hw-35' 2024-10-08T20:53:40.1313164Z Runner group name: 'linux.rocm.gpu.group' 2024-10-08T20:53:40.1313940Z Machine name: 'pytorch-rocm-hw-35' 2024-10-08T20:53:40.1327513Z Testing runner upgrade compatibility 2024-10-08T20:53:40.2077803Z ##[group]GITHUB_TOKEN Permissions 2024-10-08T20:53:40.2079869Z Contents: read 2024-10-08T20:53:40.2080609Z Metadata: read 2024-10-08T20:53:40.2081784Z ##[endgroup] 2024-10-08T20:53:40.2088303Z Secret source: Actions 2024-10-08T20:53:40.2089714Z Prepare workflow directory 2024-10-08T20:53:40.5900803Z Prepare all required actions 2024-10-08T20:53:40.6063257Z Getting action download info 2024-10-08T20:53:41.1201054Z Download action repository 'pytorch/pytorch@main' (SHA:5e3e1c01515bac92bbe303d3fdbfee571949a96f) 2024-10-08T20:53:48.0503903Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-10-08T20:53:48.6144900Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-10-08T20:53:49.1332573Z Download action repository 'pytorch/test-infra@main' (SHA:d420136f568806947b1ab3a763151299d32d5621) 2024-10-08T20:53:50.1032873Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-10-08T20:53:50.6694329Z Getting action download info 2024-10-08T20:53:50.8378473Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-10-08T20:53:51.4313731Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/134247 (47952c0a8001fd60c9ffe52b121d022d440bef8b) 2024-10-08T20:53:51.4315718Z ##[group] Inputs 2024-10-08T20:53:51.4316101Z build-environment: linux-focal-rocm6.2-py3.10 2024-10-08T20:53:51.4317247Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-10-08T20:53:51.4318683Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:53:51.4319425Z sync-tag: 2024-10-08T20:53:51.4320102Z timeout-minutes: 300 2024-10-08T20:53:51.4320930Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-10-08T20:53:51.4321827Z ##[endgroup] 2024-10-08T20:53:51.4322638Z Complete job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T20:53:51.5512788Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-10-08T20:53:51.5513269Z with: 2024-10-08T20:53:51.5513492Z no-sudo: true 2024-10-08T20:53:51.5513729Z submodules: recursive 2024-10-08T20:53:51.5514000Z fetch-depth: 0 2024-10-08T20:53:51.5514234Z env: 2024-10-08T20:53:51.5514451Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:53:51.5514721Z ##[endgroup] 2024-10-08T20:53:51.5698777Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T20:53:51.5699823Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-10-08T20:53:51.5727643Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:53:51.5728067Z env: 2024-10-08T20:53:51.5728306Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:53:51.5728581Z ##[endgroup] 2024-10-08T20:53:51.6167118Z ##[group]Run retry () { 2024-10-08T20:53:51.6167699Z retry () { 2024-10-08T20:53:51.6168407Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-10-08T20:53:51.6169344Z } 2024-10-08T20:53:51.6169795Z echo "${GITHUB_WORKSPACE}" 2024-10-08T20:53:51.6170763Z if [ -z "${NO_SUDO}" ]; then 2024-10-08T20:53:51.6171415Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-10-08T20:53:51.6172041Z else 2024-10-08T20:53:51.6172505Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-10-08T20:53:51.6173205Z fi 2024-10-08T20:53:51.6173689Z mkdir "${GITHUB_WORKSPACE}" 2024-10-08T20:53:51.6209178Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:53:51.6209882Z env: 2024-10-08T20:53:51.6210256Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:53:51.6210729Z NO_SUDO: true 2024-10-08T20:53:51.6211102Z ##[endgroup] 2024-10-08T20:53:51.6321302Z /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T20:53:55.2647624Z ##[group]Run malfet/checkout@silent-checkout 2024-10-08T20:53:55.2648493Z with: 2024-10-08T20:53:55.2649204Z ref: 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T20:53:55.2649961Z fetch-depth: 0 2024-10-08T20:53:55.2650486Z submodules: recursive 2024-10-08T20:53:55.2651088Z quiet-checkout: true 2024-10-08T20:53:55.2651667Z repository: pytorch/pytorch 2024-10-08T20:53:55.2652584Z token: *** 2024-10-08T20:53:55.2653086Z ssh-strict: true 2024-10-08T20:53:55.2653646Z persist-credentials: true 2024-10-08T20:53:55.2654237Z clean: true 2024-10-08T20:53:55.2654815Z sparse-checkout-cone-mode: true 2024-10-08T20:53:55.2655569Z lfs: false 2024-10-08T20:53:55.2656154Z set-safe-directory: true 2024-10-08T20:53:55.2656811Z env: 2024-10-08T20:53:55.2657319Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:53:55.2657992Z ##[endgroup] 2024-10-08T20:53:55.3614297Z Syncing repository: pytorch/pytorch 2024-10-08T20:53:55.3617533Z ##[group]Getting Git version info 2024-10-08T20:53:55.3619049Z Working directory is '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-10-08T20:53:55.3620670Z [command]/usr/bin/git version 2024-10-08T20:53:55.3621331Z git version 2.34.1 2024-10-08T20:53:55.3623833Z ##[endgroup] 2024-10-08T20:53:55.3638947Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/c3c2a6d2-f809-4a6c-b7ee-e1a08dab6320' before making global git config changes 2024-10-08T20:53:55.3641227Z Adding repository directory to the temporary git global config as a safe directory 2024-10-08T20:53:55.3643273Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T20:53:55.3646369Z Deleting the contents of '/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-10-08T20:53:55.3648033Z ##[group]Initializing the repository 2024-10-08T20:53:55.3649432Z [command]/usr/bin/git init /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T20:53:55.3651962Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-10-08T20:53:55.3653486Z hint: is subject to change. To configure the initial branch name to use in all 2024-10-08T20:53:55.3654890Z hint: of your new repositories, which will suppress this warning, call: 2024-10-08T20:53:55.3655880Z hint: 2024-10-08T20:53:55.3656662Z hint: git config --global init.defaultBranch 2024-10-08T20:53:55.3657511Z hint: 2024-10-08T20:53:55.3658431Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-10-08T20:53:55.3659925Z hint: 'development'. The just-created branch can be renamed via this command: 2024-10-08T20:53:55.3660946Z hint: 2024-10-08T20:53:55.3661487Z hint: git branch -m 2024-10-08T20:53:55.3662802Z Initialized empty Git repository in /home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-10-08T20:53:55.3665308Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-10-08T20:53:55.3668782Z ##[endgroup] 2024-10-08T20:53:55.3669366Z ##[group]Disabling automatic garbage collection 2024-10-08T20:53:55.3673136Z [command]/usr/bin/git config --local gc.auto 0 2024-10-08T20:53:55.3709388Z ##[endgroup] 2024-10-08T20:53:55.3709800Z ##[group]Setting up auth 2024-10-08T20:53:55.3718102Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-10-08T20:53:55.3746743Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-10-08T20:53:55.4013920Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-10-08T20:53:55.4037148Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-10-08T20:53:55.4308494Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-10-08T20:53:55.4346486Z ##[endgroup] 2024-10-08T20:53:55.4346960Z ##[group]Fetching the repository 2024-10-08T20:53:55.4351721Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-10-08T20:53:58.6294912Z remote: Enumerating objects: 1011033 2024-10-08T20:53:58.6295990Z remote: Enumerating objects: 1012099, done. 2024-10-08T20:53:58.6297232Z remote: Counting objects: 0% (1/1066) 2024-10-08T20:53:58.6298286Z remote: Counting objects: 1% (11/1066) 2024-10-08T20:53:58.6299336Z remote: Counting objects: 2% (22/1066) 2024-10-08T20:53:58.6300323Z remote: Counting objects: 3% (32/1066) 2024-10-08T20:53:58.6301296Z remote: Counting objects: 4% (43/1066) 2024-10-08T20:53:58.6302212Z remote: Counting objects: 5% (54/1066) 2024-10-08T20:53:58.6303055Z remote: Counting objects: 6% (64/1066) 2024-10-08T20:53:58.6303876Z remote: Counting objects: 7% (75/1066) 2024-10-08T20:53:58.6304704Z remote: Counting objects: 8% (86/1066) 2024-10-08T20:53:58.6305520Z remote: Counting objects: 9% (96/1066) 2024-10-08T20:53:58.6306337Z remote: Counting objects: 10% (107/1066) 2024-10-08T20:53:58.6307219Z remote: Counting objects: 11% (118/1066) 2024-10-08T20:53:58.6308048Z remote: Counting objects: 12% (128/1066) 2024-10-08T20:53:58.6308890Z remote: Counting objects: 13% (139/1066) 2024-10-08T20:53:58.6309701Z remote: Counting objects: 14% (150/1066) 2024-10-08T20:53:58.6310665Z remote: Counting objects: 15% (160/1066) 2024-10-08T20:53:58.6311657Z remote: Counting objects: 16% (171/1066) 2024-10-08T20:53:58.6312623Z remote: Counting objects: 17% (182/1066) 2024-10-08T20:53:58.6313452Z remote: Counting objects: 18% (192/1066) 2024-10-08T20:53:58.6314282Z remote: Counting objects: 19% (203/1066) 2024-10-08T20:53:58.6315740Z remote: Counting objects: 20% (214/1066) 2024-10-08T20:53:58.6316603Z remote: Counting objects: 21% (224/1066) 2024-10-08T20:53:58.6317456Z remote: Counting objects: 22% (235/1066) 2024-10-08T20:53:58.6318286Z remote: Counting objects: 23% (246/1066) 2024-10-08T20:53:58.6319107Z remote: Counting objects: 24% (256/1066) 2024-10-08T20:53:58.6320031Z remote: Counting objects: 25% (267/1066) 2024-10-08T20:53:58.6320871Z remote: Counting objects: 26% (278/1066) 2024-10-08T20:53:58.6321677Z remote: Counting objects: 27% (288/1066) 2024-10-08T20:53:58.6322473Z remote: Counting objects: 28% (299/1066) 2024-10-08T20:53:58.6323310Z remote: Counting objects: 29% (310/1066) 2024-10-08T20:53:58.6324145Z remote: Counting objects: 30% (320/1066) 2024-10-08T20:53:58.6324960Z remote: Counting objects: 31% (331/1066) 2024-10-08T20:53:58.6325759Z remote: Counting objects: 32% (342/1066) 2024-10-08T20:53:58.6326623Z remote: Counting objects: 33% (352/1066) 2024-10-08T20:53:58.6327449Z remote: Counting objects: 34% (363/1066) 2024-10-08T20:53:58.6328243Z remote: Counting objects: 35% (374/1066) 2024-10-08T20:53:58.6329268Z remote: Counting objects: 36% (384/1066) 2024-10-08T20:53:58.6330452Z remote: Counting objects: 37% (395/1066) 2024-10-08T20:53:58.6331261Z remote: Counting objects: 38% (406/1066) 2024-10-08T20:53:58.6332079Z remote: Counting objects: 39% (416/1066) 2024-10-08T20:53:58.6332947Z remote: Counting objects: 40% (427/1066) 2024-10-08T20:53:58.6333795Z remote: Counting objects: 41% (438/1066) 2024-10-08T20:53:58.6334615Z remote: Counting objects: 42% (448/1066) 2024-10-08T20:53:58.6335455Z remote: Counting objects: 43% (459/1066) 2024-10-08T20:53:58.6336267Z remote: Counting objects: 44% (470/1066) 2024-10-08T20:53:58.6337085Z remote: Counting objects: 45% (480/1066) 2024-10-08T20:53:58.6337951Z remote: Counting objects: 46% (491/1066) 2024-10-08T20:53:58.6338760Z remote: Counting objects: 47% (502/1066) 2024-10-08T20:53:58.6339581Z remote: Counting objects: 48% (512/1066) 2024-10-08T20:53:58.6340401Z remote: Counting objects: 49% (523/1066) 2024-10-08T20:53:58.6341212Z remote: Counting objects: 50% (533/1066) 2024-10-08T20:53:58.6342035Z remote: Counting objects: 51% (544/1066) 2024-10-08T20:53:58.6342876Z remote: Counting objects: 52% (555/1066) 2024-10-08T20:53:58.6343677Z remote: Counting objects: 53% (565/1066) 2024-10-08T20:53:58.6344486Z remote: Counting objects: 54% (576/1066) 2024-10-08T20:53:58.6345311Z remote: Counting objects: 55% (587/1066) 2024-10-08T20:53:58.6346102Z remote: Counting objects: 56% (597/1066) 2024-10-08T20:53:58.6346913Z remote: Counting objects: 57% (608/1066) 2024-10-08T20:53:58.6347720Z remote: Counting objects: 58% (619/1066) 2024-10-08T20:53:58.6348526Z remote: Counting objects: 59% (629/1066) 2024-10-08T20:53:58.6349333Z remote: Counting objects: 60% (640/1066) 2024-10-08T20:53:58.6350156Z remote: Counting objects: 61% (651/1066) 2024-10-08T20:53:58.6350980Z remote: Counting objects: 62% (661/1066) 2024-10-08T20:53:58.6351916Z remote: Counting objects: 63% (672/1066) 2024-10-08T20:53:58.6352762Z remote: Counting objects: 64% (683/1066) 2024-10-08T20:53:58.6353576Z remote: Counting objects: 65% (693/1066) 2024-10-08T20:53:58.6354383Z remote: Counting objects: 66% (704/1066) 2024-10-08T20:53:58.6355179Z remote: Counting objects: 67% (715/1066) 2024-10-08T20:53:58.6355989Z remote: Counting objects: 68% (725/1066) 2024-10-08T20:53:58.6356779Z remote: Counting objects: 69% (736/1066) 2024-10-08T20:53:58.6357599Z remote: Counting objects: 70% (747/1066) 2024-10-08T20:53:58.6358411Z remote: Counting objects: 71% (757/1066) 2024-10-08T20:53:58.6359571Z remote: Counting objects: 72% (768/1066) 2024-10-08T20:53:58.6360407Z remote: Counting objects: 73% (779/1066) 2024-10-08T20:53:58.6361222Z remote: Counting objects: 74% (789/1066) 2024-10-08T20:53:58.6362038Z remote: Counting objects: 75% (800/1066) 2024-10-08T20:53:58.6362846Z remote: Counting objects: 76% (811/1066) 2024-10-08T20:53:58.6363644Z remote: Counting objects: 77% (821/1066) 2024-10-08T20:53:58.6364452Z remote: Counting objects: 78% (832/1066) 2024-10-08T20:53:58.6365271Z remote: Counting objects: 79% (843/1066) 2024-10-08T20:53:58.6366074Z remote: Counting objects: 80% (853/1066) 2024-10-08T20:53:58.6366890Z remote: Counting objects: 81% (864/1066) 2024-10-08T20:53:58.6367707Z remote: Counting objects: 82% (875/1066) 2024-10-08T20:53:58.6368510Z remote: Counting objects: 83% (885/1066) 2024-10-08T20:53:58.7011657Z remote: Counting objects: 84% (896/1066) 2024-10-08T20:53:58.7012683Z remote: Counting objects: 85% (907/1066) 2024-10-08T20:53:58.7013582Z remote: Counting objects: 86% (917/1066) 2024-10-08T20:53:58.7014430Z remote: Counting objects: 87% (928/1066) 2024-10-08T20:53:58.7015299Z remote: Counting objects: 88% (939/1066) 2024-10-08T20:53:58.7016669Z remote: Counting objects: 89% (949/1066) 2024-10-08T20:53:58.7017489Z remote: Counting objects: 90% (960/1066) 2024-10-08T20:53:58.7018338Z remote: Counting objects: 91% (971/1066) 2024-10-08T20:53:58.7019327Z remote: Counting objects: 92% (981/1066) 2024-10-08T20:53:58.7020279Z remote: Counting objects: 93% (992/1066) 2024-10-08T20:53:58.7021316Z remote: Counting objects: 94% (1003/1066) 2024-10-08T20:53:58.7022361Z remote: Counting objects: 95% (1013/1066) 2024-10-08T20:53:58.7023402Z remote: Counting objects: 96% (1024/1066) 2024-10-08T20:53:58.7024412Z remote: Counting objects: 97% (1035/1066) 2024-10-08T20:53:58.7025315Z remote: Counting objects: 98% (1045/1066) 2024-10-08T20:53:58.7026160Z remote: Counting objects: 99% (1056/1066) 2024-10-08T20:53:58.7026996Z remote: Counting objects: 100% (1066/1066) 2024-10-08T20:53:58.7027919Z remote: Counting objects: 100% (1066/1066), done. 2024-10-08T20:53:58.7028870Z remote: Compressing objects: 0% (1/540) 2024-10-08T20:53:58.7029723Z remote: Compressing objects: 1% (6/540) 2024-10-08T20:53:58.7030609Z remote: Compressing objects: 2% (11/540) 2024-10-08T20:53:58.7728210Z remote: Compressing objects: 3% (17/540) 2024-10-08T20:53:58.7729425Z remote: Compressing objects: 4% (22/540) 2024-10-08T20:53:58.8446794Z remote: Compressing objects: 5% (27/540) 2024-10-08T20:53:58.9165664Z remote: Compressing objects: 6% (33/540) 2024-10-08T20:53:58.9884275Z remote: Compressing objects: 7% (38/540) 2024-10-08T20:53:58.9885302Z remote: Compressing objects: 8% (44/540) 2024-10-08T20:53:58.9886246Z remote: Compressing objects: 9% (49/540) 2024-10-08T20:53:59.0602058Z remote: Compressing objects: 10% (54/540) 2024-10-08T20:53:59.0603094Z remote: Compressing objects: 11% (60/540) 2024-10-08T20:53:59.0604052Z remote: Compressing objects: 12% (65/540) 2024-10-08T20:53:59.0604981Z remote: Compressing objects: 13% (71/540) 2024-10-08T20:53:59.0605883Z remote: Compressing objects: 14% (76/540) 2024-10-08T20:53:59.0606748Z remote: Compressing objects: 15% (81/540) 2024-10-08T20:53:59.0607660Z remote: Compressing objects: 16% (87/540) 2024-10-08T20:53:59.0608527Z remote: Compressing objects: 17% (92/540) 2024-10-08T20:53:59.0902495Z remote: Compressing objects: 18% (98/540) 2024-10-08T20:53:59.0903608Z remote: Compressing objects: 19% (103/540) 2024-10-08T20:53:59.0904519Z remote: Compressing objects: 20% (108/540) 2024-10-08T20:53:59.0905976Z remote: Compressing objects: 21% (114/540) 2024-10-08T20:53:59.0907037Z remote: Compressing objects: 22% (119/540) 2024-10-08T20:53:59.0907989Z remote: Compressing objects: 23% (125/540) 2024-10-08T20:53:59.0908888Z remote: Compressing objects: 24% (130/540) 2024-10-08T20:53:59.0909761Z remote: Compressing objects: 25% (135/540) 2024-10-08T20:53:59.0910617Z remote: Compressing objects: 26% (141/540) 2024-10-08T20:53:59.0911532Z remote: Compressing objects: 27% (146/540) 2024-10-08T20:53:59.0912411Z remote: Compressing objects: 28% (152/540) 2024-10-08T20:53:59.0913285Z remote: Compressing objects: 29% (157/540) 2024-10-08T20:53:59.0914160Z remote: Compressing objects: 30% (162/540) 2024-10-08T20:53:59.0915026Z remote: Compressing objects: 31% (168/540) 2024-10-08T20:53:59.0915892Z remote: Compressing objects: 32% (173/540) 2024-10-08T20:53:59.0916761Z remote: Compressing objects: 33% (179/540) 2024-10-08T20:53:59.0917649Z remote: Compressing objects: 34% (184/540) 2024-10-08T20:53:59.0918518Z remote: Compressing objects: 35% (189/540) 2024-10-08T20:53:59.0919376Z remote: Compressing objects: 36% (195/540) 2024-10-08T20:53:59.0920273Z remote: Compressing objects: 37% (200/540) 2024-10-08T20:53:59.0956585Z remote: Compressing objects: 38% (206/540) 2024-10-08T20:53:59.0957624Z remote: Compressing objects: 39% (211/540) 2024-10-08T20:53:59.0958590Z remote: Compressing objects: 40% (216/540) 2024-10-08T20:53:59.0959488Z remote: Compressing objects: 41% (222/540) 2024-10-08T20:53:59.0960395Z remote: Compressing objects: 42% (227/540) 2024-10-08T20:53:59.0961270Z remote: Compressing objects: 43% (233/540) 2024-10-08T20:53:59.0962150Z remote: Compressing objects: 44% (238/540) 2024-10-08T20:53:59.0963061Z remote: Compressing objects: 45% (243/540) 2024-10-08T20:53:59.0963967Z remote: Compressing objects: 46% (249/540) 2024-10-08T20:53:59.0964871Z remote: Compressing objects: 47% (254/540) 2024-10-08T20:53:59.0965733Z remote: Compressing objects: 48% (260/540) 2024-10-08T20:53:59.0966625Z remote: Compressing objects: 49% (265/540) 2024-10-08T20:53:59.0967529Z remote: Compressing objects: 50% (270/540) 2024-10-08T20:53:59.0968396Z remote: Compressing objects: 51% (276/540) 2024-10-08T20:53:59.0969494Z remote: Compressing objects: 52% (281/540) 2024-10-08T20:53:59.0970400Z remote: Compressing objects: 53% (287/540) 2024-10-08T20:53:59.0971302Z remote: Compressing objects: 54% (292/540) 2024-10-08T20:53:59.0972183Z remote: Compressing objects: 55% (297/540) 2024-10-08T20:53:59.0973089Z remote: Compressing objects: 56% (303/540) 2024-10-08T20:53:59.0973975Z remote: Compressing objects: 57% (308/540) 2024-10-08T20:53:59.0974866Z remote: Compressing objects: 58% (314/540) 2024-10-08T20:53:59.0996853Z remote: Compressing objects: 59% (319/540) 2024-10-08T20:53:59.0997942Z remote: Compressing objects: 60% (324/540) 2024-10-08T20:53:59.0998865Z remote: Compressing objects: 61% (330/540) 2024-10-08T20:53:59.0999767Z remote: Compressing objects: 62% (335/540) 2024-10-08T20:53:59.1000718Z remote: Compressing objects: 63% (341/540) 2024-10-08T20:53:59.1001624Z remote: Compressing objects: 64% (346/540) 2024-10-08T20:53:59.1002509Z remote: Compressing objects: 65% (351/540) 2024-10-08T20:53:59.1003403Z remote: Compressing objects: 66% (357/540) 2024-10-08T20:53:59.1004300Z remote: Compressing objects: 67% (362/540) 2024-10-08T20:53:59.1005234Z remote: Compressing objects: 68% (368/540) 2024-10-08T20:53:59.1006172Z remote: Compressing objects: 69% (373/540) 2024-10-08T20:53:59.1007073Z remote: Compressing objects: 70% (378/540) 2024-10-08T20:53:59.1007958Z remote: Compressing objects: 71% (384/540) 2024-10-08T20:53:59.1009435Z remote: Compressing objects: 72% (389/540) 2024-10-08T20:53:59.1010378Z remote: Compressing objects: 73% (395/540) 2024-10-08T20:53:59.1011248Z remote: Compressing objects: 74% (400/540) 2024-10-08T20:53:59.1012102Z remote: Compressing objects: 75% (405/540) 2024-10-08T20:53:59.1012966Z remote: Compressing objects: 76% (411/540) 2024-10-08T20:53:59.1013814Z remote: Compressing objects: 77% (416/540) 2024-10-08T20:53:59.1014685Z remote: Compressing objects: 78% (422/540) 2024-10-08T20:53:59.1015579Z remote: Compressing objects: 79% (427/540) 2024-10-08T20:53:59.1019824Z remote: Compressing objects: 80% (432/540) 2024-10-08T20:53:59.1020928Z remote: Compressing objects: 81% (438/540) 2024-10-08T20:53:59.1021970Z remote: Compressing objects: 82% (443/540) 2024-10-08T20:53:59.1022921Z remote: Compressing objects: 83% (449/540) 2024-10-08T20:53:59.1023885Z remote: Compressing objects: 84% (454/540) 2024-10-08T20:53:59.1024789Z remote: Compressing objects: 85% (459/540) 2024-10-08T20:53:59.1025684Z remote: Compressing objects: 86% (465/540) 2024-10-08T20:53:59.1026583Z remote: Compressing objects: 87% (470/540) 2024-10-08T20:53:59.1028414Z remote: Compressing objects: 88% (476/540) 2024-10-08T20:53:59.1029324Z remote: Compressing objects: 89% (481/540) 2024-10-08T20:53:59.1030210Z remote: Compressing objects: 90% (486/540) 2024-10-08T20:53:59.1031103Z remote: Compressing objects: 91% (492/540) 2024-10-08T20:53:59.1031969Z remote: Compressing objects: 92% (497/540) 2024-10-08T20:53:59.1032849Z remote: Compressing objects: 93% (503/540) 2024-10-08T20:53:59.1033711Z remote: Compressing objects: 94% (508/540) 2024-10-08T20:53:59.1034622Z remote: Compressing objects: 95% (513/540) 2024-10-08T20:53:59.1035477Z remote: Compressing objects: 96% (519/540) 2024-10-08T20:53:59.1036374Z remote: Compressing objects: 97% (524/540) 2024-10-08T20:53:59.1037244Z remote: Compressing objects: 98% (530/540) 2024-10-08T20:53:59.1038081Z remote: Compressing objects: 99% (535/540) 2024-10-08T20:53:59.1038944Z remote: Compressing objects: 100% (540/540) 2024-10-08T20:53:59.1039898Z remote: Compressing objects: 100% (540/540), done. 2024-10-08T20:54:21.1446551Z remote: Total 1012099 (delta 637), reused 877 (delta 523), pack-reused 1011033 (from 1) 2024-10-08T20:54:30.4302896Z [command]/usr/bin/git rev-parse --verify --quiet 47952c0a8001fd60c9ffe52b121d022d440bef8b^{object} 2024-10-08T20:54:30.4344060Z 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T20:54:30.4350261Z ##[endgroup] 2024-10-08T20:54:30.4352403Z ##[group]Determining the checkout info 2024-10-08T20:54:30.4353459Z ##[endgroup] 2024-10-08T20:54:30.4354298Z ##[group]Checking out the ref 2024-10-08T20:54:30.4360574Z [command]/usr/bin/git checkout --quiet --force 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T20:54:31.8096130Z ##[endgroup] 2024-10-08T20:54:31.8097219Z ##[group]Setting up auth for fetching submodules 2024-10-08T20:54:31.8103457Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-10-08T20:54:31.8163701Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-10-08T20:54:31.8205859Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-10-08T20:54:31.8244458Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-10-08T20:54:31.8294441Z ##[endgroup] 2024-10-08T20:54:31.8295293Z ##[group]Fetching submodules 2024-10-08T20:54:31.8300969Z [command]/usr/bin/git submodule sync --recursive 2024-10-08T20:54:31.8584918Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-10-08T20:54:31.8851667Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-10-08T20:54:31.8859873Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-10-08T20:54:31.8863492Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-10-08T20:54:31.8867698Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-10-08T20:54:31.8871840Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-10-08T20:54:31.8876405Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-10-08T20:54:31.8880502Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-10-08T20:54:31.8885387Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-10-08T20:54:31.8894946Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-10-08T20:54:31.8899126Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-10-08T20:54:31.8903389Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-10-08T20:54:31.8906808Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-10-08T20:54:31.8910613Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-10-08T20:54:31.8914505Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-10-08T20:54:31.8918607Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-10-08T20:54:31.8925046Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-10-08T20:54:31.8929529Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:54:31.8933868Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-10-08T20:54:31.8938112Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-10-08T20:54:31.8942304Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-10-08T20:54:31.8946722Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-10-08T20:54:31.8951141Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-10-08T20:54:31.8955696Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-10-08T20:54:31.8960303Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-10-08T20:54:31.8964866Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-10-08T20:54:31.8969624Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-10-08T20:54:31.8974877Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-10-08T20:54:31.8979449Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-10-08T20:54:31.8984545Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-10-08T20:54:31.8989202Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-10-08T20:54:31.8994210Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-10-08T20:54:31.8999071Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-10-08T20:54:31.9004150Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-10-08T20:54:31.9009206Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-10-08T20:54:31.9014862Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-10-08T20:54:31.9082461Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-10-08T20:54:32.6148307Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-10-08T20:54:33.1393104Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-10-08T20:54:33.6901423Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-10-08T20:54:34.3599968Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-10-08T20:54:35.1166036Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-10-08T20:54:37.2838317Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-10-08T20:54:44.0067829Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-10-08T20:54:44.8363605Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-10-08T20:54:45.7216612Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-10-08T20:54:46.6979069Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-10-08T20:54:48.3963665Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-10-08T20:54:50.5132473Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-10-08T20:54:54.6144292Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-10-08T20:54:56.4256257Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-10-08T20:54:58.2507807Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-10-08T20:54:59.7022364Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-10-08T20:55:00.5297916Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-10-08T20:55:01.5716006Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-10-08T20:55:02.8422133Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-10-08T20:55:03.6240464Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-10-08T20:55:05.7018513Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-10-08T20:55:08.4009866Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-10-08T20:55:09.5107963Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-10-08T20:55:10.4546237Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-10-08T20:55:16.7362085Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-10-08T20:55:19.3779118Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-10-08T20:55:23.3755266Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-10-08T20:55:24.6175125Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-10-08T20:55:33.8879865Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-10-08T20:55:34.4578932Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-10-08T20:55:35.0703242Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-10-08T20:55:36.4919323Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-10-08T20:56:19.3110060Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-10-08T20:56:20.3834208Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-10-08T20:56:21.2113860Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-10-08T20:56:21.2383842Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-10-08T20:56:21.2622105Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-10-08T20:56:21.2987457Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-10-08T20:56:21.3427284Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-10-08T20:56:21.3941111Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-10-08T20:56:22.2100598Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-10-08T20:56:22.2529805Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-10-08T20:56:22.3157655Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-10-08T20:56:22.4273349Z Submodule path 'third_party/cpuinfo': checked out 'a5ff6df40ce528721cfc310c7ed43946d77404d5' 2024-10-08T20:56:22.4785108Z Submodule path 'third_party/cudnn_frontend': checked out '2533f5e5c1877fd76266133c1479ef1643ce3a8b' 2024-10-08T20:56:22.9885214Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-10-08T20:56:23.2407818Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-10-08T20:56:23.3395110Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-10-08T20:56:23.3474138Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:56:23.3477435Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:56:23.3481541Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:56:23.3484746Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:56:23.3489182Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:56:23.3549129Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-10-08T20:56:24.7855289Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-10-08T20:56:25.7808623Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-10-08T20:56:27.6585954Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-10-08T20:56:28.8429951Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-10-08T20:56:31.8617142Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-10-08T20:56:31.9761189Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-10-08T20:56:32.3974003Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-10-08T20:56:32.4725669Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-10-08T20:56:32.5029391Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-10-08T20:56:32.6414729Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-10-08T20:56:32.6945516Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-10-08T20:56:32.7511793Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-10-08T20:56:32.7901288Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-10-08T20:56:32.8503706Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-10-08T20:56:32.8795416Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-10-08T20:56:32.8853991Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-10-08T20:56:32.8898090Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-10-08T20:56:41.7856674Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-10-08T20:56:41.8302215Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-10-08T20:56:41.9342211Z Submodule path 'third_party/kineto': checked out 'b5c85daac1ee123aa7f04eb6f2bc71363f429e68' 2024-10-08T20:56:41.9542135Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:56:41.9545691Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:56:41.9548768Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:56:41.9551615Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-10-08T20:56:42.7813424Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-10-08T20:56:44.2536774Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-10-08T20:56:45.5686447Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-10-08T20:56:45.5759901Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:56:45.5767670Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:56:45.5776440Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:56:45.5783517Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:56:45.5787397Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:56:45.5791008Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:56:45.5794767Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:56:45.5798572Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:56:45.5842420Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-10-08T20:56:46.7505951Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-10-08T20:56:47.6354598Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-10-08T20:56:49.0787636Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-10-08T20:56:49.8900276Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-10-08T20:56:50.9256144Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-10-08T20:56:52.2148621Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-10-08T20:56:58.3213099Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-10-08T20:56:59.1076819Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-10-08T20:56:59.1415362Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-10-08T20:56:59.1937094Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-10-08T20:56:59.2245821Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-10-08T20:56:59.2299589Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:56:59.2346788Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-10-08T20:57:00.0177535Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-10-08T20:57:00.0522073Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-10-08T20:57:00.1050344Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-10-08T20:57:00.2224429Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-10-08T20:57:00.2553732Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-10-08T20:57:00.3072505Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-10-08T20:57:00.3778856Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-10-08T20:57:00.4320754Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-10-08T20:57:00.4739372Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-10-08T20:57:00.5931365Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-10-08T20:57:01.0057941Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-10-08T20:57:01.0158478Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-10-08T20:57:01.0161017Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-10-08T20:57:01.0227530Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-10-08T20:57:01.9590705Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-10-08T20:57:03.4564001Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-10-08T20:57:03.5058690Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-10-08T20:57:03.5889783Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-10-08T20:57:03.5955712Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:57:03.5962636Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:57:03.5965964Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:57:03.5969392Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:57:03.5976035Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:57:03.5980410Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:57:03.5983786Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:57:03.5987406Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:57:03.6030408Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-10-08T20:57:04.8049525Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-10-08T20:57:06.4575748Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-10-08T20:57:07.2123698Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-10-08T20:57:13.9778064Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-10-08T20:57:14.6659066Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-10-08T20:57:15.2880112Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-10-08T20:58:19.0186451Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-10-08T20:58:29.8545203Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-10-08T20:58:29.9094510Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-10-08T20:58:29.9368870Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-10-08T20:58:30.0527113Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-10-08T20:58:30.0865607Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-10-08T20:58:30.1167899Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-10-08T20:58:30.1476991Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-10-08T20:58:30.1522094Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:30.1525333Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:30.1581476Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-10-08T20:58:32.2407111Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-10-08T20:58:33.6132641Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-10-08T20:58:33.6753897Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-10-08T20:58:34.1708718Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-10-08T20:58:34.2042898Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-10-08T20:58:34.4838728Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-10-08T20:58:34.4918900Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:34.4921895Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:34.4999011Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-10-08T20:58:35.3392893Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-10-08T20:58:45.1267251Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-10-08T20:58:45.2045626Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-10-08T20:58:45.2344401Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-10-08T20:58:45.2638907Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-10-08T20:58:45.3119753Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-10-08T20:58:45.3580604Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-10-08T20:58:45.4224849Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-10-08T20:58:45.4626869Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-10-08T20:58:45.4680848Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:45.4683698Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:45.4687417Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:45.4691601Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:45.4737581Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-10-08T20:58:46.6682067Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-10-08T20:58:47.3269732Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-10-08T20:58:49.0102625Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-10-08T20:58:50.8871259Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-10-08T20:58:50.9215623Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-10-08T20:58:51.0040794Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-10-08T20:58:51.0475980Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-10-08T20:58:51.0545688Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:51.0589258Z Cloning into '/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-10-08T20:58:51.6414861Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-10-08T20:58:51.6523972Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-10-08T20:58:51.6786644Z Entering 'android/libs/fbjni' 2024-10-08T20:58:51.6825325Z Entering 'third_party/FP16' 2024-10-08T20:58:51.6862404Z Entering 'third_party/FXdiv' 2024-10-08T20:58:51.6898292Z Entering 'third_party/NNPACK' 2024-10-08T20:58:51.6936171Z Entering 'third_party/NVTX' 2024-10-08T20:58:51.6975468Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T20:58:51.7009271Z Entering 'third_party/XNNPACK' 2024-10-08T20:58:51.7066045Z Entering 'third_party/benchmark' 2024-10-08T20:58:51.7101933Z Entering 'third_party/cpp-httplib' 2024-10-08T20:58:51.7136900Z Entering 'third_party/cpuinfo' 2024-10-08T20:58:51.7176605Z Entering 'third_party/cudnn_frontend' 2024-10-08T20:58:51.7212481Z Entering 'third_party/cutlass' 2024-10-08T20:58:51.7268562Z Entering 'third_party/eigen' 2024-10-08T20:58:51.7324116Z Entering 'third_party/fbgemm' 2024-10-08T20:58:51.7363814Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:58:51.7400226Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:58:51.7439818Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:58:51.7500597Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:58:51.7537850Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:58:51.7577375Z Entering 'third_party/flatbuffers' 2024-10-08T20:58:51.7620684Z Entering 'third_party/fmt' 2024-10-08T20:58:51.7662907Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:58:51.7704045Z Entering 'third_party/gloo' 2024-10-08T20:58:51.7745825Z Entering 'third_party/googletest' 2024-10-08T20:58:51.7785651Z Entering 'third_party/ideep' 2024-10-08T20:58:51.7825233Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T20:58:51.7871779Z Entering 'third_party/ittapi' 2024-10-08T20:58:51.7909406Z Entering 'third_party/kineto' 2024-10-08T20:58:51.7945886Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:58:51.7995255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:58:51.8038741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:58:51.8076967Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:58:51.8112755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:58:51.8152765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:58:51.8194784Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:58:51.8234782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:58:51.8272491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:58:51.8310085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:58:51.8345580Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:58:51.8381633Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:58:51.8419533Z Entering 'third_party/mimalloc' 2024-10-08T20:58:51.8460681Z Entering 'third_party/nccl/nccl' 2024-10-08T20:58:51.8502107Z Entering 'third_party/nlohmann' 2024-10-08T20:58:51.8548840Z Entering 'third_party/onnx' 2024-10-08T20:58:51.8602160Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T20:58:51.8648949Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T20:58:51.8693472Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T20:58:51.8730486Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:58:51.8774325Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:58:51.8807235Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:58:51.8847671Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:58:51.8883619Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:58:51.8919560Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:58:51.8955843Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:58:51.8999351Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:51.9039801Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:51.9081624Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:58:51.9139160Z Entering 'third_party/pocketfft' 2024-10-08T20:58:51.9174808Z Entering 'third_party/protobuf' 2024-10-08T20:58:51.9211714Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:51.9254684Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:51.9296778Z Entering 'third_party/psimd' 2024-10-08T20:58:51.9338808Z Entering 'third_party/pthreadpool' 2024-10-08T20:58:51.9374500Z Entering 'third_party/pybind11' 2024-10-08T20:58:51.9412666Z Entering 'third_party/python-peachpy' 2024-10-08T20:58:51.9453451Z Entering 'third_party/sleef' 2024-10-08T20:58:51.9495285Z Entering 'third_party/tensorpipe' 2024-10-08T20:58:51.9535447Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:51.9573100Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:51.9611785Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:51.9646893Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:51.9678291Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:51.9733967Z ##[endgroup] 2024-10-08T20:58:51.9735773Z ##[group]Persisting credentials for submodules 2024-10-08T20:58:51.9739288Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-10-08T20:58:52.0023784Z Entering 'android/libs/fbjni' 2024-10-08T20:58:52.0066527Z Entering 'third_party/FP16' 2024-10-08T20:58:52.0105239Z Entering 'third_party/FXdiv' 2024-10-08T20:58:52.0143475Z Entering 'third_party/NNPACK' 2024-10-08T20:58:52.0182355Z Entering 'third_party/NVTX' 2024-10-08T20:58:52.0221390Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T20:58:52.0259757Z Entering 'third_party/XNNPACK' 2024-10-08T20:58:52.0312798Z Entering 'third_party/benchmark' 2024-10-08T20:58:52.0368947Z Entering 'third_party/cpp-httplib' 2024-10-08T20:58:52.0406844Z Entering 'third_party/cpuinfo' 2024-10-08T20:58:52.0461715Z Entering 'third_party/cudnn_frontend' 2024-10-08T20:58:52.0503945Z Entering 'third_party/cutlass' 2024-10-08T20:58:52.0547656Z Entering 'third_party/eigen' 2024-10-08T20:58:52.0592906Z Entering 'third_party/fbgemm' 2024-10-08T20:58:52.0635741Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:58:52.0676649Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:58:52.0718367Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:58:52.0760471Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:58:52.0800303Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:58:52.0851737Z Entering 'third_party/flatbuffers' 2024-10-08T20:58:52.0903158Z Entering 'third_party/fmt' 2024-10-08T20:58:52.0949232Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:58:52.0994381Z Entering 'third_party/gloo' 2024-10-08T20:58:52.1033931Z Entering 'third_party/googletest' 2024-10-08T20:58:52.1075646Z Entering 'third_party/ideep' 2024-10-08T20:58:52.1121011Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T20:58:52.1178052Z Entering 'third_party/ittapi' 2024-10-08T20:58:52.1222138Z Entering 'third_party/kineto' 2024-10-08T20:58:52.1267922Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:58:52.1314346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:58:52.1356363Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:58:52.1394557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:58:52.1433802Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:58:52.1474847Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:58:52.1518834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:58:52.1560818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:58:52.1601041Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:58:52.1642348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:58:52.1690986Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:58:52.1738780Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:58:52.1783588Z Entering 'third_party/mimalloc' 2024-10-08T20:58:52.1823609Z Entering 'third_party/nccl/nccl' 2024-10-08T20:58:52.1863419Z Entering 'third_party/nlohmann' 2024-10-08T20:58:52.1902832Z Entering 'third_party/onnx' 2024-10-08T20:58:52.1954511Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T20:58:52.1996360Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T20:58:52.2037627Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T20:58:52.2076743Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:58:52.2115128Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:58:52.2156448Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:58:52.2193661Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:58:52.2232624Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:58:52.2271814Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:58:52.2315054Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:58:52.2353163Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:52.2396160Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:52.2438803Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:58:52.2506385Z Entering 'third_party/pocketfft' 2024-10-08T20:58:52.2555760Z Entering 'third_party/protobuf' 2024-10-08T20:58:52.2596367Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:52.2636276Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:52.2677658Z Entering 'third_party/psimd' 2024-10-08T20:58:52.2716212Z Entering 'third_party/pthreadpool' 2024-10-08T20:58:52.2757008Z Entering 'third_party/pybind11' 2024-10-08T20:58:52.2797107Z Entering 'third_party/python-peachpy' 2024-10-08T20:58:52.2836278Z Entering 'third_party/sleef' 2024-10-08T20:58:52.2875361Z Entering 'third_party/tensorpipe' 2024-10-08T20:58:52.2914116Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:52.2956136Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:52.2993694Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:52.3032576Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:52.3078513Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:52.3147459Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-10-08T20:58:52.3434593Z Entering 'android/libs/fbjni' 2024-10-08T20:58:52.3487391Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-10-08T20:58:52.3518231Z Entering 'third_party/FP16' 2024-10-08T20:58:52.3553792Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-10-08T20:58:52.3574984Z Entering 'third_party/FXdiv' 2024-10-08T20:58:52.3617317Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-10-08T20:58:52.3634854Z Entering 'third_party/NNPACK' 2024-10-08T20:58:52.3671716Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-10-08T20:58:52.3692063Z Entering 'third_party/NVTX' 2024-10-08T20:58:52.3736039Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-10-08T20:58:52.3753157Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T20:58:52.3789324Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-10-08T20:58:52.3815026Z Entering 'third_party/XNNPACK' 2024-10-08T20:58:52.3854383Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-10-08T20:58:52.3888699Z Entering 'third_party/benchmark' 2024-10-08T20:58:52.3927347Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-10-08T20:58:52.3940078Z Entering 'third_party/cpp-httplib' 2024-10-08T20:58:52.3978240Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-10-08T20:58:52.3996332Z Entering 'third_party/cpuinfo' 2024-10-08T20:58:52.4030497Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-10-08T20:58:52.4048619Z Entering 'third_party/cudnn_frontend' 2024-10-08T20:58:52.4087316Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-10-08T20:58:52.4104848Z Entering 'third_party/cutlass' 2024-10-08T20:58:52.4149148Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-10-08T20:58:52.4178639Z Entering 'third_party/eigen' 2024-10-08T20:58:52.4214423Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-10-08T20:58:52.4232370Z Entering 'third_party/fbgemm' 2024-10-08T20:58:52.4270245Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-10-08T20:58:52.4288037Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:58:52.4328274Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-10-08T20:58:52.4346493Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:58:52.4380484Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-10-08T20:58:52.4396964Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:58:52.4429625Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-10-08T20:58:52.4452953Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:58:52.4495540Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.4513464Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:58:52.4548182Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-10-08T20:58:52.4573108Z Entering 'third_party/flatbuffers' 2024-10-08T20:58:52.4608135Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-10-08T20:58:52.4627402Z Entering 'third_party/fmt' 2024-10-08T20:58:52.4669135Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-10-08T20:58:52.4692095Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:58:52.4731225Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-10-08T20:58:52.4750132Z Entering 'third_party/gloo' 2024-10-08T20:58:52.4789465Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-10-08T20:58:52.4815638Z Entering 'third_party/googletest' 2024-10-08T20:58:52.4849802Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.4869034Z Entering 'third_party/ideep' 2024-10-08T20:58:52.4906312Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-10-08T20:58:52.4923618Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T20:58:52.4962269Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-10-08T20:58:52.4986753Z Entering 'third_party/ittapi' 2024-10-08T20:58:52.5024222Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-10-08T20:58:52.5042681Z Entering 'third_party/kineto' 2024-10-08T20:58:52.5088247Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-10-08T20:58:52.5109196Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:58:52.5148014Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-10-08T20:58:52.5165273Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:58:52.5201085Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-10-08T20:58:52.5220477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:58:52.5256732Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-10-08T20:58:52.5273827Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:58:52.5307600Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-10-08T20:58:52.5324560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:58:52.5357414Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-10-08T20:58:52.5377037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:58:52.5416535Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-10-08T20:58:52.5438060Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:58:52.5470782Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-10-08T20:58:52.5487964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:58:52.5522942Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.5539912Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:58:52.5579435Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-10-08T20:58:52.5595764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:58:52.5631563Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-10-08T20:58:52.5657513Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:58:52.5689117Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-10-08T20:58:52.5706995Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:58:52.5743613Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.5764727Z Entering 'third_party/mimalloc' 2024-10-08T20:58:52.5806295Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-10-08T20:58:52.5823252Z Entering 'third_party/nccl/nccl' 2024-10-08T20:58:52.5862186Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-10-08T20:58:52.5881924Z Entering 'third_party/nlohmann' 2024-10-08T20:58:52.5932568Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-10-08T20:58:52.5954049Z Entering 'third_party/onnx' 2024-10-08T20:58:52.5985255Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-10-08T20:58:52.6016307Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T20:58:52.6056815Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-10-08T20:58:52.6075597Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T20:58:52.6110394Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-10-08T20:58:52.6133545Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T20:58:52.6167539Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-10-08T20:58:52.6187197Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:58:52.6226627Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-10-08T20:58:52.6245614Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:58:52.6281110Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.6298604Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:58:52.6335521Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-10-08T20:58:52.6352502Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:58:52.6386082Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-10-08T20:58:52.6404426Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:58:52.6446259Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-10-08T20:58:52.6462661Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:58:52.6498988Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-10-08T20:58:52.6517167Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:58:52.6552994Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-10-08T20:58:52.6570772Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:52.6618768Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-10-08T20:58:52.6638891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:52.6672364Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-10-08T20:58:52.6691787Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:58:52.6727281Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-10-08T20:58:52.6769748Z Entering 'third_party/pocketfft' 2024-10-08T20:58:52.6816636Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-10-08T20:58:52.6836196Z Entering 'third_party/protobuf' 2024-10-08T20:58:52.6873098Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-10-08T20:58:52.6895162Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:52.6931178Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-10-08T20:58:52.6949987Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:52.6985972Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.7006600Z Entering 'third_party/psimd' 2024-10-08T20:58:52.7048038Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-10-08T20:58:52.7063846Z Entering 'third_party/pthreadpool' 2024-10-08T20:58:52.7102645Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-10-08T20:58:52.7120372Z Entering 'third_party/pybind11' 2024-10-08T20:58:52.7159332Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-10-08T20:58:52.7179646Z Entering 'third_party/python-peachpy' 2024-10-08T20:58:52.7217302Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-10-08T20:58:52.7236977Z Entering 'third_party/sleef' 2024-10-08T20:58:52.7272463Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-10-08T20:58:52.7290968Z Entering 'third_party/tensorpipe' 2024-10-08T20:58:52.7326651Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-10-08T20:58:52.7349168Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:52.7386801Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-10-08T20:58:52.7404161Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:52.7437455Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-10-08T20:58:52.7457927Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:52.7499750Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-10-08T20:58:52.7516840Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:52.7550523Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-10-08T20:58:52.7566493Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:52.7605306Z file:/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-10-08T20:58:52.7899880Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-10-08T20:58:52.8172798Z Entering 'android/libs/fbjni' 2024-10-08T20:58:52.8216616Z Entering 'third_party/FP16' 2024-10-08T20:58:52.8260716Z Entering 'third_party/FXdiv' 2024-10-08T20:58:52.8299753Z Entering 'third_party/NNPACK' 2024-10-08T20:58:52.8351410Z Entering 'third_party/NVTX' 2024-10-08T20:58:52.8395268Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T20:58:52.8429801Z Entering 'third_party/XNNPACK' 2024-10-08T20:58:52.8485879Z Entering 'third_party/benchmark' 2024-10-08T20:58:52.8525651Z Entering 'third_party/cpp-httplib' 2024-10-08T20:58:52.8562005Z Entering 'third_party/cpuinfo' 2024-10-08T20:58:52.8599160Z Entering 'third_party/cudnn_frontend' 2024-10-08T20:58:52.8637713Z Entering 'third_party/cutlass' 2024-10-08T20:58:52.8683269Z Entering 'third_party/eigen' 2024-10-08T20:58:52.8721943Z Entering 'third_party/fbgemm' 2024-10-08T20:58:52.8759184Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:58:52.8797040Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:58:52.8830642Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:58:52.8876654Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:58:52.8909292Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:58:52.8946313Z Entering 'third_party/flatbuffers' 2024-10-08T20:58:52.8988290Z Entering 'third_party/fmt' 2024-10-08T20:58:52.9027518Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:58:52.9061375Z Entering 'third_party/gloo' 2024-10-08T20:58:52.9095516Z Entering 'third_party/googletest' 2024-10-08T20:58:52.9137133Z Entering 'third_party/ideep' 2024-10-08T20:58:52.9169162Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T20:58:52.9217197Z Entering 'third_party/ittapi' 2024-10-08T20:58:52.9251952Z Entering 'third_party/kineto' 2024-10-08T20:58:52.9297336Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:58:52.9330114Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:58:52.9373160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:58:52.9412253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:58:52.9453492Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:58:52.9486302Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:58:52.9522604Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:58:52.9555743Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:58:52.9595631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:58:52.9635818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:58:52.9677392Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:58:52.9713467Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:58:52.9751961Z Entering 'third_party/mimalloc' 2024-10-08T20:58:52.9791917Z Entering 'third_party/nccl/nccl' 2024-10-08T20:58:52.9831803Z Entering 'third_party/nlohmann' 2024-10-08T20:58:52.9874283Z Entering 'third_party/onnx' 2024-10-08T20:58:52.9925813Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T20:58:52.9963019Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T20:58:53.0003240Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T20:58:53.0050734Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:58:53.0097684Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:58:53.0132530Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:58:53.0172215Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:58:53.0208199Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:58:53.0242979Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:58:53.0276378Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:58:53.0308588Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:53.0353657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:53.0394141Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:58:53.0450745Z Entering 'third_party/pocketfft' 2024-10-08T20:58:53.0492731Z Entering 'third_party/protobuf' 2024-10-08T20:58:53.0537441Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:53.0578059Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:53.0617576Z Entering 'third_party/psimd' 2024-10-08T20:58:53.0654341Z Entering 'third_party/pthreadpool' 2024-10-08T20:58:53.0688325Z Entering 'third_party/pybind11' 2024-10-08T20:58:53.0722418Z Entering 'third_party/python-peachpy' 2024-10-08T20:58:53.0757923Z Entering 'third_party/sleef' 2024-10-08T20:58:53.0796976Z Entering 'third_party/tensorpipe' 2024-10-08T20:58:53.0834780Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:53.0874288Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:53.0913336Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:53.0949531Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:53.0983994Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:53.1042568Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-10-08T20:58:53.1295516Z Entering 'android/libs/fbjni' 2024-10-08T20:58:53.1334374Z Entering 'third_party/FP16' 2024-10-08T20:58:53.1373394Z Entering 'third_party/FXdiv' 2024-10-08T20:58:53.1413675Z Entering 'third_party/NNPACK' 2024-10-08T20:58:53.1449439Z Entering 'third_party/NVTX' 2024-10-08T20:58:53.1484965Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T20:58:53.1519269Z Entering 'third_party/XNNPACK' 2024-10-08T20:58:53.1569431Z Entering 'third_party/benchmark' 2024-10-08T20:58:53.1607139Z Entering 'third_party/cpp-httplib' 2024-10-08T20:58:53.1642069Z Entering 'third_party/cpuinfo' 2024-10-08T20:58:53.1678384Z Entering 'third_party/cudnn_frontend' 2024-10-08T20:58:53.1715569Z Entering 'third_party/cutlass' 2024-10-08T20:58:53.1763242Z Entering 'third_party/eigen' 2024-10-08T20:58:53.1802384Z Entering 'third_party/fbgemm' 2024-10-08T20:58:53.1841799Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T20:58:53.1881586Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T20:58:53.1919157Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T20:58:53.1965023Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T20:58:53.2008470Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T20:58:53.2053183Z Entering 'third_party/flatbuffers' 2024-10-08T20:58:53.2095480Z Entering 'third_party/fmt' 2024-10-08T20:58:53.2134227Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T20:58:53.2171847Z Entering 'third_party/gloo' 2024-10-08T20:58:53.2206101Z Entering 'third_party/googletest' 2024-10-08T20:58:53.2241532Z Entering 'third_party/ideep' 2024-10-08T20:58:53.2289959Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T20:58:53.2340062Z Entering 'third_party/ittapi' 2024-10-08T20:58:53.2381138Z Entering 'third_party/kineto' 2024-10-08T20:58:53.2427961Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T20:58:53.2466997Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T20:58:53.2505356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T20:58:53.2543134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T20:58:53.2585514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T20:58:53.2622280Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T20:58:53.2667630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T20:58:53.2714549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T20:58:53.2750278Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T20:58:53.2788285Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T20:58:53.2839502Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T20:58:53.2865347Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T20:58:53.2901115Z Entering 'third_party/mimalloc' 2024-10-08T20:58:53.2941411Z Entering 'third_party/nccl/nccl' 2024-10-08T20:58:53.2983106Z Entering 'third_party/nlohmann' 2024-10-08T20:58:53.3026249Z Entering 'third_party/onnx' 2024-10-08T20:58:53.3072291Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T20:58:53.3115717Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T20:58:53.3157934Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T20:58:53.3195338Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T20:58:53.3228905Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T20:58:53.3268004Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T20:58:53.3303469Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T20:58:53.3342891Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T20:58:53.3375724Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T20:58:53.3408000Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T20:58:53.3442072Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T20:58:53.3479660Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T20:58:53.3516299Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T20:58:53.3568349Z Entering 'third_party/pocketfft' 2024-10-08T20:58:53.3604452Z Entering 'third_party/protobuf' 2024-10-08T20:58:53.3653201Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T20:58:53.3690292Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T20:58:53.3727119Z Entering 'third_party/psimd' 2024-10-08T20:58:53.3761837Z Entering 'third_party/pthreadpool' 2024-10-08T20:58:53.3799543Z Entering 'third_party/pybind11' 2024-10-08T20:58:53.3842885Z Entering 'third_party/python-peachpy' 2024-10-08T20:58:53.3876539Z Entering 'third_party/sleef' 2024-10-08T20:58:53.3913940Z Entering 'third_party/tensorpipe' 2024-10-08T20:58:53.3953473Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T20:58:53.3994986Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T20:58:53.4028739Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T20:58:53.4064091Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T20:58:53.4096894Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T20:58:53.4157970Z ##[endgroup] 2024-10-08T20:58:53.4192038Z [command]/usr/bin/git log -1 --format='%H' 2024-10-08T20:58:53.4211375Z '47952c0a8001fd60c9ffe52b121d022d440bef8b' 2024-10-08T20:58:53.4470317Z Prepare all required actions 2024-10-08T20:58:53.4470752Z Getting action download info 2024-10-08T20:58:53.4602238Z ##[group]Run ./.github/actions/setup-rocm 2024-10-08T20:58:53.4602934Z env: 2024-10-08T20:58:53.4603375Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.4603958Z ##[endgroup] 2024-10-08T20:58:53.4649634Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-10-08T20:58:53.4651159Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-10-08T20:58:53.4693603Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.4694381Z env: 2024-10-08T20:58:53.4694805Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.4695320Z ##[endgroup] 2024-10-08T20:58:53.4812655Z ##[group]Run set -ex 2024-10-08T20:58:53.4813102Z set -ex 2024-10-08T20:58:53.4813513Z  2024-10-08T20:58:53.4814014Z cat ~/.docker/config.json || true 2024-10-08T20:58:53.4815766Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-10-08T20:58:53.4817425Z rm -f ~/.docker/config.json 2024-10-08T20:58:53.4857332Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.4858013Z env: 2024-10-08T20:58:53.4858377Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.4858911Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.4859518Z ##[endgroup] 2024-10-08T20:58:53.4914605Z + cat /home/pytorchci/.docker/config.json 2024-10-08T20:58:53.4925492Z { 2024-10-08T20:58:53.4925841Z "auths": {} 2024-10-08T20:58:53.4927888Z + rm -f /home/pytorchci/.docker/config.json 2024-10-08T20:58:53.4941878Z } 2024-10-08T20:58:53.4970090Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T20:58:53.4971153Z # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T20:58:53.4971944Z # shellcheck disable=SC2046 2024-10-08T20:58:53.4972569Z docker stop $(docker ps -q) || true 2024-10-08T20:58:53.4973512Z # Prune all stopped containers. 2024-10-08T20:58:53.4974099Z docker container prune -f 2024-10-08T20:58:53.5022263Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.5023284Z env: 2024-10-08T20:58:53.5023806Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.5024632Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.5025518Z ##[endgroup] 2024-10-08T20:58:53.5380308Z "docker stop" requires at least 1 argument. 2024-10-08T20:58:53.5380831Z See 'docker stop --help'. 2024-10-08T20:58:53.5381053Z 2024-10-08T20:58:53.5381262Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-10-08T20:58:53.5381590Z 2024-10-08T20:58:53.5381722Z Stop one or more running containers 2024-10-08T20:58:53.5519585Z Total reclaimed space: 0B 2024-10-08T20:58:53.5563716Z ##[group]Run cat /etc/os-release || true 2024-10-08T20:58:53.5564361Z cat /etc/os-release || true 2024-10-08T20:58:53.5565026Z cat /etc/apt/sources.list.d/rocm.list || true 2024-10-08T20:58:53.5565753Z cat /opt/rocm/.info/version || true 2024-10-08T20:58:53.5566326Z whoami 2024-10-08T20:58:53.5599280Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.5599960Z env: 2024-10-08T20:58:53.5600320Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.5600863Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.5601453Z ##[endgroup] 2024-10-08T20:58:53.5666609Z PRETTY_NAME="Ubuntu 22.04.3 LTS" 2024-10-08T20:58:53.5666949Z NAME="Ubuntu" 2024-10-08T20:58:53.5667216Z VERSION_ID="22.04" 2024-10-08T20:58:53.5667516Z VERSION="22.04.3 LTS (Jammy Jellyfish)" 2024-10-08T20:58:53.5667878Z VERSION_CODENAME=jammy 2024-10-08T20:58:53.5668159Z ID=ubuntu 2024-10-08T20:58:53.5668394Z ID_LIKE=debian 2024-10-08T20:58:53.5668674Z HOME_URL="https://www.ubuntu.com/" 2024-10-08T20:58:53.5669061Z SUPPORT_URL="https://help.ubuntu.com/" 2024-10-08T20:58:53.5669783Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-10-08T20:58:53.5674103Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-10-08T20:58:53.5674655Z UBUNTU_CODENAME=jammy 2024-10-08T20:58:53.5675026Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.2.1 jammy main 2024-10-08T20:58:53.5682599Z 6.2.1-112 2024-10-08T20:58:53.5693203Z pytorchci 2024-10-08T20:58:53.5722381Z ##[group]Run rocm-smi 2024-10-08T20:58:53.5722911Z rocm-smi 2024-10-08T20:58:53.5761023Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.5761833Z env: 2024-10-08T20:58:53.5762250Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.5762907Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.5763622Z ##[endgroup] 2024-10-08T20:58:53.6589746Z 2024-10-08T20:58:53.6589879Z 2024-10-08T20:58:53.6590320Z ========================================= ROCm System Management Interface ========================================= 2024-10-08T20:58:53.6590957Z =================================================== Concise Info =================================================== 2024-10-08T20:58:53.6591614Z Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-10-08T20:58:53.6593279Z  (DID, GUID) (Edge) (Avg) (Mem, Compute, ID)  2024-10-08T20:58:53.6594368Z ==================================================================================================================== 2024-10-08T20:58:53.6596352Z 0 2 0x740f, 12261 48.0°C 40.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-10-08T20:58:53.6597945Z 1 3 0x740f, 36740 47.0°C 41.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 300.0W 0% 0% 2024-10-08T20:58:53.6599494Z ==================================================================================================================== 2024-10-08T20:58:53.6600740Z =============================================== End of ROCm SMI Log ================================================ 2024-10-08T20:58:53.6736762Z ##[group]Run rocminfo 2024-10-08T20:58:53.6737318Z rocminfo 2024-10-08T20:58:53.6781691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.6782737Z env: 2024-10-08T20:58:53.6783276Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.6784124Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.6785023Z ##[endgroup] 2024-10-08T20:58:53.7589590Z ROCk module version 6.8.5 is loaded 2024-10-08T20:58:53.7590066Z ===================== 2024-10-08T20:58:53.7590370Z HSA System Attributes 2024-10-08T20:58:53.7590671Z ===================== 2024-10-08T20:58:53.7591014Z Runtime Version: 1.14 2024-10-08T20:58:53.7591405Z Runtime Ext Version: 1.6 2024-10-08T20:58:53.7591866Z System Timestamp Freq.: 1000.000000MHz 2024-10-08T20:58:53.7593237Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-10-08T20:58:53.7594535Z Machine Model: LARGE 2024-10-08T20:58:53.7597708Z System Endianness: LITTLE 2024-10-08T20:58:53.7598245Z Mwaitx: DISABLED 2024-10-08T20:58:53.7598607Z DMAbuf Support: YES 2024-10-08T20:58:53.7598853Z 2024-10-08T20:58:53.7598955Z ========== 2024-10-08T20:58:53.7599256Z HSA Agents 2024-10-08T20:58:53.7599532Z ========== 2024-10-08T20:58:53.7599810Z ******* 2024-10-08T20:58:53.7600093Z Agent 1 2024-10-08T20:58:53.7600364Z ******* 2024-10-08T20:58:53.7602501Z Name: AMD EPYC 7513 32-Core Processor 2024-10-08T20:58:53.7603116Z Uuid: CPU-XX 2024-10-08T20:58:53.7603753Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T20:58:53.7606619Z Vendor Name: CPU 2024-10-08T20:58:53.7607144Z Feature: None specified 2024-10-08T20:58:53.7607662Z Profile: FULL_PROFILE 2024-10-08T20:58:53.7608171Z Float Round Mode: NEAR 2024-10-08T20:58:53.7608699Z Max Queue Number: 0(0x0) 2024-10-08T20:58:53.7609319Z Queue Min Size: 0(0x0) 2024-10-08T20:58:53.7609814Z Queue Max Size: 0(0x0) 2024-10-08T20:58:53.7610290Z Queue Type: MULTI 2024-10-08T20:58:53.7610748Z Node: 0 2024-10-08T20:58:53.7611182Z Device Type: CPU 2024-10-08T20:58:53.7611518Z Cache Info: 2024-10-08T20:58:53.7611834Z L1: 32768(0x8000) KB 2024-10-08T20:58:53.7612232Z Chip ID: 0(0x0) 2024-10-08T20:58:53.7612622Z ASIC Revision: 0(0x0) 2024-10-08T20:58:53.7613011Z Cacheline Size: 64(0x40) 2024-10-08T20:58:53.7613415Z Max Clock Freq. (MHz): 2600 2024-10-08T20:58:53.7613795Z BDFID: 0 2024-10-08T20:58:53.7614163Z Internal Node ID: 0 2024-10-08T20:58:53.7614561Z Compute Unit: 32 2024-10-08T20:58:53.7614949Z SIMDs per CU: 0 2024-10-08T20:58:53.7615330Z Shader Engines: 0 2024-10-08T20:58:53.7615733Z Shader Arrs. per Eng.: 0 2024-10-08T20:58:53.7616145Z WatchPts on Addr. Ranges:1 2024-10-08T20:58:53.7616501Z Memory Properties: 2024-10-08T20:58:53.7616768Z Features: None 2024-10-08T20:58:53.7617242Z Pool Info: 2024-10-08T20:58:53.7617488Z Pool 1 2024-10-08T20:58:53.7617820Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T20:58:53.7618234Z Size: 65839400(0x3eca128) KB 2024-10-08T20:58:53.7618635Z Allocatable: TRUE 2024-10-08T20:58:53.7619058Z Alloc Granule: 4KB 2024-10-08T20:58:53.7619498Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7619934Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7620349Z Accessible by all: TRUE 2024-10-08T20:58:53.7620704Z Pool 2 2024-10-08T20:58:53.7621041Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-10-08T20:58:53.7621443Z Size: 65839400(0x3eca128) KB 2024-10-08T20:58:53.7621837Z Allocatable: TRUE 2024-10-08T20:58:53.7622243Z Alloc Granule: 4KB 2024-10-08T20:58:53.7622662Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7623102Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7623523Z Accessible by all: TRUE 2024-10-08T20:58:53.7623870Z Pool 3 2024-10-08T20:58:53.7624186Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T20:58:53.7624577Z Size: 65839400(0x3eca128) KB 2024-10-08T20:58:53.7624956Z Allocatable: TRUE 2024-10-08T20:58:53.7625492Z Alloc Granule: 4KB 2024-10-08T20:58:53.7625920Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7626354Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7626765Z Accessible by all: TRUE 2024-10-08T20:58:53.7627107Z ISA Info: 2024-10-08T20:58:53.7627347Z ******* 2024-10-08T20:58:53.7627567Z Agent 2 2024-10-08T20:58:53.7627795Z ******* 2024-10-08T20:58:53.7628151Z Name: AMD EPYC 7513 32-Core Processor 2024-10-08T20:58:53.7628567Z Uuid: CPU-XX 2024-10-08T20:58:53.7629018Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T20:58:53.7629431Z Vendor Name: CPU 2024-10-08T20:58:53.7629807Z Feature: None specified 2024-10-08T20:58:53.7630211Z Profile: FULL_PROFILE 2024-10-08T20:58:53.7630622Z Float Round Mode: NEAR 2024-10-08T20:58:53.7631055Z Max Queue Number: 0(0x0) 2024-10-08T20:58:53.7631541Z Queue Min Size: 0(0x0) 2024-10-08T20:58:53.7632041Z Queue Max Size: 0(0x0) 2024-10-08T20:58:53.7632502Z Queue Type: MULTI 2024-10-08T20:58:53.7632943Z Node: 1 2024-10-08T20:58:53.7633359Z Device Type: CPU 2024-10-08T20:58:53.7633700Z Cache Info: 2024-10-08T20:58:53.7634099Z L1: 32768(0x8000) KB 2024-10-08T20:58:53.7634472Z Chip ID: 0(0x0) 2024-10-08T20:58:53.7634861Z ASIC Revision: 0(0x0) 2024-10-08T20:58:53.7635251Z Cacheline Size: 64(0x40) 2024-10-08T20:58:53.7635768Z Max Clock Freq. (MHz): 2600 2024-10-08T20:58:53.7636152Z BDFID: 0 2024-10-08T20:58:53.7636542Z Internal Node ID: 1 2024-10-08T20:58:53.7636934Z Compute Unit: 32 2024-10-08T20:58:53.7637314Z SIMDs per CU: 0 2024-10-08T20:58:53.7637711Z Shader Engines: 0 2024-10-08T20:58:53.7638121Z Shader Arrs. per Eng.: 0 2024-10-08T20:58:53.7638532Z WatchPts on Addr. Ranges:1 2024-10-08T20:58:53.7638889Z Memory Properties: 2024-10-08T20:58:53.7639142Z Features: None 2024-10-08T20:58:53.7639397Z Pool Info: 2024-10-08T20:58:53.7639646Z Pool 1 2024-10-08T20:58:53.7639980Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T20:58:53.7640386Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T20:58:53.7640789Z Allocatable: TRUE 2024-10-08T20:58:53.7641198Z Alloc Granule: 4KB 2024-10-08T20:58:53.7641620Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7642052Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7642475Z Accessible by all: TRUE 2024-10-08T20:58:53.7642816Z Pool 2 2024-10-08T20:58:53.7643146Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-10-08T20:58:53.7643552Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T20:58:53.7644049Z Allocatable: TRUE 2024-10-08T20:58:53.7644458Z Alloc Granule: 4KB 2024-10-08T20:58:53.7644886Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7645310Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7645716Z Accessible by all: TRUE 2024-10-08T20:58:53.7646054Z Pool 3 2024-10-08T20:58:53.7646371Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T20:58:53.7646751Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T20:58:53.7647135Z Allocatable: TRUE 2024-10-08T20:58:53.7647538Z Alloc Granule: 4KB 2024-10-08T20:58:53.7647949Z Alloc Recommended Granule:4KB 2024-10-08T20:58:53.7648382Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7648841Z Accessible by all: TRUE 2024-10-08T20:58:53.7649180Z ISA Info: 2024-10-08T20:58:53.7649422Z ******* 2024-10-08T20:58:53.7649649Z Agent 3 2024-10-08T20:58:53.7649869Z ******* 2024-10-08T20:58:53.7650143Z Name: gfx90a 2024-10-08T20:58:53.7650581Z Uuid: GPU-acdd3d2233dd93e8 2024-10-08T20:58:53.7650987Z Marketing Name: AMD Instinct MI210 2024-10-08T20:58:53.7651379Z Vendor Name: AMD 2024-10-08T20:58:53.7651776Z Feature: KERNEL_DISPATCH 2024-10-08T20:58:53.7652171Z Profile: BASE_PROFILE 2024-10-08T20:58:53.7652555Z Float Round Mode: NEAR 2024-10-08T20:58:53.7652969Z Max Queue Number: 128(0x80) 2024-10-08T20:58:53.7653527Z Queue Min Size: 64(0x40) 2024-10-08T20:58:53.7653917Z Queue Max Size: 131072(0x20000) 2024-10-08T20:58:53.7654312Z Queue Type: MULTI 2024-10-08T20:58:53.7654682Z Node: 2 2024-10-08T20:58:53.7655045Z Device Type: GPU 2024-10-08T20:58:53.7655375Z Cache Info: 2024-10-08T20:58:53.7655674Z L1: 16(0x10) KB 2024-10-08T20:58:53.7656032Z L2: 8192(0x2000) KB 2024-10-08T20:58:53.7656408Z Chip ID: 29711(0x740f) 2024-10-08T20:58:53.7656789Z ASIC Revision: 1(0x1) 2024-10-08T20:58:53.7657186Z Cacheline Size: 64(0x40) 2024-10-08T20:58:53.7657578Z Max Clock Freq. (MHz): 1700 2024-10-08T20:58:53.7657967Z BDFID: 768 2024-10-08T20:58:53.7658348Z Internal Node ID: 2 2024-10-08T20:58:53.7658732Z Compute Unit: 104 2024-10-08T20:58:53.7659110Z SIMDs per CU: 4 2024-10-08T20:58:53.7659510Z Shader Engines: 8 2024-10-08T20:58:53.7659908Z Shader Arrs. per Eng.: 1 2024-10-08T20:58:53.7660331Z WatchPts on Addr. Ranges:4 2024-10-08T20:58:53.7660755Z Coherent Host Access: FALSE 2024-10-08T20:58:53.7661106Z Memory Properties: 2024-10-08T20:58:53.7661390Z Features: KERNEL_DISPATCH 2024-10-08T20:58:53.7661920Z Fast F16 Operation: TRUE 2024-10-08T20:58:53.7662327Z Wavefront Size: 64(0x40) 2024-10-08T20:58:53.7662745Z Workgroup Max Size: 1024(0x400) 2024-10-08T20:58:53.7663111Z Workgroup Max Size per Dimension: 2024-10-08T20:58:53.7663429Z x 1024(0x400) 2024-10-08T20:58:53.7663787Z y 1024(0x400) 2024-10-08T20:58:53.7664132Z z 1024(0x400) 2024-10-08T20:58:53.7664519Z Max Waves Per CU: 32(0x20) 2024-10-08T20:58:53.7664978Z Max Work-item Per CU: 2048(0x800) 2024-10-08T20:58:53.7665393Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T20:58:53.7665759Z Grid Max Size per Dimension: 2024-10-08T20:58:53.7666056Z x 4294967295(0xffffffff) 2024-10-08T20:58:53.7666425Z y 4294967295(0xffffffff) 2024-10-08T20:58:53.7666785Z z 4294967295(0xffffffff) 2024-10-08T20:58:53.7667174Z Max fbarriers/Workgrp: 32 2024-10-08T20:58:53.7677804Z Packet Processor uCode:: 83 2024-10-08T20:58:53.7678285Z SDMA engine uCode:: 8 2024-10-08T20:58:53.7678717Z IOMMU Support:: None 2024-10-08T20:58:53.7679054Z Pool Info: 2024-10-08T20:58:53.7679302Z Pool 1 2024-10-08T20:58:53.7679633Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T20:58:53.7680040Z Size: 67092480(0x3ffc000) KB 2024-10-08T20:58:53.7680445Z Allocatable: TRUE 2024-10-08T20:58:53.7680875Z Alloc Granule: 4KB 2024-10-08T20:58:53.7681308Z Alloc Recommended Granule:2048KB 2024-10-08T20:58:53.7681913Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7682344Z Accessible by all: FALSE 2024-10-08T20:58:53.7682685Z Pool 2 2024-10-08T20:58:53.7683009Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-10-08T20:58:53.7683425Z Size: 67092480(0x3ffc000) KB 2024-10-08T20:58:53.7683825Z Allocatable: TRUE 2024-10-08T20:58:53.7684242Z Alloc Granule: 4KB 2024-10-08T20:58:53.7684675Z Alloc Recommended Granule:2048KB 2024-10-08T20:58:53.7685103Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7685535Z Accessible by all: FALSE 2024-10-08T20:58:53.7685889Z Pool 3 2024-10-08T20:58:53.7686206Z Segment: GROUP 2024-10-08T20:58:53.7686583Z Size: 64(0x40) KB 2024-10-08T20:58:53.7686964Z Allocatable: FALSE 2024-10-08T20:58:53.7687371Z Alloc Granule: 0KB 2024-10-08T20:58:53.7687785Z Alloc Recommended Granule:0KB 2024-10-08T20:58:53.7688217Z Alloc Alignment: 0KB 2024-10-08T20:58:53.7688626Z Accessible by all: FALSE 2024-10-08T20:58:53.7689026Z ISA Info: 2024-10-08T20:58:53.7689271Z ISA 1 2024-10-08T20:58:53.7689691Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-10-08T20:58:53.7690259Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-10-08T20:58:53.7690687Z Profiles: HSA_PROFILE_BASE 2024-10-08T20:58:53.7691107Z Default Rounding Mode: NEAR 2024-10-08T20:58:53.7691533Z Default Rounding Mode: NEAR 2024-10-08T20:58:53.7691938Z Fast f16: TRUE 2024-10-08T20:58:53.7692345Z Workgroup Max Size: 1024(0x400) 2024-10-08T20:58:53.7692716Z Workgroup Max Size per Dimension: 2024-10-08T20:58:53.7693047Z x 1024(0x400) 2024-10-08T20:58:53.7693407Z y 1024(0x400) 2024-10-08T20:58:53.7693774Z z 1024(0x400) 2024-10-08T20:58:53.7694160Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T20:58:53.7694542Z Grid Max Size per Dimension: 2024-10-08T20:58:53.7694855Z x 4294967295(0xffffffff) 2024-10-08T20:58:53.7695211Z y 4294967295(0xffffffff) 2024-10-08T20:58:53.7695572Z z 4294967295(0xffffffff) 2024-10-08T20:58:53.7695959Z FBarrier Max Size: 32 2024-10-08T20:58:53.7696295Z ******* 2024-10-08T20:58:53.7696519Z Agent 4 2024-10-08T20:58:53.7696748Z ******* 2024-10-08T20:58:53.7697011Z Name: gfx90a 2024-10-08T20:58:53.7697436Z Uuid: GPU-25354a01ff6db6f2 2024-10-08T20:58:53.7697836Z Marketing Name: AMD Instinct MI210 2024-10-08T20:58:53.7698226Z Vendor Name: AMD 2024-10-08T20:58:53.7698621Z Feature: KERNEL_DISPATCH 2024-10-08T20:58:53.7699019Z Profile: BASE_PROFILE 2024-10-08T20:58:53.7699553Z Float Round Mode: NEAR 2024-10-08T20:58:53.7699949Z Max Queue Number: 128(0x80) 2024-10-08T20:58:53.7700342Z Queue Min Size: 64(0x40) 2024-10-08T20:58:53.7700725Z Queue Max Size: 131072(0x20000) 2024-10-08T20:58:53.7701104Z Queue Type: MULTI 2024-10-08T20:58:53.7701473Z Node: 3 2024-10-08T20:58:53.7701837Z Device Type: GPU 2024-10-08T20:58:53.7702162Z Cache Info: 2024-10-08T20:58:53.7702453Z L1: 16(0x10) KB 2024-10-08T20:58:53.7702815Z L2: 8192(0x2000) KB 2024-10-08T20:58:53.7703184Z Chip ID: 29711(0x740f) 2024-10-08T20:58:53.7703566Z ASIC Revision: 1(0x1) 2024-10-08T20:58:53.7703960Z Cacheline Size: 64(0x40) 2024-10-08T20:58:53.7704348Z Max Clock Freq. (MHz): 1700 2024-10-08T20:58:53.7704715Z BDFID: 33536 2024-10-08T20:58:53.7705099Z Internal Node ID: 3 2024-10-08T20:58:53.7705488Z Compute Unit: 104 2024-10-08T20:58:53.7705853Z SIMDs per CU: 4 2024-10-08T20:58:53.7706227Z Shader Engines: 8 2024-10-08T20:58:53.7706623Z Shader Arrs. per Eng.: 1 2024-10-08T20:58:53.7707029Z WatchPts on Addr. Ranges:4 2024-10-08T20:58:53.7707552Z Coherent Host Access: FALSE 2024-10-08T20:58:53.7707908Z Memory Properties: 2024-10-08T20:58:53.7708185Z Features: KERNEL_DISPATCH 2024-10-08T20:58:53.7708598Z Fast F16 Operation: TRUE 2024-10-08T20:58:53.7709005Z Wavefront Size: 64(0x40) 2024-10-08T20:58:53.7709401Z Workgroup Max Size: 1024(0x400) 2024-10-08T20:58:53.7709762Z Workgroup Max Size per Dimension: 2024-10-08T20:58:53.7710094Z x 1024(0x400) 2024-10-08T20:58:53.7710434Z y 1024(0x400) 2024-10-08T20:58:53.7710780Z z 1024(0x400) 2024-10-08T20:58:53.7711194Z Max Waves Per CU: 32(0x20) 2024-10-08T20:58:53.7711747Z Max Work-item Per CU: 2048(0x800) 2024-10-08T20:58:53.7712244Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T20:58:53.7712671Z Grid Max Size per Dimension: 2024-10-08T20:58:53.7713028Z x 4294967295(0xffffffff) 2024-10-08T20:58:53.7713402Z y 4294967295(0xffffffff) 2024-10-08T20:58:53.7713756Z z 4294967295(0xffffffff) 2024-10-08T20:58:53.7714145Z Max fbarriers/Workgrp: 32 2024-10-08T20:58:53.7714558Z Packet Processor uCode:: 83 2024-10-08T20:58:53.7714980Z SDMA engine uCode:: 8 2024-10-08T20:58:53.7715384Z IOMMU Support:: None 2024-10-08T20:58:53.7715708Z Pool Info: 2024-10-08T20:58:53.7715953Z Pool 1 2024-10-08T20:58:53.7716286Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T20:58:53.7716690Z Size: 67092480(0x3ffc000) KB 2024-10-08T20:58:53.7717084Z Allocatable: TRUE 2024-10-08T20:58:53.7717617Z Alloc Granule: 4KB 2024-10-08T20:58:53.7718048Z Alloc Recommended Granule:2048KB 2024-10-08T20:58:53.7718479Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7718901Z Accessible by all: FALSE 2024-10-08T20:58:53.7719255Z Pool 2 2024-10-08T20:58:53.7719580Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-10-08T20:58:53.7719985Z Size: 67092480(0x3ffc000) KB 2024-10-08T20:58:53.7720362Z Allocatable: TRUE 2024-10-08T20:58:53.7720760Z Alloc Granule: 4KB 2024-10-08T20:58:53.7721184Z Alloc Recommended Granule:2048KB 2024-10-08T20:58:53.7721610Z Alloc Alignment: 4KB 2024-10-08T20:58:53.7722022Z Accessible by all: FALSE 2024-10-08T20:58:53.7722362Z Pool 3 2024-10-08T20:58:53.7722667Z Segment: GROUP 2024-10-08T20:58:53.7723030Z Size: 64(0x40) KB 2024-10-08T20:58:53.7723423Z Allocatable: FALSE 2024-10-08T20:58:53.7723830Z Alloc Granule: 0KB 2024-10-08T20:58:53.7724248Z Alloc Recommended Granule:0KB 2024-10-08T20:58:53.7724664Z Alloc Alignment: 0KB 2024-10-08T20:58:53.7725076Z Accessible by all: FALSE 2024-10-08T20:58:53.7725555Z ISA Info: 2024-10-08T20:58:53.7725797Z ISA 1 2024-10-08T20:58:53.7726185Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-10-08T20:58:53.7726625Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-10-08T20:58:53.7727041Z Profiles: HSA_PROFILE_BASE 2024-10-08T20:58:53.7727456Z Default Rounding Mode: NEAR 2024-10-08T20:58:53.7727885Z Default Rounding Mode: NEAR 2024-10-08T20:58:53.7728280Z Fast f16: TRUE 2024-10-08T20:58:53.7728681Z Workgroup Max Size: 1024(0x400) 2024-10-08T20:58:53.7729103Z Workgroup Max Size per Dimension: 2024-10-08T20:58:53.7729435Z x 1024(0x400) 2024-10-08T20:58:53.7729794Z y 1024(0x400) 2024-10-08T20:58:53.7730166Z z 1024(0x400) 2024-10-08T20:58:53.7730631Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T20:58:53.7731047Z Grid Max Size per Dimension: 2024-10-08T20:58:53.7731421Z x 4294967295(0xffffffff) 2024-10-08T20:58:53.7731848Z y 4294967295(0xffffffff) 2024-10-08T20:58:53.7732251Z z 4294967295(0xffffffff) 2024-10-08T20:58:53.7732643Z FBarrier Max Size: 32 2024-10-08T20:58:53.7732996Z *** Done *** 2024-10-08T20:58:53.7815006Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-10-08T20:58:53.7816046Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-10-08T20:58:53.7817920Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-10-08T20:58:53.7819657Z if [[ $ngpu -eq 0 ]]; then 2024-10-08T20:58:53.7820916Z  echo "Error: Failed to detect any GPUs on the runner" 2024-10-08T20:58:53.7821902Z  echo "$msg" 2024-10-08T20:58:53.7822560Z  exit 1 2024-10-08T20:58:53.7823156Z fi 2024-10-08T20:58:53.7823722Z if [[ $ngpu -eq 1 ]]; then 2024-10-08T20:58:53.7825085Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-10-08T20:58:53.7826462Z  echo "$msg" 2024-10-08T20:58:53.7826908Z  exit 1 2024-10-08T20:58:53.7827202Z fi 2024-10-08T20:58:53.7871013Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.7871536Z env: 2024-10-08T20:58:53.7872021Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.7872830Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.7873708Z ##[endgroup] 2024-10-08T20:58:53.8820582Z Prepare all required actions 2024-10-08T20:58:53.8868955Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-10-08T20:58:53.8869344Z with: 2024-10-08T20:58:53.8869580Z diskspace-cutoff: 70 2024-10-08T20:58:53.8869849Z env: 2024-10-08T20:58:53.8870071Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.8870421Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.8870799Z ##[endgroup] 2024-10-08T20:58:53.8957023Z ##[group]Run set -ex 2024-10-08T20:58:53.8957479Z set -ex 2024-10-08T20:58:53.8957882Z diskspace_cutoff=70 2024-10-08T20:58:53.8958529Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-10-08T20:58:53.8959630Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-10-08T20:58:53.8961318Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-10-08T20:58:53.8963038Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-10-08T20:58:53.8963736Z  docker system prune -af 2024-10-08T20:58:53.8964677Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-10-08T20:58:53.8965770Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-10-08T20:58:53.8966905Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-10-08T20:58:53.8967859Z  echo "$msg" 2024-10-08T20:58:53.8968301Z  exit 1 2024-10-08T20:58:53.8968701Z  else 2024-10-08T20:58:53.8969355Z  difference=$((diskspace - diskspace_new)) 2024-10-08T20:58:53.8970112Z  echo "Diskspace saved: $difference percent" 2024-10-08T20:58:53.8970716Z  fi 2024-10-08T20:58:53.8971070Z fi 2024-10-08T20:58:53.9007488Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.9008150Z env: 2024-10-08T20:58:53.9008495Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.9009117Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.9009679Z ##[endgroup] 2024-10-08T20:58:53.9071609Z + diskspace_cutoff=70 2024-10-08T20:58:53.9077474Z ++ docker info -f '{{.DockerRootDir}}' 2024-10-08T20:58:53.9560308Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-10-08T20:58:53.9568482Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-10-08T20:58:53.9569320Z ++ sed -n 2p 2024-10-08T20:58:53.9570298Z ++ sed s/%// 2024-10-08T20:58:53.9571197Z ++ sed 's/ //' 2024-10-08T20:58:53.9587243Z + diskspace=47 2024-10-08T20:58:53.9588015Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-10-08T20:58:53.9588747Z + [[ 47 -ge 70 ]] 2024-10-08T20:58:53.9630650Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T20:58:53.9631512Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T20:58:53.9632556Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-10-08T20:58:53.9666216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.9666664Z env: 2024-10-08T20:58:53.9666899Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.9667262Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.9667641Z ##[endgroup] 2024-10-08T20:58:53.9784839Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-10-08T20:58:53.9785601Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-10-08T20:58:53.9786443Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-10-08T20:58:53.9810730Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:53.9811114Z env: 2024-10-08T20:58:53.9811331Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.9811631Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.9811952Z ##[endgroup] 2024-10-08T20:58:53.9908726Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-10-08T20:58:53.9909098Z with: 2024-10-08T20:58:53.9909449Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-10-08T20:58:53.9909904Z aws-region: us-east-1 2024-10-08T20:58:53.9910161Z role-duration-seconds: 18000 2024-10-08T20:58:53.9910433Z audience: sts.amazonaws.com 2024-10-08T20:58:53.9910678Z env: 2024-10-08T20:58:53.9910869Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:53.9911161Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:53.9911711Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:53.9912215Z ##[endgroup] 2024-10-08T20:58:54.5947822Z Assuming role with OIDC 2024-10-08T20:58:54.9436281Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-10-08T20:58:55.0544114Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-10-08T20:58:55.0544910Z with: 2024-10-08T20:58:55.0545377Z mask-password: true 2024-10-08T20:58:55.0545941Z registry-type: private 2024-10-08T20:58:55.0546512Z skip-logout: false 2024-10-08T20:58:55.0547009Z env: 2024-10-08T20:58:55.0547456Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:55.0548150Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:55.0549440Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:55.0550640Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:55.0551273Z AWS_REGION: us-east-1 2024-10-08T20:58:55.0552083Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:55.0552947Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:55.0566222Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:55.0566847Z ##[endgroup] 2024-10-08T20:58:55.5323022Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2322038Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-10-08T20:58:56.2322563Z with: 2024-10-08T20:58:56.2323214Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2323961Z docker-build-dir: .ci/docker 2024-10-08T20:58:56.2324248Z working-directory: . 2024-10-08T20:58:56.2324602Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2325001Z force-push: false 2024-10-08T20:58:56.2325232Z env: 2024-10-08T20:58:56.2325442Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:56.2325750Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:56.2326335Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:56.2326872Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:56.2327163Z AWS_REGION: us-east-1 2024-10-08T20:58:56.2327535Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:56.2327918Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:56.2333596Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:56.2334074Z ##[endgroup] 2024-10-08T20:58:56.2350444Z ##[group]Run set -ex 2024-10-08T20:58:56.2350714Z set -ex 2024-10-08T20:58:56.2350940Z  2024-10-08T20:58:56.2351373Z # If the docker build directory or the build script doesn't exist, the action will 2024-10-08T20:58:56.2352099Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-10-08T20:58:56.2352709Z # job could then download the pre-built image as usual 2024-10-08T20:58:56.2353272Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-10-08T20:58:56.2353778Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2354252Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2354668Z  2024-10-08T20:58:56.2355063Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-10-08T20:58:56.2355534Z  exit 0 2024-10-08T20:58:56.2355775Z else 2024-10-08T20:58:56.2356054Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2356395Z fi 2024-10-08T20:58:56.2356607Z  2024-10-08T20:58:56.2356960Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-10-08T20:58:56.2357598Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-10-08T20:58:56.2358183Z  # use it as it is, but first let's extract the tag 2024-10-08T20:58:56.2358709Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-10-08T20:58:56.2359249Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2359772Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2360191Z else 2024-10-08T20:58:56.2360518Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-10-08T20:58:56.2361007Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2361677Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2362238Z fi 2024-10-08T20:58:56.2386345Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:56.2386733Z env: 2024-10-08T20:58:56.2386946Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:56.2387263Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:56.2387821Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:56.2388356Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:56.2388658Z AWS_REGION: us-east-1 2024-10-08T20:58:56.2388973Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:56.2389361Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:56.2395154Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:56.2395430Z REPO_NAME: pytorch 2024-10-08T20:58:56.2396112Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2396839Z DOCKER_BUILD_DIR: .ci/docker 2024-10-08T20:58:56.2397216Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2397607Z ##[endgroup] 2024-10-08T20:58:56.2441339Z + [[ ! -d .ci/docker ]] 2024-10-08T20:58:56.2441658Z + [[ ! -f .ci/docker/build.sh ]] 2024-10-08T20:58:56.2441953Z + echo skip=false 2024-10-08T20:58:56.2444902Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-10-08T20:58:56.2459028Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2460023Z ++ awk -F '[:,]' '{print $2}' 2024-10-08T20:58:56.2480765Z + DOCKER_TAG=46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2481422Z + echo docker-tag=46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2482358Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2553285Z ##[group]Run set +e 2024-10-08T20:58:56.2553945Z set +e 2024-10-08T20:58:56.2554492Z set -x 2024-10-08T20:58:56.2555005Z  2024-10-08T20:58:56.2555497Z login() { 2024-10-08T20:58:56.2556735Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-10-08T20:58:56.2558046Z } 2024-10-08T20:58:56.2558534Z  2024-10-08T20:58:56.2559022Z retry () { 2024-10-08T20:58:56.2559767Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-10-08T20:58:56.2560573Z } 2024-10-08T20:58:56.2561056Z  2024-10-08T20:58:56.2561610Z retry login "${DOCKER_REGISTRY}" 2024-10-08T20:58:56.2562374Z  2024-10-08T20:58:56.2563218Z # Check if image already exists, if it does then skip building it 2024-10-08T20:58:56.2564496Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-10-08T20:58:56.2565406Z  exit 0 2024-10-08T20:58:56.2565939Z fi 2024-10-08T20:58:56.2566428Z  2024-10-08T20:58:56.2567326Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-10-08T20:58:56.2568990Z # be empty. The default action would be to continue rebuild the image 2024-10-08T20:58:56.2570323Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-10-08T20:58:56.2571545Z  # if we're on the base branch then use the parent commit 2024-10-08T20:58:56.2572624Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-10-08T20:58:56.2573402Z else 2024-10-08T20:58:56.2574240Z  # otherwise we're on a PR, so use the most recent base commit 2024-10-08T20:58:56.2575485Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-10-08T20:58:56.2576380Z fi 2024-10-08T20:58:56.2576871Z  2024-10-08T20:58:56.2577431Z if [[ -z "${MERGE_BASE}" ]]; then 2024-10-08T20:58:56.2578335Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2579144Z  2024-10-08T20:58:56.2580314Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-10-08T20:58:56.2581678Z  exit 0 2024-10-08T20:58:56.2582210Z fi 2024-10-08T20:58:56.2582776Z  2024-10-08T20:58:56.2583693Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-10-08T20:58:56.2586000Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-10-08T20:58:56.2587479Z  exit 1 2024-10-08T20:58:56.2588011Z fi 2024-10-08T20:58:56.2588503Z  2024-10-08T20:58:56.2589401Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-10-08T20:58:56.2591133Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-10-08T20:58:56.2592788Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-10-08T20:58:56.2594887Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-10-08T20:58:56.2596924Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-10-08T20:58:56.2598092Z fi 2024-10-08T20:58:56.2598584Z  2024-10-08T20:58:56.2599215Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-10-08T20:58:56.2652897Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:56.2653807Z env: 2024-10-08T20:58:56.2654304Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:56.2655676Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:56.2657322Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:56.2658856Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:56.2659682Z AWS_REGION: us-east-1 2024-10-08T20:58:56.2660608Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:56.2661544Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:56.2676067Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:56.2676754Z DOCKER_BUILD_DIR: .ci/docker 2024-10-08T20:58:56.2677588Z BASE_REVISION: 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T20:58:56.2679473Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2681309Z DOCKER_TAG: 46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:56.2682403Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2683342Z ##[endgroup] 2024-10-08T20:58:56.2762915Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2764388Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:56.2769766Z + aws ecr get-login-password --region us-east-1 2024-10-08T20:58:56.2773235Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:57.6788311Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-10-08T20:58:57.6790065Z Configure a credential helper to remove this warning. See 2024-10-08T20:58:57.6791948Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-10-08T20:58:57.6792911Z 2024-10-08T20:58:57.6793397Z Login Succeeded 2024-10-08T20:58:57.6822051Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:59.0518857Z { 2024-10-08T20:58:59.0519449Z "schemaVersion": 2, 2024-10-08T20:58:59.0520536Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-10-08T20:58:59.0521801Z "config": { 2024-10-08T20:58:59.0522922Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-10-08T20:58:59.0524086Z "size": 24313, 2024-10-08T20:58:59.0525225Z "digest": "sha256:88227c49651e28ea3119acf85b7b78a20905866778992f06b72a24991ecb0597" 2024-10-08T20:58:59.0526584Z }, 2024-10-08T20:58:59.0527119Z "layers": [ 2024-10-08T20:58:59.0527712Z { 2024-10-08T20:58:59.0528579Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0529260Z "size": 28583886, 2024-10-08T20:58:59.0529783Z "digest": "sha256:708f64f82dd88cf1254c197fe8a812ffc7c3ebf200e9ac71f489d96160efa1d6" 2024-10-08T20:58:59.0530370Z }, 2024-10-08T20:58:59.0530600Z { 2024-10-08T20:58:59.0531303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0531838Z "size": 1822, 2024-10-08T20:58:59.0532355Z "digest": "sha256:2fb85aafeb0f699a68fadc39e9d676831cf9593cd25b0a0469e470c0299ea0f0" 2024-10-08T20:58:59.0532940Z }, 2024-10-08T20:58:59.0533170Z { 2024-10-08T20:58:59.0533569Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0534083Z "size": 312567771, 2024-10-08T20:58:59.0534611Z "digest": "sha256:364c1425fe730e5433806b5afda11e443a4187ea03f82c52da28a7082555d00a" 2024-10-08T20:58:59.0535183Z }, 2024-10-08T20:58:59.0535412Z { 2024-10-08T20:58:59.0535811Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0536320Z "size": 844, 2024-10-08T20:58:59.0536827Z "digest": "sha256:f241ef385a0c67180ea4de31eb3fc914c100077debe9f05ab0b53de47abb9619" 2024-10-08T20:58:59.0537405Z }, 2024-10-08T20:58:59.0537634Z { 2024-10-08T20:58:59.0538031Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0538548Z "size": 106, 2024-10-08T20:58:59.0539051Z "digest": "sha256:26dc666f7f84d8f4ccec33cf44f73fa12cc948288a165654df604ce588962aba" 2024-10-08T20:58:59.0539851Z }, 2024-10-08T20:58:59.0540081Z { 2024-10-08T20:58:59.0540484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0540996Z "size": 544, 2024-10-08T20:58:59.0541479Z "digest": "sha256:855716c86139073a59128e39db0202969b89202e91a5beac6a5dc1944ca12645" 2024-10-08T20:58:59.0542041Z }, 2024-10-08T20:58:59.0542272Z { 2024-10-08T20:58:59.0542669Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0543177Z "size": 1258, 2024-10-08T20:58:59.0543678Z "digest": "sha256:b409476bef7cb9789898d501f5b6e111b1b083135b26b2ed05fe35c1dec03a4d" 2024-10-08T20:58:59.0544247Z }, 2024-10-08T20:58:59.0544472Z { 2024-10-08T20:58:59.0544874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0545382Z "size": 3482, 2024-10-08T20:58:59.0545883Z "digest": "sha256:8d984b3a88fb542a7896055661622cac9dcdae28f8b8346cda0032084df21f62" 2024-10-08T20:58:59.0546454Z }, 2024-10-08T20:58:59.0546684Z { 2024-10-08T20:58:59.0547083Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0547588Z "size": 1959, 2024-10-08T20:58:59.0548082Z "digest": "sha256:69b61c71134f3edd16a7d45ccbab75d657334742ef2850a558810977a35d4c3f" 2024-10-08T20:58:59.0548643Z }, 2024-10-08T20:58:59.0548869Z { 2024-10-08T20:58:59.0549437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0549940Z "size": 702, 2024-10-08T20:58:59.0550449Z "digest": "sha256:ccad0fc55991ccb91a8481c65defeb58a8e88f3eb5d9ade108a985e430e176d6" 2024-10-08T20:58:59.0551027Z }, 2024-10-08T20:58:59.0551265Z { 2024-10-08T20:58:59.0551658Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0552162Z "size": 2666878929, 2024-10-08T20:58:59.0552693Z "digest": "sha256:ed247529a9c951c4a45bff699203be290ef1ff224747ba0c2cfd2a4a75428d4c" 2024-10-08T20:58:59.0553276Z }, 2024-10-08T20:58:59.0553541Z { 2024-10-08T20:58:59.0554236Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0555392Z "size": 380, 2024-10-08T20:58:59.0556494Z "digest": "sha256:f60cecf49794ecb89234a72e261cacf1114620cab68c2f7004f99633a53353c8" 2024-10-08T20:58:59.0557582Z }, 2024-10-08T20:58:59.0558007Z { 2024-10-08T20:58:59.0558765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0559728Z "size": 12121, 2024-10-08T20:58:59.0560717Z "digest": "sha256:590abb1ef3538da5ff1ace7d76f291e7f02af9edf835440ad3b37675b9d94c29" 2024-10-08T20:58:59.0561824Z }, 2024-10-08T20:58:59.0562255Z { 2024-10-08T20:58:59.0563010Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0563979Z "size": 504, 2024-10-08T20:58:59.0565192Z "digest": "sha256:fdbf607f8291c58c8d21ac5fe057ef6e95d0404e76f9b5c9829ec4095cb4cc6c" 2024-10-08T20:58:59.0566300Z }, 2024-10-08T20:58:59.0566729Z { 2024-10-08T20:58:59.0567478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0568464Z "size": 121477519, 2024-10-08T20:58:59.0569580Z "digest": "sha256:7528f1a81f33b9a81067acc191088caab7ba3200df03f0c9c0d1c6c702039be5" 2024-10-08T20:58:59.0570675Z }, 2024-10-08T20:58:59.0571099Z { 2024-10-08T20:58:59.0571846Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0572994Z "size": 109, 2024-10-08T20:58:59.0573989Z "digest": "sha256:9071f0ddb0162bd313cd562641f075761918efecaf3b5c25f75e0821e2bb94da" 2024-10-08T20:58:59.0575083Z }, 2024-10-08T20:58:59.0575538Z { 2024-10-08T20:58:59.0576387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0577533Z "size": 489, 2024-10-08T20:58:59.0578687Z "digest": "sha256:fdecd86ee80b7ef3c5b6389ab0946e5dd6e0c39f97acdf300e63bb1407a7b6df" 2024-10-08T20:58:59.0580025Z }, 2024-10-08T20:58:59.0580533Z { 2024-10-08T20:58:59.0581401Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0582214Z "size": 299, 2024-10-08T20:58:59.0582707Z "digest": "sha256:f7dac7900b8817518fc81a306f611467810d1140d4afac2a8c187f0bbd2e733d" 2024-10-08T20:58:59.0583272Z }, 2024-10-08T20:58:59.0583502Z { 2024-10-08T20:58:59.0583899Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0584404Z "size": 103, 2024-10-08T20:58:59.0584898Z "digest": "sha256:dbd308e98b3c409c31e159eda42f61a359a5994a0bd17db673f2466daee92793" 2024-10-08T20:58:59.0585473Z }, 2024-10-08T20:58:59.0585699Z { 2024-10-08T20:58:59.0586088Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0586591Z "size": 1472, 2024-10-08T20:58:59.0587098Z "digest": "sha256:894f96cdb5920f1f13ef6e669d82da523eb898ff63ca3d7723a20081b43f440d" 2024-10-08T20:58:59.0587750Z }, 2024-10-08T20:58:59.0588022Z { 2024-10-08T20:58:59.0588483Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0589089Z "size": 597792409, 2024-10-08T20:58:59.0589655Z "digest": "sha256:4944fc0fe0ccb21ad39f4cf10985a5cc8aabae9c2e176b9f6ee4c7288b41705f" 2024-10-08T20:58:59.0590236Z }, 2024-10-08T20:58:59.0590467Z { 2024-10-08T20:58:59.0590856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0591361Z "size": 164, 2024-10-08T20:58:59.0591843Z "digest": "sha256:f2b4d9811e855378ea3571d0dd743b1c857ed05250b3964d73607af0e8c9b51a" 2024-10-08T20:58:59.0592394Z }, 2024-10-08T20:58:59.0592619Z { 2024-10-08T20:58:59.0593005Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0593510Z "size": 1640, 2024-10-08T20:58:59.0594012Z "digest": "sha256:b982a33cdcc7526ff5ba9bed565ca8538581c2e6658f22e1e5a6f32e78a1f740" 2024-10-08T20:58:59.0594632Z }, 2024-10-08T20:58:59.0594955Z { 2024-10-08T20:58:59.0595852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0596995Z "size": 7842040912, 2024-10-08T20:58:59.0598047Z "digest": "sha256:020353fa70e56dd095c5132ed440bd727fbe8add51ef43b783d63da300c7bf8b" 2024-10-08T20:58:59.0599142Z }, 2024-10-08T20:58:59.0599573Z { 2024-10-08T20:58:59.0600319Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0601284Z "size": 105, 2024-10-08T20:58:59.0602216Z "digest": "sha256:888d0f82882d0fec25f7cf962d77a5e21c6c15c272dac6043ec6e0284142e2a9" 2024-10-08T20:58:59.0603297Z }, 2024-10-08T20:58:59.0603721Z { 2024-10-08T20:58:59.0604455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0605411Z "size": 973, 2024-10-08T20:58:59.0606364Z "digest": "sha256:29e2cc227c231b33fd8e12c95befe145d1d2cb448b1d8695d0a71d6e7994b5c1" 2024-10-08T20:58:59.0607460Z }, 2024-10-08T20:58:59.0607885Z { 2024-10-08T20:58:59.0609047Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0610099Z "size": 318523819, 2024-10-08T20:58:59.0611133Z "digest": "sha256:f0dbe6dcdcf05c91e9dba0973e7ddd9b7f5245be8a77ee4117d79bd76a67bfb7" 2024-10-08T20:58:59.0612271Z }, 2024-10-08T20:58:59.0612692Z { 2024-10-08T20:58:59.0613455Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0614418Z "size": 111, 2024-10-08T20:58:59.0615339Z "digest": "sha256:a5411746726de495d0ed190c23262c68bf1bff5e44f856ba349830301ab8c82e" 2024-10-08T20:58:59.0616585Z }, 2024-10-08T20:58:59.0617088Z { 2024-10-08T20:58:59.0617967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0619114Z "size": 1829, 2024-10-08T20:58:59.0620233Z "digest": "sha256:4bd3d3dc2459202ba6e77310a1c3a696b82ac421378ef92531ccc37092dde543" 2024-10-08T20:58:59.0621425Z }, 2024-10-08T20:58:59.0621688Z { 2024-10-08T20:58:59.0622108Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0622623Z "size": 619472979, 2024-10-08T20:58:59.0623128Z "digest": "sha256:078a2b25eab622942ee0e409cc516d1f91d60f40554081d44467339fb397d10d" 2024-10-08T20:58:59.0623854Z }, 2024-10-08T20:58:59.0624082Z { 2024-10-08T20:58:59.0624479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0624981Z "size": 166, 2024-10-08T20:58:59.0625468Z "digest": "sha256:f0ab9f85112ad166e29a032cc6136bee2b53df459a7b75509df49d2b87ffe911" 2024-10-08T20:58:59.0626036Z }, 2024-10-08T20:58:59.0626263Z { 2024-10-08T20:58:59.0626654Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0627182Z "size": 2850016, 2024-10-08T20:58:59.0627931Z "digest": "sha256:935001a734f35ae759855755a0e5e9fe7e24bcf6ea3ff8a38d6ce308862963ab" 2024-10-08T20:58:59.0629182Z }, 2024-10-08T20:58:59.0629690Z { 2024-10-08T20:58:59.0630489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0631440Z "size": 107, 2024-10-08T20:58:59.0632398Z "digest": "sha256:609aba5c6a84c664aa8c41f273754345098fd44c24decafb50c10ed0cbfa5bd5" 2024-10-08T20:58:59.0633503Z }, 2024-10-08T20:58:59.0633926Z { 2024-10-08T20:58:59.0634667Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0635632Z "size": 565, 2024-10-08T20:58:59.0636586Z "digest": "sha256:d03add5dc732b3fc5a0f0d109033fa9e20db310cd815782abd95fac7ee70a02e" 2024-10-08T20:58:59.0637696Z }, 2024-10-08T20:58:59.0638124Z { 2024-10-08T20:58:59.0638853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0639821Z "size": 43193853, 2024-10-08T20:58:59.0640812Z "digest": "sha256:fe97dd154164ed60b207ff918bbef7c610f6f7f5ec281df5c9f788ceb83cfdf8" 2024-10-08T20:58:59.0641906Z }, 2024-10-08T20:58:59.0642329Z { 2024-10-08T20:58:59.0643077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0644033Z "size": 106, 2024-10-08T20:58:59.0645006Z "digest": "sha256:a3247cea85dd33d244a1aad718c3ea42a3ccc5a0e519b2812e932b30bdd6f7cb" 2024-10-08T20:58:59.0646122Z }, 2024-10-08T20:58:59.0646553Z { 2024-10-08T20:58:59.0647297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0648267Z "size": 295, 2024-10-08T20:58:59.0649302Z "digest": "sha256:cba178c95290b80c7851639d4c0ae8743196751c87c63e31fa97151786f28b25" 2024-10-08T20:58:59.0650385Z }, 2024-10-08T20:58:59.0650805Z { 2024-10-08T20:58:59.0651546Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0652495Z "size": 88288, 2024-10-08T20:58:59.0653453Z "digest": "sha256:a2b3287aa011e5ddedbd1524b5273030500360c1cb7af3166ea38739d9f57adf" 2024-10-08T20:58:59.0654543Z }, 2024-10-08T20:58:59.0654966Z { 2024-10-08T20:58:59.0655747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0656879Z "size": 104, 2024-10-08T20:58:59.0658326Z "digest": "sha256:ce601e88ad1347f2034a7cbad87753b740874727159a44e6a0038901c77ae4c8" 2024-10-08T20:58:59.0659613Z }, 2024-10-08T20:58:59.0660116Z { 2024-10-08T20:58:59.0660997Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0661739Z "size": 1402, 2024-10-08T20:58:59.0662261Z "digest": "sha256:6f38a7722467ffeadc6daf780a3b8787e97ec0e2d419e94ac195afb285b31bbc" 2024-10-08T20:58:59.0662827Z }, 2024-10-08T20:58:59.0663058Z { 2024-10-08T20:58:59.0663447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0663946Z "size": 702, 2024-10-08T20:58:59.0664448Z "digest": "sha256:ccad0fc55991ccb91a8481c65defeb58a8e88f3eb5d9ade108a985e430e176d6" 2024-10-08T20:58:59.0665024Z }, 2024-10-08T20:58:59.0665249Z { 2024-10-08T20:58:59.0665636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0666141Z "size": 136, 2024-10-08T20:58:59.0666630Z "digest": "sha256:93977d3c7c12d04ac0686624e59f324ccdf33b452e1113a5422d959e5b3a47f2" 2024-10-08T20:58:59.0667218Z }, 2024-10-08T20:58:59.0667488Z { 2024-10-08T20:58:59.0668295Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0669843Z "size": 120, 2024-10-08T20:58:59.0670820Z "digest": "sha256:6fc88b7ef267ac019b9e637fbb89867f03f9d98ed4a1059a5a01fb6b0c7df40b" 2024-10-08T20:58:59.0671917Z }, 2024-10-08T20:58:59.0672342Z { 2024-10-08T20:58:59.0673091Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0674066Z "size": 4406327397, 2024-10-08T20:58:59.0675076Z "digest": "sha256:14e12711e6aa25d9c6c3c282ecbdc5c3403c3978ca13a6af22f257a0bb1105f3" 2024-10-08T20:58:59.0676161Z }, 2024-10-08T20:58:59.0676587Z { 2024-10-08T20:58:59.0677325Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0678281Z "size": 174, 2024-10-08T20:58:59.0679250Z "digest": "sha256:ceafacffdb8b98c09a79abe601dc162b01e9e2422f7cd36e931f09330c4726dd" 2024-10-08T20:58:59.0680368Z }, 2024-10-08T20:58:59.0680808Z { 2024-10-08T20:58:59.0681547Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0682512Z "size": 209, 2024-10-08T20:58:59.0683455Z "digest": "sha256:4c2d1343a07f7f5bac94561c89be63a56a9587c7aff3b16adc097fbc84ea6b20" 2024-10-08T20:58:59.0684540Z }, 2024-10-08T20:58:59.0684962Z { 2024-10-08T20:58:59.0685704Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0686670Z "size": 702, 2024-10-08T20:58:59.0687636Z "digest": "sha256:ccad0fc55991ccb91a8481c65defeb58a8e88f3eb5d9ade108a985e430e176d6" 2024-10-08T20:58:59.0688749Z }, 2024-10-08T20:58:59.0689295Z { 2024-10-08T20:58:59.0690049Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0691014Z "size": 634, 2024-10-08T20:58:59.0691928Z "digest": "sha256:34749e4bc2224d5590f263bb3abd34b4312e0154100f895a83636718a20695f0" 2024-10-08T20:58:59.0692995Z }, 2024-10-08T20:58:59.0693419Z { 2024-10-08T20:58:59.0694171Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0695146Z "size": 652359923, 2024-10-08T20:58:59.0696260Z "digest": "sha256:be0890515218ed2524bfcaa57972c9171c0fb8c202e2d210e88fd5be613fd9ca" 2024-10-08T20:58:59.0697538Z }, 2024-10-08T20:58:59.0698039Z { 2024-10-08T20:58:59.0698915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0700050Z "size": 1841, 2024-10-08T20:58:59.0701162Z "digest": "sha256:d733d2cebd2d2d18d7d5d022103696f2094ef7457a8423dd3a8ea25b064c164d" 2024-10-08T20:58:59.0701892Z }, 2024-10-08T20:58:59.0702125Z { 2024-10-08T20:58:59.0702511Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0703012Z "size": 15982978, 2024-10-08T20:58:59.0703538Z "digest": "sha256:c95d8bebb08902efc65cdb53ffba15e3441d603d6fa74ff4cf1f4e1fd6b5e949" 2024-10-08T20:58:59.0704106Z }, 2024-10-08T20:58:59.0704331Z { 2024-10-08T20:58:59.0704884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-10-08T20:58:59.0705383Z "size": 54145659, 2024-10-08T20:58:59.0705893Z "digest": "sha256:eadebda12f3d4bcb3209649505f285a521ba238c157b3e6a49e07f0531c965cc" 2024-10-08T20:58:59.0706473Z } 2024-10-08T20:58:59.0706698Z ] 2024-10-08T20:58:59.0706929Z } 2024-10-08T20:58:59.0707198Z + exit 0 2024-10-08T20:58:59.0943951Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-10-08T20:58:59.0944760Z with: 2024-10-08T20:58:59.0945912Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:59.0947371Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:59.0948060Z env: 2024-10-08T20:58:59.0948433Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:59.0948999Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:59.0950021Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:59.0950988Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:59.0951498Z AWS_REGION: us-east-1 2024-10-08T20:58:59.0952480Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:59.0953183Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:59.0963917Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:59.0964386Z ##[endgroup] 2024-10-08T20:58:59.0991692Z ##[group]Run set -x 2024-10-08T20:58:59.0992154Z set -x 2024-10-08T20:58:59.0992565Z set +e 2024-10-08T20:58:59.0992957Z  2024-10-08T20:58:59.0993358Z login() { 2024-10-08T20:58:59.0994454Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-10-08T20:58:59.0995608Z } 2024-10-08T20:58:59.0996045Z  2024-10-08T20:58:59.0996447Z retry () { 2024-10-08T20:58:59.0996987Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-10-08T20:58:59.0997594Z } 2024-10-08T20:58:59.0997963Z  2024-10-08T20:58:59.0998398Z retry login "${DOCKER_REGISTRY}" 2024-10-08T20:58:59.0998960Z  2024-10-08T20:58:59.0999323Z set -e 2024-10-08T20:58:59.0999977Z # ignore output since only exit code is used for conditional 2024-10-08T20:58:59.1000936Z # only pull docker image if it's not available locally 2024-10-08T20:58:59.1001989Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-10-08T20:58:59.1002932Z  retry docker pull "${DOCKER_IMAGE}" 2024-10-08T20:58:59.1003523Z fi 2024-10-08T20:58:59.1044251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:58:59.1044950Z env: 2024-10-08T20:58:59.1045328Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:58:59.1045898Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:58:59.1046943Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:58:59.1047938Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:58:59.1048458Z AWS_REGION: us-east-1 2024-10-08T20:58:59.1049139Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:58:59.1049838Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:58:59.1059840Z AWS_SESSION_TOKEN: *** 2024-10-08T20:58:59.1060526Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:58:59.1061329Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:59.1061720Z ##[endgroup] 2024-10-08T20:58:59.1102588Z + set +e 2024-10-08T20:58:59.1103213Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:59.1103857Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:59.1108092Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T20:58:59.1108851Z + aws ecr get-login-password --region us-east-1 2024-10-08T20:59:00.5159397Z WARNING! Your password will be stored unencrypted in /home/pytorchci/.docker/config.json. 2024-10-08T20:59:00.5161266Z Configure a credential helper to remove this warning. See 2024-10-08T20:59:00.5163140Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-10-08T20:59:00.5164191Z 2024-10-08T20:59:00.5165001Z Login Succeeded 2024-10-08T20:59:00.5199699Z + set -e 2024-10-08T20:59:00.5201299Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T20:59:00.5406636Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-10-08T20:59:00.5407422Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-10-08T20:59:00.5408043Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-10-08T20:59:00.5408645Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-10-08T20:59:00.5433613Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T20:59:00.5434142Z env: 2024-10-08T20:59:00.5434463Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:59:00.5434872Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:59:00.5435800Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:59:00.5436434Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:59:00.5436823Z AWS_REGION: us-east-1 2024-10-08T20:59:00.5437335Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:59:00.5437811Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:59:00.5443614Z AWS_SESSION_TOKEN: *** 2024-10-08T20:59:00.5443988Z ##[endgroup] 2024-10-08T20:59:00.8000050Z Defaulting to user installation because normal site-packages is not writeable 2024-10-08T20:59:00.8719300Z Requirement already satisfied: psutil==5.9.1 in /home/pytorchci/.local/lib/python3.10/site-packages (5.9.1) 2024-10-08T20:59:00.8723344Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/pytorchci/.local/lib/python3.10/site-packages (11.525.84) 2024-10-08T20:59:01.0107821Z Prepare all required actions 2024-10-08T20:59:01.0108818Z Getting action download info 2024-10-08T20:59:01.2026202Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-10-08T20:59:01.8548228Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-10-08T20:59:02.3088605Z ##[group]Run ./.github/actions/download-build-artifacts 2024-10-08T20:59:02.3089594Z with: 2024-10-08T20:59:02.3090122Z name: linux-focal-rocm6.2-py3.10 2024-10-08T20:59:02.3090842Z s3-bucket: gha-artifacts 2024-10-08T20:59:02.3091426Z env: 2024-10-08T20:59:02.3091906Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:59:02.3092630Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:59:02.3094007Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:59:02.3095286Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:59:02.3096139Z AWS_REGION: us-east-1 2024-10-08T20:59:02.3097020Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:59:02.3098096Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:59:02.3107161Z AWS_SESSION_TOKEN: *** 2024-10-08T20:59:02.3107549Z ##[endgroup] 2024-10-08T20:59:02.3174742Z ##[group]Run seemethere/download-artifact-s3@v4 2024-10-08T20:59:02.3175627Z with: 2024-10-08T20:59:02.3176270Z name: linux-focal-rocm6.2-py3.10 2024-10-08T20:59:02.3177148Z s3-bucket: gha-artifacts 2024-10-08T20:59:02.3177919Z region: us-east-1 2024-10-08T20:59:02.3178537Z env: 2024-10-08T20:59:02.3179113Z GIT_DEFAULT_BRANCH: main 2024-10-08T20:59:02.3179995Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T20:59:02.3181525Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T20:59:02.3182263Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T20:59:02.3182613Z AWS_REGION: us-east-1 2024-10-08T20:59:02.3183026Z AWS_ACCESS_KEY_ID: *** 2024-10-08T20:59:02.3183515Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T20:59:02.3190700Z AWS_SESSION_TOKEN: *** 2024-10-08T20:59:02.3191015Z ##[endgroup] 2024-10-08T20:59:02.7134361Z (node:2519132) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-10-08T20:59:02.7135699Z 2024-10-08T20:59:02.7136129Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-10-08T20:59:02.7137235Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-10-08T20:59:02.7138733Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-10-08T20:59:03.0180660Z Found 1 objects with prefix pytorch/pytorch/11242664656/linux-focal-rocm6.2-py3.10/ 2024-10-08T20:59:03.0182718Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-10-08T21:00:10.8329684Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-10-08T21:00:10.8348219Z Artifact download has finished successfully 2024-10-08T21:00:10.8838151Z ##[group]Run unzip -o artifacts.zip 2024-10-08T21:00:10.8838768Z unzip -o artifacts.zip 2024-10-08T21:00:10.8878869Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:10.8879920Z env: 2024-10-08T21:00:10.8880308Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:10.8880885Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:10.8881953Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:10.8882951Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:10.8883478Z AWS_REGION: us-east-1 2024-10-08T21:00:10.8884057Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:10.8884769Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:10.8895494Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:10.8895974Z ##[endgroup] 2024-10-08T21:00:10.8991161Z Archive: artifacts.zip 2024-10-08T21:00:10.8991880Z creating: dist/ 2024-10-08T21:00:15.4282115Z inflating: dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl 2024-10-08T21:00:15.4406935Z inflating: dist/.ninja_log 2024-10-08T21:00:15.4407871Z creating: build/custom_test_artifacts/ 2024-10-08T21:00:15.4409277Z creating: build/custom_test_artifacts/custom-op-build/ 2024-10-08T21:00:15.4410628Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-10-08T21:00:15.4412159Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-10-08T21:00:15.4413963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T21:00:15.4415681Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-10-08T21:00:15.4417390Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T21:00:15.4419168Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T21:00:15.4420186Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T21:00:15.4421250Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T21:00:15.4422264Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T21:00:15.4423182Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T21:00:15.4424102Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T21:00:15.4425172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T21:00:15.4426237Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T21:00:15.4427316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T21:00:15.4428434Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T21:00:15.4430555Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T21:00:15.4432630Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T21:00:15.4434403Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-10-08T21:00:15.4435964Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-10-08T21:00:15.4437580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-10-08T21:00:15.4439253Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-10-08T21:00:15.4441099Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-10-08T21:00:15.4443164Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-10-08T21:00:15.4445148Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-10-08T21:00:15.4446992Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-10-08T21:00:15.4449538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-10-08T21:00:15.4451495Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-10-08T21:00:15.4453427Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-10-08T21:00:15.4455336Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-10-08T21:00:15.4457246Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-10-08T21:00:15.4459128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-10-08T21:00:15.4582754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-10-08T21:00:15.4583791Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-10-08T21:00:15.4584846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-10-08T21:00:15.4585983Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-10-08T21:00:15.4587061Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-10-08T21:00:15.4588092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-10-08T21:00:15.4589652Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-10-08T21:00:15.4591684Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-10-08T21:00:15.4593759Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-10-08T21:00:15.4595777Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-10-08T21:00:15.4597801Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-10-08T21:00:15.4607704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-10-08T21:00:15.4683521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-10-08T21:00:15.4685874Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T21:00:15.4687863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-10-08T21:00:15.4689787Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-10-08T21:00:15.4691440Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-10-08T21:00:15.4693054Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-10-08T21:00:15.4694698Z inflating: build/custom_test_artifacts/custom-op-build/detect_rocm_version.cc 2024-10-08T21:00:15.4696230Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-10-08T21:00:15.4697700Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-10-08T21:00:15.4699107Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-10-08T21:00:15.4700050Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-10-08T21:00:15.4798190Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-10-08T21:00:15.4854677Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-10-08T21:00:15.4856207Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-10-08T21:00:15.4857512Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-10-08T21:00:15.4859032Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-10-08T21:00:15.4860671Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T21:00:15.4861545Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-10-08T21:00:15.4862420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T21:00:15.4863326Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T21:00:15.4864223Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T21:00:15.4865248Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T21:00:15.4866420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T21:00:15.4867352Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T21:00:15.4868268Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T21:00:15.4870001Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T21:00:15.4872071Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T21:00:15.4874114Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T21:00:15.4876141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T21:00:15.4878210Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T21:00:15.4880280Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T21:00:15.4882033Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-10-08T21:00:15.4883587Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-10-08T21:00:15.4885189Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-10-08T21:00:15.4886873Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-10-08T21:00:15.4888950Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-10-08T21:00:15.4891089Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-10-08T21:00:15.4893127Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-10-08T21:00:15.4895054Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-10-08T21:00:15.4897058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-10-08T21:00:15.4899041Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-10-08T21:00:15.4900339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-10-08T21:00:15.4901339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-10-08T21:00:15.4902340Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-10-08T21:00:15.4903430Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-10-08T21:00:15.4954315Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-10-08T21:00:15.4956010Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T21:00:15.4957938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-10-08T21:00:15.4960067Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-10-08T21:00:15.4961713Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-10-08T21:00:15.4963296Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-10-08T21:00:15.4964890Z inflating: build/custom_test_artifacts/jit-hook-build/detect_rocm_version.cc 2024-10-08T21:00:15.4966410Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-10-08T21:00:15.4967900Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-10-08T21:00:15.4969439Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-10-08T21:00:15.4971439Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-10-08T21:00:15.5005840Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-10-08T21:00:15.5007210Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-10-08T21:00:15.5008628Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-10-08T21:00:15.5010348Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-10-08T21:00:15.5012256Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-10-08T21:00:15.5014008Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-10-08T21:00:15.5015800Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-10-08T21:00:15.5017700Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-10-08T21:00:15.5019532Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-10-08T21:00:15.5020635Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-10-08T21:00:15.5021707Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-10-08T21:00:15.5022685Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-10-08T21:00:15.5023665Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-10-08T21:00:15.5024792Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-10-08T21:00:15.5025921Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-10-08T21:00:15.5027025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-10-08T21:00:15.5028151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-10-08T21:00:15.5030000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-10-08T21:00:15.5032193Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-10-08T21:00:15.5034087Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-10-08T21:00:15.5035749Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-10-08T21:00:15.5037481Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-10-08T21:00:15.5039296Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-10-08T21:00:15.5041333Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-10-08T21:00:15.5043612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-10-08T21:00:15.5045793Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-10-08T21:00:15.5048265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-10-08T21:00:15.5050499Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-10-08T21:00:15.5052634Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-10-08T21:00:15.5054773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-10-08T21:00:15.5056896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-10-08T21:00:15.5059325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-10-08T21:00:15.5060589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-10-08T21:00:15.5143178Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-10-08T21:00:15.5144272Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-10-08T21:00:15.5145386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-10-08T21:00:15.5146603Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-10-08T21:00:15.5147798Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-10-08T21:00:15.5149234Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-10-08T21:00:15.5151467Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-10-08T21:00:15.5153713Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-10-08T21:00:15.5155971Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-10-08T21:00:15.5158210Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-10-08T21:00:15.5160442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-10-08T21:00:15.5168075Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-10-08T21:00:15.5218570Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-10-08T21:00:15.5220269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-10-08T21:00:15.5221323Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-10-08T21:00:15.5222269Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-10-08T21:00:15.5223160Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-10-08T21:00:15.5224050Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-10-08T21:00:15.5224943Z inflating: build/custom_test_artifacts/custom-backend-build/detect_rocm_version.cc 2024-10-08T21:00:15.5225787Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-10-08T21:00:15.5226606Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-10-08T21:00:15.5227370Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-10-08T21:00:15.5228164Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-10-08T21:00:15.5314454Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-10-08T21:00:15.5352939Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-10-08T21:00:15.5354101Z creating: build/lib/ 2024-10-08T21:00:15.5432713Z inflating: build/lib/libprotobuf-lite.a 2024-10-08T21:00:15.5842629Z inflating: build/lib/libprotobuf.a 2024-10-08T21:00:15.6295292Z inflating: build/lib/libprotoc.a 2024-10-08T21:00:15.6303843Z inflating: build/lib/libpthreadpool.a 2024-10-08T21:00:15.6311054Z inflating: build/lib/libcpuinfo.a 2024-10-08T21:00:15.6318091Z inflating: build/lib/libcpuinfo_internals.a 2024-10-08T21:00:15.6319083Z inflating: build/lib/libclog.a 2024-10-08T21:00:15.6336108Z inflating: build/lib/libpytorch_qnnpack.a 2024-10-08T21:00:15.6337637Z inflating: build/lib/libnnpack_reference_layers.a 2024-10-08T21:00:15.6353826Z inflating: build/lib/libnnpack.a 2024-10-08T21:00:15.6360552Z inflating: build/lib/libittnotify.a 2024-10-08T21:00:15.6385289Z inflating: build/lib/libtensorpipe_uv.a 2024-10-08T21:00:15.6443587Z inflating: build/lib/libgtest.a 2024-10-08T21:00:15.6444502Z inflating: build/lib/libgtest_main.a 2024-10-08T21:00:15.6511084Z inflating: build/lib/libbenchmark.a 2024-10-08T21:00:15.6512067Z inflating: build/lib/libbenchmark_main.a 2024-10-08T21:00:15.6527739Z inflating: build/lib/libgmock.a 2024-10-08T21:00:15.6584637Z inflating: build/lib/libasmjit.a 2024-10-08T21:00:15.6585557Z inflating: build/lib/libgmock_main.a 2024-10-08T21:00:15.7725808Z inflating: build/lib/libfbgemm.a 2024-10-08T21:00:15.8246275Z inflating: build/lib/libtensorpipe.a 2024-10-08T21:00:15.8418075Z inflating: build/lib/libXNNPACK.a 2024-10-08T21:00:15.8519991Z inflating: build/lib/libgloo.a 2024-10-08T21:00:15.8875060Z inflating: build/lib/libgloo_hip.a 2024-10-08T21:00:15.8913272Z inflating: build/lib/libonnx_proto.a 2024-10-08T21:00:15.9538781Z inflating: build/lib/libonnx.a 2024-10-08T21:00:16.8885916Z inflating: build/lib/libdnnl.a 2024-10-08T21:00:16.8904590Z inflating: build/lib/libfmt.a 2024-10-08T21:00:16.9195726Z inflating: build/lib/libkineto.a 2024-10-08T21:00:16.9283616Z inflating: build/lib/libc10.so 2024-10-08T21:00:16.9284585Z inflating: build/lib/libtorch_global_deps.so 2024-10-08T21:00:16.9326292Z inflating: build/lib/libc10_hip.so 2024-10-08T21:00:16.9327257Z inflating: build/lib/libcaffe2_nvrtc.so 2024-10-08T21:00:19.1499754Z inflating: build/lib/libtorch_cpu.so 2024-10-08T21:00:19.1503795Z inflating: build/lib/libunbox_lib.a 2024-10-08T21:00:21.0169783Z inflating: build/lib/libtorch_hip.so 2024-10-08T21:00:21.0170731Z inflating: build/lib/libtorch.so 2024-10-08T21:00:21.0188490Z inflating: build/lib/libjitbackend_test.so 2024-10-08T21:00:21.0252173Z inflating: build/lib/libtorchbind_test.so 2024-10-08T21:00:21.0273959Z inflating: build/lib/libbackend_with_compiler.so 2024-10-08T21:00:21.0295953Z inflating: build/lib/libaoti_custom_ops.so 2024-10-08T21:00:21.0298956Z inflating: build/lib/libshm.so 2024-10-08T21:00:21.2083491Z inflating: build/lib/libtorch_python.so 2024-10-08T21:00:21.2113850Z inflating: build/lib/libnnapi_backend.so 2024-10-08T21:00:21.2114768Z creating: build/bin/ 2024-10-08T21:00:21.2115418Z creating: build/bin/CMakeFiles/ 2024-10-08T21:00:21.2116226Z inflating: build/bin/cmake_install.cmake 2024-10-08T21:00:21.2117113Z inflating: build/bin/CTestTestfile.cmake 2024-10-08T21:00:21.2512242Z inflating: build/bin/protoc-3.13.0.0 2024-10-08T21:00:21.2907952Z inflating: build/bin/protoc 2024-10-08T21:00:21.2952794Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-10-08T21:00:21.2998743Z inflating: build/bin/c10_DeviceGuard_test 2024-10-08T21:00:21.3044658Z inflating: build/bin/c10_Device_test 2024-10-08T21:00:21.3097132Z inflating: build/bin/c10_DispatchKeySet_test 2024-10-08T21:00:21.3140506Z inflating: build/bin/c10_StreamGuard_test 2024-10-08T21:00:21.3187937Z inflating: build/bin/c10_Scalar_test 2024-10-08T21:00:21.3233194Z inflating: build/bin/c10_SymInt_test 2024-10-08T21:00:21.3281516Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-10-08T21:00:21.3330974Z inflating: build/bin/c10_InlineStreamGuard_test 2024-10-08T21:00:21.3381140Z inflating: build/bin/c10_SizesAndStrides_test 2024-10-08T21:00:21.3424688Z inflating: build/bin/c10_ConstexprCrc_test 2024-10-08T21:00:21.3471885Z inflating: build/bin/c10_Bitset_test 2024-10-08T21:00:21.3535342Z inflating: build/bin/c10_cow_test 2024-10-08T21:00:21.3579825Z inflating: build/bin/c10_DeadlockDetection_test 2024-10-08T21:00:21.3625032Z inflating: build/bin/c10_Half_test 2024-10-08T21:00:21.3675519Z inflating: build/bin/c10_LeftRight_test 2024-10-08T21:00:21.3723483Z inflating: build/bin/c10_NetworkFlow_test 2024-10-08T21:00:21.3772302Z inflating: build/bin/c10_Metaprogramming_test 2024-10-08T21:00:21.3816652Z inflating: build/bin/c10_Synchronized_test 2024-10-08T21:00:21.3866148Z inflating: build/bin/c10_ThreadLocal_test 2024-10-08T21:00:21.3912161Z inflating: build/bin/c10_TypeIndex_test 2024-10-08T21:00:21.3958429Z inflating: build/bin/c10_accumulate_test 2024-10-08T21:00:21.4002525Z inflating: build/bin/c10_TypeTraits_test 2024-10-08T21:00:21.4047879Z inflating: build/bin/c10_TypeList_test 2024-10-08T21:00:21.4092978Z inflating: build/bin/c10_bit_cast_test 2024-10-08T21:00:21.4142503Z inflating: build/bin/c10_bfloat16_test 2024-10-08T21:00:21.4193277Z inflating: build/bin/c10_complex_math_test 2024-10-08T21:00:21.4242901Z inflating: build/bin/c10_complex_test 2024-10-08T21:00:21.4289894Z inflating: build/bin/c10_exception_test 2024-10-08T21:00:21.4334727Z inflating: build/bin/c10_flags_test 2024-10-08T21:00:21.4379436Z inflating: build/bin/c10_generic_math_test 2024-10-08T21:00:21.4424766Z inflating: build/bin/c10_irange_test 2024-10-08T21:00:21.4472657Z inflating: build/bin/c10_lazy_test 2024-10-08T21:00:21.4523523Z inflating: build/bin/c10_logging_test 2024-10-08T21:00:21.4672793Z inflating: build/bin/c10_intrusive_ptr_test 2024-10-08T21:00:21.4740068Z inflating: build/bin/c10_optional_test 2024-10-08T21:00:21.4788178Z inflating: build/bin/c10_registry_test 2024-10-08T21:00:21.4843982Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-10-08T21:00:21.4889899Z inflating: build/bin/c10_ssize_test 2024-10-08T21:00:21.5026117Z inflating: build/bin/c10_small_vector_test 2024-10-08T21:00:21.5072555Z inflating: build/bin/c10_string_util_test 2024-10-08T21:00:21.5117573Z inflating: build/bin/c10_tempfile_test 2024-10-08T21:00:21.5170418Z inflating: build/bin/c10_string_view_test 2024-10-08T21:00:21.5220543Z inflating: build/bin/c10_typeid_test 2024-10-08T21:00:21.5263797Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-10-08T21:00:21.5307622Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-10-08T21:00:21.5351364Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-10-08T21:00:21.5395211Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-10-08T21:00:21.5438726Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-10-08T21:00:21.5482563Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-10-08T21:00:21.5526167Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-10-08T21:00:21.5569967Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-10-08T21:00:21.5613201Z inflating: build/bin/c10_hip_HIPTest 2024-10-08T21:00:21.5958875Z inflating: build/bin/vec_test_all_types_AVX512 2024-10-08T21:00:21.6316711Z inflating: build/bin/vec_test_all_types_AVX2 2024-10-08T21:00:21.6652226Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-10-08T21:00:21.6698868Z inflating: build/bin/BackoffTest 2024-10-08T21:00:21.6746977Z inflating: build/bin/test_edge_op_registration 2024-10-08T21:00:21.6793991Z inflating: build/bin/FileStoreTest 2024-10-08T21:00:21.6843448Z inflating: build/bin/TCPStoreTest 2024-10-08T21:00:21.6891352Z inflating: build/bin/HashStoreTest 2024-10-08T21:00:21.6949577Z inflating: build/bin/ProcessGroupGlooTest 2024-10-08T21:00:21.6951225Z inflating: build/bin/example_allreduce 2024-10-08T21:00:21.7004762Z inflating: build/bin/static_runtime_bench 2024-10-08T21:00:21.7230583Z inflating: build/bin/static_runtime_test 2024-10-08T21:00:21.7296941Z inflating: build/bin/Dict_test 2024-10-08T21:00:21.7343115Z inflating: build/bin/Dimname_test 2024-10-08T21:00:21.7400664Z inflating: build/bin/MaybeOwned_test 2024-10-08T21:00:21.7444640Z inflating: build/bin/hip_optional_test 2024-10-08T21:00:21.7495489Z inflating: build/bin/NamedTensor_test 2024-10-08T21:00:21.7548280Z inflating: build/bin/apply_utils_test 2024-10-08T21:00:21.7600511Z inflating: build/bin/atest 2024-10-08T21:00:21.7656543Z inflating: build/bin/basic 2024-10-08T21:00:21.7705325Z inflating: build/bin/broadcast_test 2024-10-08T21:00:21.7750697Z inflating: build/bin/cpu_allocator_test 2024-10-08T21:00:21.7802562Z inflating: build/bin/cpu_generator_test 2024-10-08T21:00:21.7849895Z inflating: build/bin/cpu_profiling_allocator_test 2024-10-08T21:00:21.7933436Z inflating: build/bin/cpu_rng_test 2024-10-08T21:00:21.7977926Z inflating: build/bin/dispatch_key_set_test 2024-10-08T21:00:21.8022619Z inflating: build/bin/dlconvertor_test 2024-10-08T21:00:21.8074324Z inflating: build/bin/extension_backend_test 2024-10-08T21:00:21.8123559Z inflating: build/bin/half_test 2024-10-08T21:00:21.8208971Z inflating: build/bin/ivalue_test 2024-10-08T21:00:21.8253257Z inflating: build/bin/lazy_tensor_test 2024-10-08T21:00:21.8301762Z inflating: build/bin/math_kernel_test 2024-10-08T21:00:21.8349794Z inflating: build/bin/memory_format_test 2024-10-08T21:00:21.8397281Z inflating: build/bin/memory_overlapping_test 2024-10-08T21:00:21.8444325Z inflating: build/bin/mobile_memory_cleanup 2024-10-08T21:00:21.8494010Z inflating: build/bin/native_test 2024-10-08T21:00:21.8539094Z inflating: build/bin/operator_name_test 2024-10-08T21:00:21.8584952Z inflating: build/bin/operators_test 2024-10-08T21:00:21.8630976Z inflating: build/bin/packedtensoraccessor_test 2024-10-08T21:00:21.8690796Z inflating: build/bin/pow_test 2024-10-08T21:00:21.8741575Z inflating: build/bin/quantized_test 2024-10-08T21:00:21.8785984Z inflating: build/bin/reduce_ops_test 2024-10-08T21:00:21.8831361Z inflating: build/bin/reportMemoryUsage_test 2024-10-08T21:00:21.8881724Z inflating: build/bin/scalar_tensor_test 2024-10-08T21:00:21.8933940Z inflating: build/bin/scalar_test 2024-10-08T21:00:21.8979862Z inflating: build/bin/StorageUtils_test 2024-10-08T21:00:21.9026692Z inflating: build/bin/stride_properties_test 2024-10-08T21:00:21.9097434Z inflating: build/bin/tensor_iterator_test 2024-10-08T21:00:21.9145436Z inflating: build/bin/test_parallel 2024-10-08T21:00:21.9147150Z inflating: build/bin/thread_init_test 2024-10-08T21:00:21.9197479Z inflating: build/bin/type_ptr_test 2024-10-08T21:00:21.9250766Z inflating: build/bin/type_test 2024-10-08T21:00:21.9296944Z inflating: build/bin/undefined_tensor_test 2024-10-08T21:00:21.9297998Z inflating: build/bin/verify_api_visibility 2024-10-08T21:00:21.9360154Z inflating: build/bin/legacy_vmap_test 2024-10-08T21:00:21.9405990Z inflating: build/bin/weakref_test 2024-10-08T21:00:21.9451734Z inflating: build/bin/wrapdim_test 2024-10-08T21:00:21.9497671Z inflating: build/bin/xla_tensor_test 2024-10-08T21:00:21.9551213Z inflating: build/bin/IListRef_test 2024-10-08T21:00:21.9646392Z inflating: build/bin/List_test 2024-10-08T21:00:21.9756003Z inflating: build/bin/kernel_function_legacy_test 2024-10-08T21:00:21.9815517Z inflating: build/bin/KernelFunction_test 2024-10-08T21:00:21.9902421Z inflating: build/bin/kernel_function_test 2024-10-08T21:00:21.9995318Z inflating: build/bin/kernel_lambda_test 2024-10-08T21:00:22.0110460Z inflating: build/bin/kernel_lambda_legacy_test 2024-10-08T21:00:22.0164899Z inflating: build/bin/kernel_stackbased_test 2024-10-08T21:00:22.0251257Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-10-08T21:00:22.0296748Z inflating: build/bin/CppSignature_test 2024-10-08T21:00:22.0340294Z inflating: build/bin/op_allowlist_test 2024-10-08T21:00:22.0612791Z inflating: build/bin/op_registration_test 2024-10-08T21:00:22.0662408Z inflating: build/bin/backend_fallback_test 2024-10-08T21:00:22.0719048Z inflating: build/bin/inline_container_test 2024-10-08T21:00:22.0763226Z inflating: build/bin/hip_complex_math_test 2024-10-08T21:00:22.0809810Z inflating: build/bin/hip_apply_test 2024-10-08T21:00:22.0853427Z inflating: build/bin/hip_complex_test 2024-10-08T21:00:22.0896805Z inflating: build/bin/hip_distributions_test 2024-10-08T21:00:22.0940241Z inflating: build/bin/hip_generator_test 2024-10-08T21:00:22.0983644Z inflating: build/bin/hip_half_test 2024-10-08T21:00:22.1028853Z inflating: build/bin/hip_dlconvertor_test 2024-10-08T21:00:22.1072328Z inflating: build/bin/hip_integer_divider_test 2024-10-08T21:00:22.1115761Z inflating: build/bin/hip_packedtensoraccessor_test 2024-10-08T21:00:22.1159866Z inflating: build/bin/hip_vectorized_test 2024-10-08T21:00:22.1674308Z inflating: build/bin/test_jit 2024-10-08T21:00:22.1687103Z inflating: build/bin/tutorial_tensorexpr 2024-10-08T21:00:22.2394271Z inflating: build/bin/test_tensorexpr 2024-10-08T21:00:22.2443253Z inflating: build/bin/test_dist_autograd 2024-10-08T21:00:22.2505042Z inflating: build/bin/test_cpp_rpc 2024-10-08T21:00:22.2506480Z inflating: build/bin/parallel_benchmark 2024-10-08T21:00:22.3561889Z inflating: build/bin/test_api 2024-10-08T21:00:22.3621889Z inflating: build/bin/test_mobile_nnc 2024-10-08T21:00:22.3629945Z inflating: build/bin/aot_model_compiler_test 2024-10-08T21:00:22.3940270Z inflating: build/bin/test_lazy 2024-10-08T21:00:22.3944098Z inflating: build/bin/torch_shm_manager 2024-10-08T21:00:22.3945029Z creating: .additional_ci_files/ 2024-10-08T21:00:22.4015982Z inflating: .additional_ci_files/test-times.json 2024-10-08T21:00:22.4297716Z inflating: .additional_ci_files/test-class-times.json 2024-10-08T21:00:22.4340945Z ##[group]Run rm artifacts.zip 2024-10-08T21:00:22.4341315Z rm artifacts.zip 2024-10-08T21:00:22.4369846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:22.4370315Z env: 2024-10-08T21:00:22.4370576Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:22.4370991Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:22.4371714Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:22.4372387Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:22.4372749Z AWS_REGION: us-east-1 2024-10-08T21:00:22.4373150Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:22.4373625Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:22.4380614Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:22.4380940Z ##[endgroup] 2024-10-08T21:00:22.6415626Z ##[group]Run df -H 2024-10-08T21:00:22.6415919Z df -H 2024-10-08T21:00:22.6458591Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:22.6459497Z env: 2024-10-08T21:00:22.6460002Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:22.6460775Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:22.6462406Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:22.6463942Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:22.6464746Z AWS_REGION: us-east-1 2024-10-08T21:00:22.6465622Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:22.6466702Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:22.6481512Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:22.6482130Z ##[endgroup] 2024-10-08T21:00:22.6580731Z Filesystem Size Used Avail Use% Mounted on 2024-10-08T21:00:22.6581319Z tmpfs 14G 17M 14G 1% /run 2024-10-08T21:00:22.6582009Z /dev/mapper/ubuntu--vg-ubuntu--lv 1.9T 827G 964G 47% / 2024-10-08T21:00:22.6582582Z tmpfs 68G 8.2k 68G 1% /dev/shm 2024-10-08T21:00:22.6583119Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-10-08T21:00:22.6583660Z /dev/sda2 2.1G 339M 1.6G 18% /boot 2024-10-08T21:00:22.6584270Z /dev/sda1 1.2G 6.4M 1.2G 1% /boot/efi 2024-10-08T21:00:22.6585163Z tmpfs 14G 17k 14G 1% /run/user/1001 2024-10-08T21:00:22.6644559Z Prepare all required actions 2024-10-08T21:00:22.6645398Z Getting action download info 2024-10-08T21:00:23.1001166Z ##[group]Run ./.github/actions/download-td-artifacts 2024-10-08T21:00:23.1002042Z with: 2024-10-08T21:00:23.1002501Z env: 2024-10-08T21:00:23.1002993Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:23.1003740Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:23.1005132Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:23.1006417Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:23.1007115Z AWS_REGION: us-east-1 2024-10-08T21:00:23.1007863Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:23.1008988Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:23.1022621Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:23.1023242Z ##[endgroup] 2024-10-08T21:00:23.1090880Z ##[group]Run seemethere/download-artifact-s3@v4 2024-10-08T21:00:23.1091695Z with: 2024-10-08T21:00:23.1092189Z name: td_results 2024-10-08T21:00:23.1092839Z s3-bucket: gha-artifacts 2024-10-08T21:00:23.1093483Z region: us-east-1 2024-10-08T21:00:23.1094018Z env: 2024-10-08T21:00:23.1094514Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:23.1095267Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:23.1096627Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:23.1097920Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:23.1098608Z AWS_REGION: us-east-1 2024-10-08T21:00:23.1099343Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:23.1100255Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:23.1113833Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:23.1114457Z ##[endgroup] 2024-10-08T21:00:23.5044754Z (node:2519168) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-10-08T21:00:23.5045460Z 2024-10-08T21:00:23.5045709Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-10-08T21:00:23.5046452Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-10-08T21:00:23.5047290Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-10-08T21:00:23.7681262Z Found 1 objects with prefix pytorch/pytorch/11242664656/td_results/ 2024-10-08T21:00:23.7682274Z Starting download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-10-08T21:00:24.1991784Z Finished download (1/1): /home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-10-08T21:00:24.2002797Z Artifact download has finished successfully 2024-10-08T21:00:24.2514793Z ##[group]Run mkdir -p .additional_ci_files 2024-10-08T21:00:24.2515700Z mkdir -p .additional_ci_files 2024-10-08T21:00:24.2516726Z mv td_results.json .additional_ci_files/td_results.json 2024-10-08T21:00:24.2570839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:24.2571759Z env: 2024-10-08T21:00:24.2572259Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:24.2573035Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:24.2574410Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:24.2576111Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:24.2576797Z AWS_REGION: us-east-1 2024-10-08T21:00:24.2577591Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:24.2578521Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:24.2592770Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:24.2593513Z ##[endgroup] 2024-10-08T21:00:24.2761173Z ##[group]Run .github/scripts/parse_ref.py 2024-10-08T21:00:24.2761564Z .github/scripts/parse_ref.py 2024-10-08T21:00:24.2783738Z shell: /usr/bin/bash -e {0} 2024-10-08T21:00:24.2784008Z env: 2024-10-08T21:00:24.2784223Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:24.2784538Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:24.2785107Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:24.2785645Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:24.2785931Z AWS_REGION: us-east-1 2024-10-08T21:00:24.2786242Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:24.2786629Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:24.2792232Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:24.2792488Z ##[endgroup] 2024-10-08T21:00:24.3068248Z Prepare all required actions 2024-10-08T21:00:24.3144192Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-10-08T21:00:24.3144967Z with: 2024-10-08T21:00:24.3145793Z github-token: *** 2024-10-08T21:00:24.3146319Z env: 2024-10-08T21:00:24.3146789Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:24.3147483Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:24.3148787Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:24.3149995Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:24.3150638Z AWS_REGION: us-east-1 2024-10-08T21:00:24.3151351Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:24.3152425Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:24.3166368Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:24.3166981Z ##[endgroup] 2024-10-08T21:00:24.3202143Z ##[group]Run set -eux 2024-10-08T21:00:24.3202758Z set -eux 2024-10-08T21:00:24.3203853Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-10-08T21:00:24.3253809Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:24.3254715Z env: 2024-10-08T21:00:24.3255213Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:24.3255961Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:24.3257343Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:24.3258621Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:24.3259301Z AWS_REGION: us-east-1 2024-10-08T21:00:24.3260038Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:24.3260946Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:24.3274962Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:24.3275780Z GITHUB_TOKEN: *** 2024-10-08T21:00:24.3276304Z ##[endgroup] 2024-10-08T21:00:24.3352304Z + python3 .github/scripts/get_workflow_job_id.py 11242664656 pytorch-rocm-hw-35 2024-10-08T21:00:25.3560408Z setting job-id=31258392692 2024-10-08T21:00:25.3561880Z setting job-name=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:25.3959840Z Prepare all required actions 2024-10-08T21:00:25.3960691Z Getting action download info 2024-10-08T21:00:25.5761521Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-10-08T21:00:26.0616729Z ##[group]Run ./.github/actions/filter-test-configs 2024-10-08T21:00:26.0617080Z with: 2024-10-08T21:00:26.0617418Z github-token: *** 2024-10-08T21:00:26.0618341Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-10-08T21:00:26.0619493Z job-name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:26.0620187Z env: 2024-10-08T21:00:26.0620405Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:26.0620727Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:26.0621338Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:26.0621873Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:26.0622161Z AWS_REGION: us-east-1 2024-10-08T21:00:26.0622460Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:26.0622839Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:26.0628423Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:26.0628686Z ##[endgroup] 2024-10-08T21:00:26.0663224Z ##[group]Run nick-fields/retry@v3.0.0 2024-10-08T21:00:26.0663533Z with: 2024-10-08T21:00:26.0663735Z shell: bash 2024-10-08T21:00:26.0663968Z timeout_minutes: 10 2024-10-08T21:00:26.0664218Z max_attempts: 5 2024-10-08T21:00:26.0664462Z retry_wait_seconds: 30 2024-10-08T21:00:26.0665315Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-10-08T21:00:26.0666209Z polling_interval_seconds: 1 2024-10-08T21:00:26.0666502Z warning_on_retry: true 2024-10-08T21:00:26.0666773Z continue_on_error: false 2024-10-08T21:00:26.0667034Z env: 2024-10-08T21:00:26.0667256Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:26.0667580Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:26.0668153Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:26.0668690Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:26.0668985Z AWS_REGION: us-east-1 2024-10-08T21:00:26.0669301Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:26.0669695Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:26.0675289Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:26.0675659Z GITHUB_TOKEN: *** 2024-10-08T21:00:26.0675906Z ##[endgroup] 2024-10-08T21:00:26.1332415Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-10-08T21:00:26.3799429Z Defaulting to user installation because normal site-packages is not writeable 2024-10-08T21:00:26.4534548Z Requirement already satisfied: requests==2.27.1 in /home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-10-08T21:00:26.4537972Z Requirement already satisfied: pyyaml==6.0.1 in /home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-10-08T21:00:26.4627505Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-10-08T21:00:26.4635529Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-10-08T21:00:26.4641003Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-10-08T21:00:26.4649414Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-10-08T21:00:27.1341467Z Command completed after 1 attempt(s). 2024-10-08T21:00:27.1475165Z ##[group]Run set -x 2024-10-08T21:00:27.1475808Z set -x 2024-10-08T21:00:27.1476362Z  2024-10-08T21:00:27.1477924Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-10-08T21:00:27.1479168Z # in runner workspace 2024-10-08T21:00:27.1480162Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-10-08T21:00:27.1533080Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:27.1534019Z env: 2024-10-08T21:00:27.1534559Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:27.1535329Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:27.1536722Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:27.1538020Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:27.1538714Z AWS_REGION: us-east-1 2024-10-08T21:00:27.1539484Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:27.1540872Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:27.1554464Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:27.1555097Z ##[endgroup] 2024-10-08T21:00:27.1641795Z + python3 /home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-10-08T21:00:27.1875519Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-10-08T21:00:27.1876517Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-10-08T21:00:27.1877362Z echo "Job name: ${JOB_NAME}" 2024-10-08T21:00:27.1878080Z  2024-10-08T21:00:27.1879078Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-10-08T21:00:27.1880312Z # in runner workspace 2024-10-08T21:00:27.1881362Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-10-08T21:00:27.1882540Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-10-08T21:00:27.1883380Z  --job-name "${JOB_NAME}" \ 2024-10-08T21:00:27.1885910Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-10-08T21:00:27.1888449Z  --selected-test-configs "" \ 2024-10-08T21:00:27.1889459Z  --pr-number "${PR_NUMBER}" \ 2024-10-08T21:00:27.1890216Z  --tag "${TAG}" \ 2024-10-08T21:00:27.1890926Z  --event-name "${EVENT_NAME}" \ 2024-10-08T21:00:27.1891725Z  --schedule "${SCHEDULE}" \ 2024-10-08T21:00:27.1892503Z  --branch "${HEAD_BRANCH}" 2024-10-08T21:00:27.1946602Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:27.1947499Z env: 2024-10-08T21:00:27.1948008Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:27.1948763Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:27.1950125Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:27.1951401Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:27.1952107Z AWS_REGION: us-east-1 2024-10-08T21:00:27.1952866Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:27.1953784Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:27.1967462Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:27.1968360Z GITHUB_TOKEN: *** 2024-10-08T21:00:27.1969510Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:27.1970660Z PR_NUMBER: 2024-10-08T21:00:27.1971224Z TAG: ciflow/trunk/134247 2024-10-08T21:00:27.1971860Z EVENT_NAME: push 2024-10-08T21:00:27.1972416Z SCHEDULE: 2024-10-08T21:00:27.1972927Z HEAD_BRANCH: 2024-10-08T21:00:27.1973467Z ##[endgroup] 2024-10-08T21:00:27.2060308Z Workflow: trunk 2024-10-08T21:00:27.2061662Z Job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:27.5418014Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-10-08T21:00:28.1030940Z ##[group]Run echo "Filtered matrix:" 2024-10-08T21:00:28.1031866Z echo "Filtered matrix:" 2024-10-08T21:00:28.1034706Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-10-08T21:00:28.1037067Z  2024-10-08T21:00:28.1037559Z echo 2024-10-08T21:00:28.1038245Z echo "Is the current job unstable? False" 2024-10-08T21:00:28.1039062Z  2024-10-08T21:00:28.1039579Z echo 2024-10-08T21:00:28.1040219Z echo "Is keep-going label set? False" 2024-10-08T21:00:28.1041007Z  2024-10-08T21:00:28.1041479Z echo 2024-10-08T21:00:28.1042049Z echo "Renabled issues? " 2024-10-08T21:00:28.1096585Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:28.1098162Z env: 2024-10-08T21:00:28.1098719Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:28.1099482Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:28.1100894Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:28.1102198Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:28.1102896Z AWS_REGION: us-east-1 2024-10-08T21:00:28.1103826Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:28.1104967Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:28.1118919Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:28.1119560Z ##[endgroup] 2024-10-08T21:00:28.1206266Z Filtered matrix: 2024-10-08T21:00:28.1208574Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-10-08T21:00:28.1210786Z 2024-10-08T21:00:28.1211091Z Is the current job unstable? False 2024-10-08T21:00:28.1211656Z 2024-10-08T21:00:28.1212181Z Is keep-going label set? False 2024-10-08T21:00:28.1212678Z 2024-10-08T21:00:28.1212899Z Renabled issues? 2024-10-08T21:00:28.1303358Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-10-08T21:00:28.1304684Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-10-08T21:00:28.1355538Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:00:28.1356464Z env: 2024-10-08T21:00:28.1356961Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:28.1357724Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:28.1359130Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:28.1360414Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:28.1361100Z AWS_REGION: us-east-1 2024-10-08T21:00:28.1361837Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:28.1362755Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:28.1376969Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:28.1377725Z JOB_TIMEOUT: 300 2024-10-08T21:00:28.1378377Z ##[endgroup] 2024-10-08T21:00:28.1547712Z ##[group]Run set -x 2024-10-08T21:00:28.1548159Z set -x 2024-10-08T21:00:28.1548483Z  2024-10-08T21:00:28.1548832Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-10-08T21:00:28.1549388Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-10-08T21:00:28.1549958Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-10-08T21:00:28.1550481Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-10-08T21:00:28.1550907Z else 2024-10-08T21:00:28.1551267Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-10-08T21:00:28.1551687Z fi 2024-10-08T21:00:28.1551969Z  2024-10-08T21:00:28.1552440Z # detached container should get cleaned up by teardown_ec2_linux 2024-10-08T21:00:28.1553623Z # TODO: Stop building test binaries as part of the build phase 2024-10-08T21:00:28.1555099Z # Used for GPU_FLAG since that doesn't play nice 2024-10-08T21:00:28.1556285Z # shellcheck disable=SC2086,SC2090 2024-10-08T21:00:28.1557160Z container_name=$(docker run \ 2024-10-08T21:00:28.1557948Z  ${GPU_FLAG:-} \ 2024-10-08T21:00:28.1558644Z  -e BUILD_ENVIRONMENT \ 2024-10-08T21:00:28.1559379Z  -e PR_NUMBER \ 2024-10-08T21:00:28.1560055Z  -e GITHUB_ACTIONS \ 2024-10-08T21:00:28.1560780Z  -e GITHUB_REPOSITORY \ 2024-10-08T21:00:28.1561540Z  -e GITHUB_WORKFLOW \ 2024-10-08T21:00:28.1562254Z  -e GITHUB_JOB \ 2024-10-08T21:00:28.1562921Z  -e GITHUB_RUN_ID \ 2024-10-08T21:00:28.1563641Z  -e GITHUB_RUN_NUMBER \ 2024-10-08T21:00:28.1564393Z  -e GITHUB_RUN_ATTEMPT \ 2024-10-08T21:00:28.1565140Z  -e JOB_ID \ 2024-10-08T21:00:28.1565791Z  -e JOB_NAME \ 2024-10-08T21:00:28.1566453Z  -e BRANCH \ 2024-10-08T21:00:28.1567080Z  -e SHA1 \ 2024-10-08T21:00:28.1568162Z  -e AWS_DEFAULT_REGION \ 2024-10-08T21:00:28.1569065Z  -e IN_WHEEL_TEST \ 2024-10-08T21:00:28.1569784Z  -e SHARD_NUMBER \ 2024-10-08T21:00:28.1570478Z  -e TEST_CONFIG \ 2024-10-08T21:00:28.1571171Z  -e NUM_TEST_SHARDS \ 2024-10-08T21:00:28.1571912Z  -e REENABLED_ISSUES \ 2024-10-08T21:00:28.1572750Z  -e CONTINUE_THROUGH_ERROR \ 2024-10-08T21:00:28.1573687Z  -e VERBOSE_TEST_LOGS \ 2024-10-08T21:00:28.1574571Z  -e TEST_SHOWLOCALS \ 2024-10-08T21:00:28.1575436Z  -e NO_TEST_TIMEOUT \ 2024-10-08T21:00:28.1576242Z  -e NO_TD \ 2024-10-08T21:00:28.1577094Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-10-08T21:00:28.1577954Z  -e SCCACHE_BUCKET \ 2024-10-08T21:00:28.1578464Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-10-08T21:00:28.1578979Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-10-08T21:00:28.1579495Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-10-08T21:00:28.1579973Z  -e TESTS_TO_INCLUDE \ 2024-10-08T21:00:28.1580451Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-10-08T21:00:28.1580981Z  --ulimit stack=10485760:83886080 \ 2024-10-08T21:00:28.1581414Z  --ulimit core=0 \ 2024-10-08T21:00:28.1581831Z  --security-opt seccomp=unconfined \ 2024-10-08T21:00:28.1582286Z  --cap-add=SYS_PTRACE \ 2024-10-08T21:00:28.1582685Z  --shm-size="8g" \ 2024-10-08T21:00:28.1583031Z  --tty \ 2024-10-08T21:00:28.1583353Z  --detach \ 2024-10-08T21:00:28.1583706Z  --name="${container_name}" \ 2024-10-08T21:00:28.1584128Z  --user jenkins \ 2024-10-08T21:00:28.1584620Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-10-08T21:00:28.1585182Z  -w /var/lib/jenkins/workspace \ 2024-10-08T21:00:28.1585629Z  "${DOCKER_IMAGE}" 2024-10-08T21:00:28.1585998Z ) 2024-10-08T21:00:28.1586352Z # save container name for later step 2024-10-08T21:00:28.1587118Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-10-08T21:00:28.1588102Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-10-08T21:00:28.1589395Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-10-08T21:00:28.1629390Z shell: /usr/bin/bash -e {0} 2024-10-08T21:00:28.1629785Z env: 2024-10-08T21:00:28.1630069Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:00:28.1630496Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:00:28.1631222Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:00:28.1631908Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:00:28.1632291Z AWS_REGION: us-east-1 2024-10-08T21:00:28.1632709Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:00:28.1633220Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:00:28.1646774Z AWS_SESSION_TOKEN: *** 2024-10-08T21:00:28.1647564Z BUILD_ENVIRONMENT: linux-focal-rocm6.2-py3.10 2024-10-08T21:00:28.1648405Z PR_NUMBER: 2024-10-08T21:00:28.1649122Z GITHUB_REPOSITORY: pytorch/pytorch 2024-10-08T21:00:28.1649906Z GITHUB_WORKFLOW: trunk 2024-10-08T21:00:28.1650542Z GITHUB_JOB: test 2024-10-08T21:00:28.1651132Z GITHUB_RUN_ID: 11242664656 2024-10-08T21:00:28.1651812Z GITHUB_RUN_NUMBER: 98699 2024-10-08T21:00:28.1652477Z GITHUB_RUN_ATTEMPT: 1 2024-10-08T21:00:28.1653095Z JOB_ID: 31258392692 2024-10-08T21:00:28.1654134Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:28.1655288Z BRANCH: 2024-10-08T21:00:28.1655949Z SHA1: 47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:28.1656826Z CONTINUE_THROUGH_ERROR: False 2024-10-08T21:00:28.1657561Z VERBOSE_TEST_LOGS: False 2024-10-08T21:00:28.1658232Z TEST_SHOWLOCALS: False 2024-10-08T21:00:28.1659247Z NO_TEST_TIMEOUT: False 2024-10-08T21:00:28.1659877Z NO_TD: False 2024-10-08T21:00:28.1660469Z TEST_CONFIG: distributed 2024-10-08T21:00:28.1661172Z SHARD_NUMBER: 1 2024-10-08T21:00:28.1661747Z NUM_TEST_SHARDS: 1 2024-10-08T21:00:28.1662361Z REENABLED_ISSUES: 2024-10-08T21:00:28.1663261Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-10-08T21:00:28.1665527Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T21:00:28.1667932Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-10-08T21:00:28.1668622Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-10-08T21:00:28.1669066Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-10-08T21:00:28.1670162Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-10-08T21:00:28.1671255Z ##[endgroup] 2024-10-08T21:00:28.1748018Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-10-08T21:00:28.1748740Z + [[ linux-focal-rocm6.2-py3.10 == *onnx* ]] 2024-10-08T21:00:28.1749211Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-10-08T21:00:28.1765284Z +++ nproc --ignore=2 2024-10-08T21:00:28.1788344Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=62 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_11242664656 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:46f3eb402e47f9fae93ec1dc1f6772950f4498aa 2024-10-08T21:00:32.4219716Z + container_name=639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:00:32.4220643Z + echo CONTAINER_NAME=639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:00:32.4227529Z + docker exec -t 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-10-08T21:00:43.1320236Z Processing ./dist/torch-2.6.0a0+git47952c0-cp310-cp310-linux_x86_64.whl 2024-10-08T21:00:43.7426880Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (3.13.1) 2024-10-08T21:00:43.7428564Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (4.12.2) 2024-10-08T21:00:43.7430657Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (2.8.8) 2024-10-08T21:00:43.7432977Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (3.1.4) 2024-10-08T21:00:43.7434402Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (2024.6.1) 2024-10-08T21:00:43.7439781Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git47952c0) (1.13.1) 2024-10-08T21:00:43.7453285Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git47952c0) (1.3.0) 2024-10-08T21:00:43.7753852Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git47952c0) (2.1.5) 2024-10-08T21:00:44.0306092Z Installing collected packages: torch 2024-10-08T21:00:54.9588058Z Successfully installed torch-2.6.0a0+git47952c0 2024-10-08T21:00:55.0126827Z + export TERM=vt100 2024-10-08T21:00:55.0132846Z + TERM=vt100 2024-10-08T21:00:55.0134600Z ++ dirname .ci/pytorch/test.sh 2024-10-08T21:00:55.0148790Z + source .ci/pytorch/common.sh 2024-10-08T21:00:55.0153937Z +++ dirname .ci/pytorch/common.sh 2024-10-08T21:00:55.0163802Z ++ source .ci/pytorch/common_utils.sh 2024-10-08T21:00:55.0164874Z +++ declare -f -t trap_add 2024-10-08T21:00:55.0172161Z ++ set -ex 2024-10-08T21:00:55.0172625Z ++ [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-10-08T21:00:55.0173095Z ++ unset HIP_PLATFORM 2024-10-08T21:00:55.0177913Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-10-08T21:00:55.0178410Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-10-08T21:00:55.0178802Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-10-08T21:00:55.0179168Z ++ HSAKMT_DEBUG_LEVEL=4 2024-10-08T21:00:55.0179529Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-10-08T21:00:55.0179965Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-10-08T21:00:55.0180662Z ++ BUILD_TEST_LIBTORCH=0 2024-10-08T21:00:55.0181179Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-10-08T21:00:55.0181686Z + echo 'Environment variables:' 2024-10-08T21:00:55.0182023Z Environment variables: 2024-10-08T21:00:55.0182301Z + env 2024-10-08T21:00:55.0189216Z INSTALLED_DB=yes 2024-10-08T21:00:55.0190254Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T21:00:55.0190916Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-10-08T21:00:55.0191430Z CONTINUE_THROUGH_ERROR=False 2024-10-08T21:00:55.0191969Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-10-08T21:00:55.0192470Z HOSTNAME=639ea538f71e 2024-10-08T21:00:55.0197388Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0198216Z GITHUB_ACTION=__self 2024-10-08T21:00:55.0198545Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-10-08T21:00:55.0198939Z GITHUB_RUN_NUMBER=98699 2024-10-08T21:00:55.0199209Z TEST_CONFIG=distributed 2024-10-08T21:00:55.0199492Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-10-08T21:00:55.0199833Z AWS_DEFAULT_REGION=us-east-1 2024-10-08T21:00:55.0200216Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-10-08T21:00:55.0200543Z GITHUB_REF_TYPE=tag 2024-10-08T21:00:55.0210241Z *** 2024-10-08T21:00:55.0210546Z GITHUB_REPOSITORY_ID=65600975 2024-10-08T21:00:55.0210858Z GITHUB_ACTIONS=true 2024-10-08T21:00:55.0211146Z SHA1=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0211534Z GITHUB_SHA=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0212112Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/134247 2024-10-08T21:00:55.0212659Z VERBOSE_TEST_LOGS=False 2024-10-08T21:00:55.0212957Z GITHUB_REF=refs/tags/ciflow/trunk/134247 2024-10-08T21:00:55.0213279Z SHARD_NUMBER=1 2024-10-08T21:00:55.0213521Z GITHUB_REF_PROTECTED=false 2024-10-08T21:00:55.0213790Z HOME=/var/lib/jenkins 2024-10-08T21:00:55.0214084Z GITHUB_API_URL=https://api.github.com 2024-10-08T21:00:55.0214424Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-10-08T21:00:55.0214769Z LANG=C.UTF-8 2024-10-08T21:00:55.0215001Z PYTORCH_TEST_WITH_ROCM=1 2024-10-08T21:00:55.0215264Z NUM_TEST_SHARDS=1 2024-10-08T21:00:55.0215919Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0216761Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:55.0217237Z MAGMA_HOME=/opt/rocm/magma 2024-10-08T21:00:55.0217893Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0218748Z HSAKMT_DEBUG_LEVEL=4 2024-10-08T21:00:55.0219287Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-10-08T21:00:55.0219780Z GITHUB_EVENT_NAME=push 2024-10-08T21:00:55.0220028Z GITHUB_RUN_ID=11242664656 2024-10-08T21:00:55.0220725Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0221413Z GITHUB_ACTOR=pytorch-bot[bot] 2024-10-08T21:00:55.0221678Z PR_NUMBER= 2024-10-08T21:00:55.0221888Z GITHUB_RUN_ATTEMPT=1 2024-10-08T21:00:55.0222140Z ANACONDA_PYTHON_VERSION=3.10 2024-10-08T21:00:55.0222480Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-10-08T21:00:55.0222883Z TERM=vt100 2024-10-08T21:00:55.0223135Z INSTALLED_VISION=yes 2024-10-08T21:00:55.0223407Z BRANCH= 2024-10-08T21:00:55.0224296Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-10-08T21:00:55.0225466Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-10-08T21:00:55.0226032Z GITHUB_SERVER_URL=https://github.com 2024-10-08T21:00:55.0226351Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-10-08T21:00:55.0226637Z REENABLED_ISSUES= 2024-10-08T21:00:55.0226858Z SHLVL=1 2024-10-08T21:00:55.0227066Z MAX_JOBS=62 2024-10-08T21:00:55.0227282Z GITHUB_ACTOR_ID=54816060 2024-10-08T21:00:55.0227633Z GITHUB_WORKFLOW_SHA=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0228036Z GITHUB_REF_NAME=ciflow/trunk/134247 2024-10-08T21:00:55.0228338Z ROCM_PATH=/opt/rocm 2024-10-08T21:00:55.0228770Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-10-08T21:00:55.0229202Z GITHUB_JOB=test 2024-10-08T21:00:55.0229427Z NO_TEST_TIMEOUT=False 2024-10-08T21:00:55.0229690Z GITHUB_REPOSITORY=pytorch/pytorch 2024-10-08T21:00:55.0229996Z LC_ALL=C.UTF-8 2024-10-08T21:00:55.0230217Z GITHUB_RETENTION_DAYS=90 2024-10-08T21:00:55.0230485Z GITHUB_ACTION_REPOSITORY= 2024-10-08T21:00:55.0231586Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T21:00:55.0232562Z GITHUB_BASE_REF= 2024-10-08T21:00:55.0232780Z CI=true 2024-10-08T21:00:55.0233008Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-10-08T21:00:55.0233296Z GITHUB_REPOSITORY_OWNER=pytorch 2024-10-08T21:00:55.0233574Z JOB_ID=31258392692 2024-10-08T21:00:55.0233806Z INSTALLED_PROTOBUF=yes 2024-10-08T21:00:55.0234056Z GITHUB_HEAD_REF= 2024-10-08T21:00:55.0234296Z GITHUB_ACTION_REF= 2024-10-08T21:00:55.0234637Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-10-08T21:00:55.0234985Z TEST_SHOWLOCALS=False 2024-10-08T21:00:55.0235229Z GITHUB_WORKFLOW=trunk 2024-10-08T21:00:55.0235480Z DEBIAN_FRONTEND=noninteractive 2024-10-08T21:00:55.0236180Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0236808Z NO_TD=False 2024-10-08T21:00:55.0237024Z OLDPWD=/var/lib/jenkins 2024-10-08T21:00:55.0237271Z _=/usr/bin/env 2024-10-08T21:00:55.0237622Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-10-08T21:00:55.0384582Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-10-08T21:00:55.0385288Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:00:55.0385939Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-10-08T21:00:55.0386635Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-10-08T21:00:55.0387183Z + BUILD_DIR=build 2024-10-08T21:00:55.0387486Z + BUILD_RENAMED_DIR=build_renamed 2024-10-08T21:00:55.0387838Z + BUILD_BIN_DIR=build/bin 2024-10-08T21:00:55.0388152Z + SHARD_NUMBER=1 2024-10-08T21:00:55.0388394Z + NUM_TEST_SHARDS=1 2024-10-08T21:00:55.0388837Z + export VALGRIND=ON 2024-10-08T21:00:55.0389082Z + VALGRIND=ON 2024-10-08T21:00:55.0390217Z + [[ linux-focal-rocm6.2-py3.10 == *clang9* ]] 2024-10-08T21:00:55.0390555Z + [[ 0 == \1 ]] 2024-10-08T21:00:55.0390806Z + [[ False == \1 ]] 2024-10-08T21:00:55.0391125Z + [[ linux-focal-rocm6.2-py3.10 != *bazel* ]] 2024-10-08T21:00:55.0391480Z ++ realpath build/custom_test_artifacts 2024-10-08T21:00:55.0412251Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-10-08T21:00:55.0413443Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-10-08T21:00:55.0414289Z + echo 'Setting INCLUDE_CLAUSE' 2024-10-08T21:00:55.0414584Z Setting INCLUDE_CLAUSE 2024-10-08T21:00:55.0415592Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-10-08T21:00:55.0416677Z + echo 'Environment variables' 2024-10-08T21:00:55.0417007Z Environment variables 2024-10-08T21:00:55.0417248Z + env 2024-10-08T21:00:55.0426242Z INSTALLED_DB=yes 2024-10-08T21:00:55.0426909Z GITHUB_WORKSPACE=/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T21:00:55.0427534Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-10-08T21:00:55.0427966Z CONTINUE_THROUGH_ERROR=False 2024-10-08T21:00:55.0428432Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-10-08T21:00:55.0428864Z HOSTNAME=639ea538f71e 2024-10-08T21:00:55.0429765Z GITHUB_PATH=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0430546Z GITHUB_ACTION=__self 2024-10-08T21:00:55.0430872Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-10-08T21:00:55.0431235Z GITHUB_RUN_NUMBER=98699 2024-10-08T21:00:55.0431544Z TEST_CONFIG=distributed 2024-10-08T21:00:55.0431917Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-10-08T21:00:55.0432352Z AWS_DEFAULT_REGION=us-east-1 2024-10-08T21:00:55.0434807Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-10-08T21:00:55.0435226Z GITHUB_REF_TYPE=tag 2024-10-08T21:00:55.0435568Z *** 2024-10-08T21:00:55.0435826Z GITHUB_REPOSITORY_ID=65600975 2024-10-08T21:00:55.0436168Z GITHUB_ACTIONS=true 2024-10-08T21:00:55.0436504Z SHA1=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0436978Z GITHUB_SHA=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0437684Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/134247 2024-10-08T21:00:55.0438323Z VERBOSE_TEST_LOGS=False 2024-10-08T21:00:55.0438670Z GITHUB_REF=refs/tags/ciflow/trunk/134247 2024-10-08T21:00:55.0439047Z SHARD_NUMBER=1 2024-10-08T21:00:55.0439333Z GITHUB_REF_PROTECTED=false 2024-10-08T21:00:55.0439649Z HOME=/var/lib/jenkins 2024-10-08T21:00:55.0439981Z GITHUB_API_URL=https://api.github.com 2024-10-08T21:00:55.0440380Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-10-08T21:00:55.0440761Z LANG=C.UTF-8 2024-10-08T21:00:55.0441042Z PYTORCH_TEST_WITH_ROCM=1 2024-10-08T21:00:55.0441363Z NUM_TEST_SHARDS=1 2024-10-08T21:00:55.0442160Z GITHUB_STATE=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0443203Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-10-08T21:00:55.0443785Z MAGMA_HOME=/opt/rocm/magma 2024-10-08T21:00:55.0444587Z GITHUB_ENV=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0445333Z HSAKMT_DEBUG_LEVEL=4 2024-10-08T21:00:55.0445940Z GITHUB_EVENT_PATH=/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-10-08T21:00:55.0446554Z GITHUB_EVENT_NAME=push 2024-10-08T21:00:55.0446869Z GITHUB_RUN_ID=11242664656 2024-10-08T21:00:55.0447736Z GITHUB_STEP_SUMMARY=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0448837Z GITHUB_ACTOR=pytorch-bot[bot] 2024-10-08T21:00:55.0449169Z PR_NUMBER= 2024-10-08T21:00:55.0449432Z GITHUB_RUN_ATTEMPT=1 2024-10-08T21:00:55.0449719Z VALGRIND=ON 2024-10-08T21:00:55.0450005Z ANACONDA_PYTHON_VERSION=3.10 2024-10-08T21:00:55.0450432Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-10-08T21:00:55.0450864Z TERM=vt100 2024-10-08T21:00:55.0451128Z INSTALLED_VISION=yes 2024-10-08T21:00:55.0451405Z BRANCH= 2024-10-08T21:00:55.0452339Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-10-08T21:00:55.0453994Z GITHUB_ACTION_PATH=/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-10-08T21:00:55.0454732Z GITHUB_SERVER_URL=https://github.com 2024-10-08T21:00:55.0455122Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-10-08T21:00:55.0455466Z REENABLED_ISSUES= 2024-10-08T21:00:55.0455729Z SHLVL=1 2024-10-08T21:00:55.0455970Z MAX_JOBS=62 2024-10-08T21:00:55.0456240Z GITHUB_ACTOR_ID=54816060 2024-10-08T21:00:55.0456677Z GITHUB_WORKFLOW_SHA=47952c0a8001fd60c9ffe52b121d022d440bef8b 2024-10-08T21:00:55.0457168Z GITHUB_REF_NAME=ciflow/trunk/134247 2024-10-08T21:00:55.0457531Z ROCM_PATH=/opt/rocm 2024-10-08T21:00:55.0458054Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-10-08T21:00:55.0458582Z GITHUB_JOB=test 2024-10-08T21:00:55.0458860Z NO_TEST_TIMEOUT=False 2024-10-08T21:00:55.0459217Z GITHUB_REPOSITORY=pytorch/pytorch 2024-10-08T21:00:55.0459588Z LC_ALL=C.UTF-8 2024-10-08T21:00:55.0459874Z GITHUB_RETENTION_DAYS=90 2024-10-08T21:00:55.0460203Z GITHUB_ACTION_REPOSITORY= 2024-10-08T21:00:55.0461405Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T21:00:55.0462626Z GITHUB_BASE_REF= 2024-10-08T21:00:55.0462907Z CI=true 2024-10-08T21:00:55.0463305Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-10-08T21:00:55.0464297Z GITHUB_REPOSITORY_OWNER=pytorch 2024-10-08T21:00:55.0464967Z JOB_ID=31258392692 2024-10-08T21:00:55.0465528Z INSTALLED_PROTOBUF=yes 2024-10-08T21:00:55.0466104Z GITHUB_HEAD_REF= 2024-10-08T21:00:55.0466635Z GITHUB_ACTION_REF= 2024-10-08T21:00:55.0467410Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-10-08T21:00:55.0468249Z TEST_SHOWLOCALS=False 2024-10-08T21:00:55.0468832Z GITHUB_WORKFLOW=trunk 2024-10-08T21:00:55.0469441Z DEBIAN_FRONTEND=noninteractive 2024-10-08T21:00:55.0471113Z GITHUB_OUTPUT=/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_7c15489d-355f-4902-85c9-cbec42f98fd5 2024-10-08T21:00:55.0472594Z NO_TD=False 2024-10-08T21:00:55.0473184Z OLDPWD=/var/lib/jenkins 2024-10-08T21:00:55.0473858Z _=/usr/bin/env 2024-10-08T21:00:55.0474561Z + echo 'Testing pytorch' 2024-10-08T21:00:55.0475267Z Testing pytorch 2024-10-08T21:00:55.0475912Z + export LANG=C.UTF-8 2024-10-08T21:00:55.0476500Z + LANG=C.UTF-8 2024-10-08T21:00:55.0476998Z + PR_NUMBER= 2024-10-08T21:00:55.0477575Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-10-08T21:00:55.0478401Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-10-08T21:00:55.0479363Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-10-08T21:00:55.0480172Z + export HIP_VISIBLE_DEVICES=0,1 2024-10-08T21:00:55.0480881Z + HIP_VISIBLE_DEVICES=0,1 2024-10-08T21:00:55.0481537Z + [[ distributed == \s\l\o\w ]] 2024-10-08T21:00:55.0482501Z + [[ linux-focal-rocm6.2-py3.10 == *slow-gradcheck* ]] 2024-10-08T21:00:55.0483524Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-10-08T21:00:55.0484445Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-10-08T21:00:55.0485309Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-10-08T21:00:55.0486157Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-10-08T21:00:55.0486928Z + [[ distributed == *crossref* ]] 2024-10-08T21:00:55.0487763Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-10-08T21:00:55.0488530Z + export VALGRIND=OFF 2024-10-08T21:00:55.0489520Z + VALGRIND=OFF 2024-10-08T21:00:55.0490024Z + rocminfo 2024-10-08T21:00:55.0595324Z ROCk module version 6.8.5 is loaded 2024-10-08T21:00:55.1179989Z ===================== 2024-10-08T21:00:55.1180935Z HSA System Attributes 2024-10-08T21:00:55.1181594Z ===================== 2024-10-08T21:00:55.1182275Z Runtime Version: 1.14 2024-10-08T21:00:55.1182985Z Runtime Ext Version: 1.6 2024-10-08T21:00:55.1183693Z System Timestamp Freq.: 1000.000000MHz 2024-10-08T21:00:55.1184885Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-10-08T21:00:55.1186484Z Machine Model: LARGE 2024-10-08T21:00:55.1187725Z System Endianness: LITTLE 2024-10-08T21:00:55.1188794Z Mwaitx: DISABLED 2024-10-08T21:00:55.1189598Z DMAbuf Support: YES 2024-10-08T21:00:55.1190163Z 2024-10-08T21:00:55.1190433Z ========== 2024-10-08T21:00:55.1191142Z HSA Agents 2024-10-08T21:00:55.1191731Z ========== 2024-10-08T21:00:55.1192254Z ******* 2024-10-08T21:00:55.1192798Z Agent 1 2024-10-08T21:00:55.1193354Z ******* 2024-10-08T21:00:55.1194277Z Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1195337Z Uuid: CPU-XX 2024-10-08T21:00:55.1196421Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1197401Z Vendor Name: CPU 2024-10-08T21:00:55.1198364Z Feature: None specified 2024-10-08T21:00:55.1199307Z Profile: FULL_PROFILE 2024-10-08T21:00:55.1200252Z Float Round Mode: NEAR 2024-10-08T21:00:55.1201258Z Max Queue Number: 0(0x0) 2024-10-08T21:00:55.1202242Z Queue Min Size: 0(0x0) 2024-10-08T21:00:55.1203647Z Queue Max Size: 0(0x0) 2024-10-08T21:00:55.1204582Z Queue Type: MULTI 2024-10-08T21:00:55.1205474Z Node: 0 2024-10-08T21:00:55.1206350Z Device Type: CPU 2024-10-08T21:00:55.1207153Z Cache Info: 2024-10-08T21:00:55.1207880Z L1: 32768(0x8000) KB 2024-10-08T21:00:55.1208924Z Chip ID: 0(0x0) 2024-10-08T21:00:55.1210047Z ASIC Revision: 0(0x0) 2024-10-08T21:00:55.1210995Z Cacheline Size: 64(0x40) 2024-10-08T21:00:55.1211955Z Max Clock Freq. (MHz): 2600 2024-10-08T21:00:55.1212939Z BDFID: 0 2024-10-08T21:00:55.1214021Z Internal Node ID: 0 2024-10-08T21:00:55.1214723Z Compute Unit: 32 2024-10-08T21:00:55.1215227Z SIMDs per CU: 0 2024-10-08T21:00:55.1215725Z Shader Engines: 0 2024-10-08T21:00:55.1216244Z Shader Arrs. per Eng.: 0 2024-10-08T21:00:55.1216789Z WatchPts on Addr. Ranges:1 2024-10-08T21:00:55.1217256Z Memory Properties: 2024-10-08T21:00:55.1217607Z Features: None 2024-10-08T21:00:55.1217973Z Pool Info: 2024-10-08T21:00:55.1218296Z Pool 1 2024-10-08T21:00:55.1218735Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T21:00:55.1219277Z Size: 65839400(0x3eca128) KB 2024-10-08T21:00:55.1219992Z Allocatable: TRUE 2024-10-08T21:00:55.1220538Z Alloc Granule: 4KB 2024-10-08T21:00:55.1221103Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1221678Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1222226Z Accessible by all: TRUE 2024-10-08T21:00:55.1222742Z Pool 2 2024-10-08T21:00:55.1223523Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-10-08T21:00:55.1224748Z Size: 65839400(0x3eca128) KB 2024-10-08T21:00:55.1225837Z Allocatable: TRUE 2024-10-08T21:00:55.1226850Z Alloc Granule: 4KB 2024-10-08T21:00:55.1227928Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1229021Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1230098Z Accessible by all: TRUE 2024-10-08T21:00:55.1230960Z Pool 3 2024-10-08T21:00:55.1231773Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T21:00:55.1232785Z Size: 65839400(0x3eca128) KB 2024-10-08T21:00:55.1233849Z Allocatable: TRUE 2024-10-08T21:00:55.1234854Z Alloc Granule: 4KB 2024-10-08T21:00:55.1235976Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1237061Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1238154Z Accessible by all: TRUE 2024-10-08T21:00:55.1239041Z ISA Info: 2024-10-08T21:00:55.1239732Z ******* 2024-10-08T21:00:55.1240328Z Agent 2 2024-10-08T21:00:55.1240905Z ******* 2024-10-08T21:00:55.1242149Z Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1243256Z Uuid: CPU-XX 2024-10-08T21:00:55.1244402Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1245436Z Vendor Name: CPU 2024-10-08T21:00:55.1246427Z Feature: None specified 2024-10-08T21:00:55.1247395Z Profile: FULL_PROFILE 2024-10-08T21:00:55.1248397Z Float Round Mode: NEAR 2024-10-08T21:00:55.1249523Z Max Queue Number: 0(0x0) 2024-10-08T21:00:55.1250523Z Queue Min Size: 0(0x0) 2024-10-08T21:00:55.1251517Z Queue Max Size: 0(0x0) 2024-10-08T21:00:55.1252505Z Queue Type: MULTI 2024-10-08T21:00:55.1253573Z Node: 1 2024-10-08T21:00:55.1254671Z Device Type: CPU 2024-10-08T21:00:55.1255646Z Cache Info: 2024-10-08T21:00:55.1256527Z L1: 32768(0x8000) KB 2024-10-08T21:00:55.1257638Z Chip ID: 0(0x0) 2024-10-08T21:00:55.1258274Z ASIC Revision: 0(0x0) 2024-10-08T21:00:55.1258813Z Cacheline Size: 64(0x40) 2024-10-08T21:00:55.1259323Z Max Clock Freq. (MHz): 2600 2024-10-08T21:00:55.1259812Z BDFID: 0 2024-10-08T21:00:55.1260304Z Internal Node ID: 1 2024-10-08T21:00:55.1260814Z Compute Unit: 32 2024-10-08T21:00:55.1261479Z SIMDs per CU: 0 2024-10-08T21:00:55.1261979Z Shader Engines: 0 2024-10-08T21:00:55.1262495Z Shader Arrs. per Eng.: 0 2024-10-08T21:00:55.1263025Z WatchPts on Addr. Ranges:1 2024-10-08T21:00:55.1263505Z Memory Properties: 2024-10-08T21:00:55.1263899Z Features: None 2024-10-08T21:00:55.1264607Z Pool Info: 2024-10-08T21:00:55.1265331Z Pool 1 2024-10-08T21:00:55.1266290Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T21:00:55.1267336Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T21:00:55.1268338Z Allocatable: TRUE 2024-10-08T21:00:55.1269360Z Alloc Granule: 4KB 2024-10-08T21:00:55.1270447Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1271537Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1272600Z Accessible by all: TRUE 2024-10-08T21:00:55.1273461Z Pool 2 2024-10-08T21:00:55.1274281Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-10-08T21:00:55.1275288Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T21:00:55.1276262Z Allocatable: TRUE 2024-10-08T21:00:55.1277252Z Alloc Granule: 4KB 2024-10-08T21:00:55.1278322Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1279371Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1280403Z Accessible by all: TRUE 2024-10-08T21:00:55.1281265Z Pool 3 2024-10-08T21:00:55.1282369Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T21:00:55.1283371Z Size: 65997868(0x3ef0c2c) KB 2024-10-08T21:00:55.1284347Z Allocatable: TRUE 2024-10-08T21:00:55.1285347Z Alloc Granule: 4KB 2024-10-08T21:00:55.1286389Z Alloc Recommended Granule:4KB 2024-10-08T21:00:55.1287461Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1288480Z Accessible by all: TRUE 2024-10-08T21:00:55.1289531Z ISA Info: 2024-10-08T21:00:55.1290238Z ******* 2024-10-08T21:00:55.1290908Z Agent 3 2024-10-08T21:00:55.1291573Z ******* 2024-10-08T21:00:55.1292402Z Name: gfx90a 2024-10-08T21:00:55.1293705Z Uuid: GPU-acdd3d2233dd93e8 2024-10-08T21:00:55.1294631Z Marketing Name: AMD Instinct MI210 2024-10-08T21:00:55.1295210Z Vendor Name: AMD 2024-10-08T21:00:55.1295722Z Feature: KERNEL_DISPATCH 2024-10-08T21:00:55.1296230Z Profile: BASE_PROFILE 2024-10-08T21:00:55.1296749Z Float Round Mode: NEAR 2024-10-08T21:00:55.1297270Z Max Queue Number: 128(0x80) 2024-10-08T21:00:55.1297779Z Queue Min Size: 64(0x40) 2024-10-08T21:00:55.1298281Z Queue Max Size: 131072(0x20000) 2024-10-08T21:00:55.1298776Z Queue Type: MULTI 2024-10-08T21:00:55.1299242Z Node: 2 2024-10-08T21:00:55.1299880Z Device Type: GPU 2024-10-08T21:00:55.1300303Z Cache Info: 2024-10-08T21:00:55.1300682Z L1: 16(0x10) KB 2024-10-08T21:00:55.1301149Z L2: 8192(0x2000) KB 2024-10-08T21:00:55.1301635Z Chip ID: 29711(0x740f) 2024-10-08T21:00:55.1302133Z ASIC Revision: 1(0x1) 2024-10-08T21:00:55.1302648Z Cacheline Size: 64(0x40) 2024-10-08T21:00:55.1303155Z Max Clock Freq. (MHz): 1700 2024-10-08T21:00:55.1303637Z BDFID: 768 2024-10-08T21:00:55.1304120Z Internal Node ID: 2 2024-10-08T21:00:55.1304616Z Compute Unit: 104 2024-10-08T21:00:55.1305097Z SIMDs per CU: 4 2024-10-08T21:00:55.1305602Z Shader Engines: 8 2024-10-08T21:00:55.1306116Z Shader Arrs. per Eng.: 1 2024-10-08T21:00:55.1306643Z WatchPts on Addr. Ranges:4 2024-10-08T21:00:55.1307171Z Coherent Host Access: FALSE 2024-10-08T21:00:55.1307617Z Memory Properties: 2024-10-08T21:00:55.1307979Z Features: KERNEL_DISPATCH 2024-10-08T21:00:55.1308461Z Fast F16 Operation: TRUE 2024-10-08T21:00:55.1308986Z Wavefront Size: 64(0x40) 2024-10-08T21:00:55.1309495Z Workgroup Max Size: 1024(0x400) 2024-10-08T21:00:55.1309952Z Workgroup Max Size per Dimension: 2024-10-08T21:00:55.1310374Z x 1024(0x400) 2024-10-08T21:00:55.1310822Z y 1024(0x400) 2024-10-08T21:00:55.1311424Z z 1024(0x400) 2024-10-08T21:00:55.1311915Z Max Waves Per CU: 32(0x20) 2024-10-08T21:00:55.1312516Z Max Work-item Per CU: 2048(0x800) 2024-10-08T21:00:55.1313111Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T21:00:55.1313980Z Grid Max Size per Dimension: 2024-10-08T21:00:55.1314849Z x 4294967295(0xffffffff) 2024-10-08T21:00:55.1315879Z y 4294967295(0xffffffff) 2024-10-08T21:00:55.1316757Z z 4294967295(0xffffffff) 2024-10-08T21:00:55.1317706Z Max fbarriers/Workgrp: 32 2024-10-08T21:00:55.1318778Z Packet Processor uCode:: 83 2024-10-08T21:00:55.1319804Z SDMA engine uCode:: 8 2024-10-08T21:00:55.1320824Z IOMMU Support:: None 2024-10-08T21:00:55.1321659Z Pool Info: 2024-10-08T21:00:55.1322278Z Pool 1 2024-10-08T21:00:55.1323100Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T21:00:55.1324134Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1325144Z Allocatable: TRUE 2024-10-08T21:00:55.1326179Z Alloc Granule: 4KB 2024-10-08T21:00:55.1327255Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1328342Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1329578Z Accessible by all: FALSE 2024-10-08T21:00:55.1330453Z Pool 2 2024-10-08T21:00:55.1331267Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-10-08T21:00:55.1332616Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1333772Z Allocatable: TRUE 2024-10-08T21:00:55.1334962Z Alloc Granule: 4KB 2024-10-08T21:00:55.1336219Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1337475Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1338423Z Accessible by all: FALSE 2024-10-08T21:00:55.1338948Z Pool 3 2024-10-08T21:00:55.1339394Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T21:00:55.1339914Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1340417Z Allocatable: TRUE 2024-10-08T21:00:55.1340934Z Alloc Granule: 4KB 2024-10-08T21:00:55.1341496Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1342041Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1342587Z Accessible by all: FALSE 2024-10-08T21:00:55.1343034Z Pool 4 2024-10-08T21:00:55.1343428Z Segment: GROUP 2024-10-08T21:00:55.1343915Z Size: 64(0x40) KB 2024-10-08T21:00:55.1344414Z Allocatable: FALSE 2024-10-08T21:00:55.1344927Z Alloc Granule: 0KB 2024-10-08T21:00:55.1345467Z Alloc Recommended Granule:0KB 2024-10-08T21:00:55.1346018Z Alloc Alignment: 0KB 2024-10-08T21:00:55.1346546Z Accessible by all: FALSE 2024-10-08T21:00:55.1346997Z ISA Info: 2024-10-08T21:00:55.1347476Z ISA 1 2024-10-08T21:00:55.1347982Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-10-08T21:00:55.1348562Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-10-08T21:00:55.1349108Z Profiles: HSA_PROFILE_BASE 2024-10-08T21:00:55.1349648Z Default Rounding Mode: NEAR 2024-10-08T21:00:55.1350195Z Default Rounding Mode: NEAR 2024-10-08T21:00:55.1350715Z Fast f16: TRUE 2024-10-08T21:00:55.1351232Z Workgroup Max Size: 1024(0x400) 2024-10-08T21:00:55.1351711Z Workgroup Max Size per Dimension: 2024-10-08T21:00:55.1352152Z x 1024(0x400) 2024-10-08T21:00:55.1352621Z y 1024(0x400) 2024-10-08T21:00:55.1353099Z z 1024(0x400) 2024-10-08T21:00:55.1353606Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T21:00:55.1354085Z Grid Max Size per Dimension: 2024-10-08T21:00:55.1354617Z x 4294967295(0xffffffff) 2024-10-08T21:00:55.1355631Z y 4294967295(0xffffffff) 2024-10-08T21:00:55.1356696Z z 4294967295(0xffffffff) 2024-10-08T21:00:55.1357743Z FBarrier Max Size: 32 2024-10-08T21:00:55.1358598Z ******* 2024-10-08T21:00:55.1359184Z Agent 4 2024-10-08T21:00:55.1359757Z ******* 2024-10-08T21:00:55.1360473Z Name: gfx90a 2024-10-08T21:00:55.1361553Z Uuid: GPU-25354a01ff6db6f2 2024-10-08T21:00:55.1362859Z Marketing Name: AMD Instinct MI210 2024-10-08T21:00:55.1363866Z Vendor Name: AMD 2024-10-08T21:00:55.1364841Z Feature: KERNEL_DISPATCH 2024-10-08T21:00:55.1365803Z Profile: BASE_PROFILE 2024-10-08T21:00:55.1366794Z Float Round Mode: NEAR 2024-10-08T21:00:55.1367796Z Max Queue Number: 128(0x80) 2024-10-08T21:00:55.1368900Z Queue Min Size: 64(0x40) 2024-10-08T21:00:55.1369883Z Queue Max Size: 131072(0x20000) 2024-10-08T21:00:55.1370851Z Queue Type: MULTI 2024-10-08T21:00:55.1371761Z Node: 3 2024-10-08T21:00:55.1372679Z Device Type: GPU 2024-10-08T21:00:55.1373637Z Cache Info: 2024-10-08T21:00:55.1374515Z L1: 16(0x10) KB 2024-10-08T21:00:55.1375585Z L2: 8192(0x2000) KB 2024-10-08T21:00:55.1376701Z Chip ID: 29711(0x740f) 2024-10-08T21:00:55.1377881Z ASIC Revision: 1(0x1) 2024-10-08T21:00:55.1378929Z Cacheline Size: 64(0x40) 2024-10-08T21:00:55.1379936Z Max Clock Freq. (MHz): 1700 2024-10-08T21:00:55.1380882Z BDFID: 33536 2024-10-08T21:00:55.1381854Z Internal Node ID: 3 2024-10-08T21:00:55.1382845Z Compute Unit: 104 2024-10-08T21:00:55.1383941Z SIMDs per CU: 4 2024-10-08T21:00:55.1385112Z Shader Engines: 8 2024-10-08T21:00:55.1386623Z Shader Arrs. per Eng.: 1 2024-10-08T21:00:55.1387682Z WatchPts on Addr. Ranges:4 2024-10-08T21:00:55.1388745Z Coherent Host Access: FALSE 2024-10-08T21:00:55.1389631Z Memory Properties: 2024-10-08T21:00:55.1390333Z Features: KERNEL_DISPATCH 2024-10-08T21:00:55.1391267Z Fast F16 Operation: TRUE 2024-10-08T21:00:55.1392299Z Wavefront Size: 64(0x40) 2024-10-08T21:00:55.1393334Z Workgroup Max Size: 1024(0x400) 2024-10-08T21:00:55.1394244Z Workgroup Max Size per Dimension: 2024-10-08T21:00:55.1395088Z x 1024(0x400) 2024-10-08T21:00:55.1395967Z y 1024(0x400) 2024-10-08T21:00:55.1396832Z z 1024(0x400) 2024-10-08T21:00:55.1397795Z Max Waves Per CU: 32(0x20) 2024-10-08T21:00:55.1398974Z Max Work-item Per CU: 2048(0x800) 2024-10-08T21:00:55.1400009Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T21:00:55.1400879Z Grid Max Size per Dimension: 2024-10-08T21:00:55.1401611Z x 4294967295(0xffffffff) 2024-10-08T21:00:55.1402494Z y 4294967295(0xffffffff) 2024-10-08T21:00:55.1403429Z z 4294967295(0xffffffff) 2024-10-08T21:00:55.1404391Z Max fbarriers/Workgrp: 32 2024-10-08T21:00:55.1405454Z Packet Processor uCode:: 83 2024-10-08T21:00:55.1406504Z SDMA engine uCode:: 8 2024-10-08T21:00:55.1407516Z IOMMU Support:: None 2024-10-08T21:00:55.1408698Z Pool Info: 2024-10-08T21:00:55.1409444Z Pool 1 2024-10-08T21:00:55.1410282Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-10-08T21:00:55.1411338Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1412349Z Allocatable: TRUE 2024-10-08T21:00:55.1413381Z Alloc Granule: 4KB 2024-10-08T21:00:55.1414460Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1415561Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1416612Z Accessible by all: FALSE 2024-10-08T21:00:55.1417473Z Pool 2 2024-10-08T21:00:55.1418315Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-10-08T21:00:55.1419418Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1420622Z Allocatable: TRUE 2024-10-08T21:00:55.1421805Z Alloc Granule: 4KB 2024-10-08T21:00:55.1423075Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1424366Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1425468Z Accessible by all: FALSE 2024-10-08T21:00:55.1426334Z Pool 3 2024-10-08T21:00:55.1427131Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-10-08T21:00:55.1428135Z Size: 67092480(0x3ffc000) KB 2024-10-08T21:00:55.1429147Z Allocatable: TRUE 2024-10-08T21:00:55.1430250Z Alloc Granule: 4KB 2024-10-08T21:00:55.1431492Z Alloc Recommended Granule:2048KB 2024-10-08T21:00:55.1433064Z Alloc Alignment: 4KB 2024-10-08T21:00:55.1434125Z Accessible by all: FALSE 2024-10-08T21:00:55.1435008Z Pool 4 2024-10-08T21:00:55.1435768Z Segment: GROUP 2024-10-08T21:00:55.1436715Z Size: 64(0x40) KB 2024-10-08T21:00:55.1437676Z Allocatable: FALSE 2024-10-08T21:00:55.1438711Z Alloc Granule: 0KB 2024-10-08T21:00:55.1439773Z Alloc Recommended Granule:0KB 2024-10-08T21:00:55.1440831Z Alloc Alignment: 0KB 2024-10-08T21:00:55.1441848Z Accessible by all: FALSE 2024-10-08T21:00:55.1442717Z ISA Info: 2024-10-08T21:00:55.1443357Z ISA 1 2024-10-08T21:00:55.1444342Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-10-08T21:00:55.1445454Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-10-08T21:00:55.1446498Z Profiles: HSA_PROFILE_BASE 2024-10-08T21:00:55.1447540Z Default Rounding Mode: NEAR 2024-10-08T21:00:55.1448618Z Default Rounding Mode: NEAR 2024-10-08T21:00:55.1449742Z Fast f16: TRUE 2024-10-08T21:00:55.1450748Z Workgroup Max Size: 1024(0x400) 2024-10-08T21:00:55.1451666Z Workgroup Max Size per Dimension: 2024-10-08T21:00:55.1452532Z x 1024(0x400) 2024-10-08T21:00:55.1453580Z y 1024(0x400) 2024-10-08T21:00:55.1455018Z z 1024(0x400) 2024-10-08T21:00:55.1456191Z Grid Max Size: 4294967295(0xffffffff) 2024-10-08T21:00:55.1457277Z Grid Max Size per Dimension: 2024-10-08T21:00:55.1458211Z x 4294967295(0xffffffff) 2024-10-08T21:00:55.1459113Z y 4294967295(0xffffffff) 2024-10-08T21:00:55.1460019Z z 4294967295(0xffffffff) 2024-10-08T21:00:55.1461008Z FBarrier Max Size: 32 2024-10-08T21:00:55.1461868Z *** Done *** 2024-10-08T21:00:55.1462461Z + rocminfo 2024-10-08T21:00:55.1463077Z + grep -E 'Name:.*\sgfx|Marketing' 2024-10-08T21:00:55.1788129Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1789372Z Marketing Name: AMD EPYC 7513 32-Core Processor 2024-10-08T21:00:55.1790396Z Name: gfx90a 2024-10-08T21:00:55.1791516Z Marketing Name: AMD Instinct MI210 2024-10-08T21:00:55.1792479Z Name: gfx90a 2024-10-08T21:00:55.1793437Z Marketing Name: AMD Instinct MI210 2024-10-08T21:00:55.1946426Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-10-08T21:00:55.1947425Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-10-08T21:00:55.1948352Z + pip_install --user ninja==1.10.2 2024-10-08T21:00:55.1949327Z + pip install --progress-bar off --user ninja==1.10.2 2024-10-08T21:00:55.5267168Z Collecting ninja==1.10.2 2024-10-08T21:00:55.5480210Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-10-08T21:00:55.5580888Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-10-08T21:00:55.8084826Z Installing collected packages: ninja 2024-10-08T21:00:55.8151152Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-10-08T21:00:55.8154169Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-10-08T21:00:55.8177141Z Successfully installed ninja-1.10.2 2024-10-08T21:00:55.8622471Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T21:00:55.8627458Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T21:00:55.8630399Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-10-08T21:00:55.8631238Z + install_tlparse 2024-10-08T21:00:55.8631951Z + pip_install --user tlparse==0.3.25 2024-10-08T21:00:55.8632966Z + pip install --progress-bar off --user tlparse==0.3.25 2024-10-08T21:00:56.1971618Z Collecting tlparse==0.3.25 2024-10-08T21:00:56.2164501Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-10-08T21:00:56.2249812Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-10-08T21:00:56.5514308Z Installing collected packages: tlparse 2024-10-08T21:00:56.5801354Z Successfully installed tlparse-0.3.25 2024-10-08T21:00:56.6366857Z ++ python -m site --user-base 2024-10-08T21:00:56.6587558Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-10-08T21:00:56.6590885Z + [[ linux-focal-rocm6.2-py3.10 == *asan* ]] 2024-10-08T21:00:56.6592507Z + [[ linux-focal-rocm6.2-py3.10 == *-debug* ]] 2024-10-08T21:00:56.6593520Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-10-08T21:00:56.6595004Z + echo 'We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass' 2024-10-08T21:00:56.6596796Z We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass 2024-10-08T21:00:56.6597945Z + cd test 2024-10-08T21:00:56.6598910Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-10-08T21:00:59.6041209Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-10-08T21:00:59.6042527Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-10-08T21:00:59.6055313Z + DYNAMO_BENCHMARK_FLAGS=() 2024-10-08T21:00:59.6056205Z + [[ distributed == *pr_time_benchmarks* ]] 2024-10-08T21:00:59.6057093Z + [[ distributed == *dynamo_eager* ]] 2024-10-08T21:00:59.6057853Z + [[ distributed == *aot_eager* ]] 2024-10-08T21:00:59.6058614Z + [[ distributed == *aot_inductor* ]] 2024-10-08T21:00:59.6059387Z + [[ distributed == *inductor* ]] 2024-10-08T21:00:59.6060153Z + [[ distributed == *dynamic* ]] 2024-10-08T21:00:59.6060882Z + [[ distributed == *cpu* ]] 2024-10-08T21:00:59.6061946Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-10-08T21:00:59.6116252Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-10-08T21:00:59.6117415Z + [[ linux-focal-rocm6.2-py3.10 == *-bazel-* ]] 2024-10-08T21:00:59.6122850Z + cd test 2024-10-08T21:00:59.6123919Z + python -c 'import torch; print(torch.__config__.show())' 2024-10-08T21:01:01.9886467Z PyTorch built with: 2024-10-08T21:01:01.9887433Z - GCC 9.4 2024-10-08T21:01:01.9888032Z - C++ Version: 201703 2024-10-08T21:01:01.9889912Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-10-08T21:01:01.9891966Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-10-08T21:01:01.9893203Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-10-08T21:01:01.9894270Z - LAPACK is enabled (usually provided by MKL) 2024-10-08T21:01:01.9895318Z - NNPACK is enabled 2024-10-08T21:01:01.9907461Z - CPU capability usage: AVX2 2024-10-08T21:01:01.9909660Z - HIP Runtime 6.2.41133 2024-10-08T21:01:01.9910428Z - MIOpen 3.2.0 2024-10-08T21:01:01.9911031Z - Magma 2.7.2 2024-10-08T21:01:01.9922976Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-10-08T21:01:01.9934242Z 2024-10-08T21:01:02.6031406Z + cd test 2024-10-08T21:01:02.6032727Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-10-08T21:01:03.8618422Z ATen/Parallel: 2024-10-08T21:01:03.8619220Z at::get_num_threads() : 64 2024-10-08T21:01:03.8619967Z at::get_num_interop_threads() : 64 2024-10-08T21:01:03.8620735Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-10-08T21:01:03.8621482Z omp_get_max_threads() : 64 2024-10-08T21:01:03.8623274Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-10-08T21:01:03.8625589Z mkl_get_max_threads() : 64 2024-10-08T21:01:03.8626794Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-10-08T21:01:03.8627957Z std::thread::hardware_concurrency() : 64 2024-10-08T21:01:03.8628722Z Environment variables: 2024-10-08T21:01:03.8629339Z OMP_NUM_THREADS : [not set] 2024-10-08T21:01:03.8630011Z MKL_NUM_THREADS : [not set] 2024-10-08T21:01:03.8630685Z ATen parallel backend: OpenMP 2024-10-08T21:01:03.8631146Z 2024-10-08T21:01:05.5595886Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-10-08T21:01:05.5596873Z + [[ distributed == *backward* ]] 2024-10-08T21:01:05.5597629Z + [[ distributed == *xla* ]] 2024-10-08T21:01:05.5598364Z + [[ distributed == *executorch* ]] 2024-10-08T21:01:05.5599150Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-10-08T21:01:05.5600143Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-10-08T21:01:05.5601002Z + [[ distributed == distributed ]] 2024-10-08T21:01:05.5601710Z + test_distributed 2024-10-08T21:01:05.5602490Z + echo 'Testing distributed python tests' 2024-10-08T21:01:05.5603303Z Testing distributed python tests 2024-10-08T21:01:05.5606503Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-10-08T21:01:05.6453222Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T21:01:05.6455390Z import pkg_resources 2024-10-08T21:01:09.9947446Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-10-08T21:01:10.3922126Z Ignoring disabled issues: [''] 2024-10-08T21:01:10.4183912Z Found test times from artifacts 2024-10-08T21:01:10.4740896Z Found test times from artifacts 2024-10-08T21:01:10.4754059Z Running all tests 2024-10-08T21:01:10.4755852Z Running parallel tests on 2 processes 2024-10-08T21:01:10.4756884Z Name: tests to run (est. time: 24.26min) 2024-10-08T21:01:10.4757630Z Serial tests (4): 2024-10-08T21:01:10.4758269Z distributed/test_c10d_common 1/1 2024-10-08T21:01:10.4759040Z distributed/test_c10d_nccl 1/3 2024-10-08T21:01:10.4759787Z distributed/test_c10d_nccl 2/3 2024-10-08T21:01:10.4760479Z distributed/test_c10d_nccl 3/3 2024-10-08T21:01:10.4761160Z Parallel tests (0): 2024-10-08T21:01:10.4761767Z Name: excluded (est. time: 0.0min) 2024-10-08T21:01:10.4762448Z Serial tests (0): 2024-10-08T21:01:10.4763005Z Parallel tests (0): 2024-10-08T21:01:10.4824626Z Running distributed/test_c10d_common 1/1 ... [2024-10-08 21:01:10.481862] 2024-10-08T21:01:10.4825936Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:01:10.4829235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 21:01:10.482128] 2024-10-08T21:04:26.3344843Z 2024-10-08T21:04:26.3348734Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_34ac93db18e4a72d_.log 2024-10-08T21:04:26.3359139Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-10-08T21:04:26.3369985Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-10-08T21:04:26.3371103Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-10-08T21:04:26.3372320Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-10-08T21:04:26.3373580Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-10-08T21:04:26.3375044Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-10-08T21:04:26.3376118Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-10-08T21:04:26.3377192Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-10-08T21:04:26.3378390Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-10-08T21:04:26.3379562Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-10-08T21:04:26.3380743Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-10-08T21:04:26.3382046Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-10-08T21:04:26.3383352Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-10-08T21:04:26.3384526Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-10-08T21:04:26.3385838Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-10-08T21:04:26.3387342Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-10-08T21:04:26.3388616Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-10-08T21:04:26.3389643Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-10-08T21:04:26.3390801Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-10-08T21:04:26.3391777Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-10-08T21:04:26.3392743Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-10-08T21:04:26.3393814Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-10-08T21:04:26.3394845Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-10-08T21:04:26.3395839Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-10-08T21:04:26.3396441Z 2024-10-08T21:04:26.3396832Z Running distributed/test_c10d_nccl 1/3 ... [2024-10-08 21:04:26.334763] 2024-10-08T21:04:26.3397793Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:04:26.3401095Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 21:04:26.335036] 2024-10-08T21:17:25.7263690Z 2024-10-08T21:17:25.7265134Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_e6712ec04b3327f0_.log 2024-10-08T21:17:25.7301160Z Running 72 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-10-08T21:17:25.7327052Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-10-08T21:17:25.7327898Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-10-08T21:17:25.7328869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-10-08T21:17:25.7329778Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-10-08T21:17:25.7330686Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-10-08T21:17:25.7331579Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg 2024-10-08T21:17:25.7332439Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-10-08T21:17:25.7333357Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_optimization 2024-10-08T21:17:25.7334338Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2024-10-08T21:17:25.7335301Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-10-08T21:17:25.7336241Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-10-08T21:17:25.7337335Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-10-08T21:17:25.7338265Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2 2024-10-08T21:17:25.7339177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-10-08T21:17:25.7340123Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-10-08T21:17:25.7341122Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-10-08T21:17:25.7342077Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-10-08T21:17:25.7343078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-10-08T21:17:25.7344187Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-10-08T21:17:25.7345317Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-10-08T21:17:25.7346431Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-10-08T21:17:25.7347433Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-10-08T21:17:25.7348383Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-10-08T21:17:25.7349382Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-10-08T21:17:25.7350578Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-10-08T21:17:25.7351648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-10-08T21:17:25.7352770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-10-08T21:17:25.7353925Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-10-08T21:17:25.7354908Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-10-08T21:17:25.7355806Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-10-08T21:17:25.7356753Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-10-08T21:17:25.7357763Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-10-08T21:17:25.7358791Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-10-08T21:17:25.7359759Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-10-08T21:17:25.7360803Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-10-08T21:17:25.7361941Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-10-08T21:17:25.7363129Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-10-08T21:17:25.7364049Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-10-08T21:17:25.7364959Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-10-08T21:17:25.7365934Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-10-08T21:17:25.7366890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-10-08T21:17:25.7367775Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-10-08T21:17:25.7368611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-10-08T21:17:25.7369495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-10-08T21:17:25.7370394Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-10-08T21:17:25.7371357Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-10-08T21:17:25.7372311Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-10-08T21:17:25.7373279Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier 2024-10-08T21:17:25.7374225Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-10-08T21:17:25.7375090Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-10-08T21:17:25.7376093Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-10-08T21:17:25.7376968Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-10-08T21:17:25.7377869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-10-08T21:17:25.7378737Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-10-08T21:17:25.7379698Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-10-08T21:17:25.7380822Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-10-08T21:17:25.7382022Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-10-08T21:17:25.7383043Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-10-08T21:17:25.7383861Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-10-08T21:17:25.7384672Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-10-08T21:17:25.7385451Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-10-08T21:17:25.7386262Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-10-08T21:17:25.7387059Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-10-08T21:17:25.7387948Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-10-08T21:17:25.7388962Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-10-08T21:17:25.7389881Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-10-08T21:17:25.7390966Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-10-08T21:17:25.7392049Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-10-08T21:17:25.7393114Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-10-08T21:17:25.7394191Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-10-08T21:17:25.7395185Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-10-08T21:17:25.7396148Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-10-08T21:17:25.7396720Z 2024-10-08T21:17:25.7397172Z Running distributed/test_c10d_nccl 2/3 ... [2024-10-08 21:17:25.727209] 2024-10-08T21:17:25.7397614Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:17:25.7398897Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 21:17:25.727489] 2024-10-08T21:28:14.0788810Z 2024-10-08T21:28:14.0793274Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_7c3205acc68158ff_.log 2024-10-08T21:28:14.0831749Z Running 64 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-10-08T21:28:14.0860275Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-10-08T21:28:14.0861382Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-10-08T21:28:14.0862546Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-10-08T21:28:14.0863776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-10-08T21:28:14.0865020Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2024-10-08T21:28:14.0866216Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-10-08T21:28:14.0867358Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-10-08T21:28:14.0868525Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn 2024-10-08T21:28:14.0869624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check 2024-10-08T21:28:14.0870705Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter 2024-10-08T21:28:14.0871872Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-10-08T21:28:14.0873067Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-10-08T21:28:14.0874520Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-10-08T21:28:14.0875854Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-10-08T21:28:14.0877099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-10-08T21:28:14.0878381Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-10-08T21:28:14.0879701Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-10-08T21:28:14.0881021Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-10-08T21:28:14.0882441Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-10-08T21:28:14.0883869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-10-08T21:28:14.0885243Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-10-08T21:28:14.0886605Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-10-08T21:28:14.0887900Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-10-08T21:28:14.0889162Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-10-08T21:28:14.0890524Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-10-08T21:28:14.0891827Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-10-08T21:28:14.0893123Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-10-08T21:28:14.0894333Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-10-08T21:28:14.0895578Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-10-08T21:28:14.0896930Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-10-08T21:28:14.0898353Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-10-08T21:28:14.0899770Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-10-08T21:28:14.0901028Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-10-08T21:28:14.0902268Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-10-08T21:28:14.0903476Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-10-08T21:28:14.0904613Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-10-08T21:28:14.0905805Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-10-08T21:28:14.0907116Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-10-08T21:28:14.0908232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-10-08T21:28:14.0909293Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-10-08T21:28:14.0910306Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-10-08T21:28:14.0911263Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-10-08T21:28:14.0912209Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-10-08T21:28:14.0913251Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-10-08T21:28:14.0914280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-10-08T21:28:14.0915339Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-10-08T21:28:14.0916393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-10-08T21:28:14.0917439Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-10-08T21:28:14.0918672Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-10-08T21:28:14.0919942Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-10-08T21:28:14.0921142Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-10-08T21:28:14.0922399Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-10-08T21:28:14.0923483Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-10-08T21:28:14.0924611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-10-08T21:28:14.0925873Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-10-08T21:28:14.0927252Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-10-08T21:28:14.0928617Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-10-08T21:28:14.0930010Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-10-08T21:28:14.0931310Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-10-08T21:28:14.0932623Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-10-08T21:28:14.0933937Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-10-08T21:28:14.0935282Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-10-08T21:28:14.0936615Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-10-08T21:28:14.0938076Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-10-08T21:28:14.0938792Z 2024-10-08T21:28:14.0939145Z Running distributed/test_c10d_nccl 3/3 ... [2024-10-08 21:28:14.080491] 2024-10-08T21:28:14.0939695Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:28:14.0941310Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-10-08 21:28:14.081141] 2024-10-08T21:38:18.9475134Z 2024-10-08T21:38:18.9476626Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_993b2c7f85ef1b8f_.log 2024-10-08T21:38:18.9503104Z Running 56 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-10-08T21:38:18.9529457Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-10-08T21:38:18.9530622Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-10-08T21:38:18.9531711Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-10-08T21:38:18.9532930Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-10-08T21:38:18.9534133Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-10-08T21:38:18.9535370Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-10-08T21:38:18.9536683Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-10-08T21:38:18.9537979Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-10-08T21:38:18.9539183Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-10-08T21:38:18.9540388Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-10-08T21:38:18.9541890Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-10-08T21:38:18.9543312Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-10-08T21:38:18.9544781Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-10-08T21:38:18.9546192Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-10-08T21:38:18.9547531Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-10-08T21:38:18.9548930Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-10-08T21:38:18.9550371Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-10-08T21:38:18.9551728Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-10-08T21:38:18.9552949Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-10-08T21:38:18.9554105Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-10-08T21:38:18.9555393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-10-08T21:38:18.9556715Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-10-08T21:38:18.9558204Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-10-08T21:38:18.9559568Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-10-08T21:38:18.9560842Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-10-08T21:38:18.9562091Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-10-08T21:38:18.9563346Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-10-08T21:38:18.9564530Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-10-08T21:38:18.9565622Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-10-08T21:38:18.9566664Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-10-08T21:38:18.9567751Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-10-08T21:38:18.9568928Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-10-08T21:38:18.9570013Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-10-08T21:38:18.9571148Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-10-08T21:38:18.9572280Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-10-08T21:38:18.9573291Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-10-08T21:38:18.9574441Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-10-08T21:38:18.9575620Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-10-08T21:38:18.9577053Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-10-08T21:38:18.9578487Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-10-08T21:38:18.9579956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-10-08T21:38:18.9581266Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-10-08T21:38:18.9582409Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-10-08T21:38:18.9583508Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-10-08T21:38:18.9584540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-10-08T21:38:18.9585636Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0 2024-10-08T21:38:18.9586829Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1 2024-10-08T21:38:18.9588115Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-10-08T21:38:18.9589596Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-10-08T21:38:18.9590886Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-10-08T21:38:18.9592184Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-10-08T21:38:18.9593506Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-10-08T21:38:18.9594833Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-10-08T21:38:18.9596155Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-10-08T21:38:18.9597396Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-10-08T21:38:18.9598580Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-10-08T21:38:18.9599236Z 2024-10-08T21:38:19.6515369Z Running test batch 'tests to run' cost 2229.18 seconds 2024-10-08T21:38:20.5035294Z 2024-10-08T21:38:20.5035737Z real 37m14.944s 2024-10-08T21:38:20.5036077Z user 49m5.358s 2024-10-08T21:38:20.5036329Z sys 55m55.049s 2024-10-08T21:38:20.5036558Z + assert_git_not_dirty 2024-10-08T21:38:20.5037128Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-10-08T21:38:20.5037542Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-10-08T21:38:20.5037939Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-10-08T21:38:20.5038268Z + [[ 1 == 1 ]] 2024-10-08T21:38:20.5038547Z + echo 'Testing distributed C++ tests' 2024-10-08T21:38:20.5038876Z Testing distributed C++ tests 2024-10-08T21:38:20.5041562Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:20.5055865Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:20.5074473Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:20.5075249Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:20.5075873Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-10-08T21:38:20.5903657Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T21:38:20.5905540Z import pkg_resources 2024-10-08T21:38:24.7987945Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-10-08T21:38:24.8089871Z Found test times from artifacts 2024-10-08T21:38:24.8522342Z Found test times from artifacts 2024-10-08T21:38:24.8535777Z Running all tests 2024-10-08T21:38:24.8536755Z Running parallel tests on 2 processes 2024-10-08T21:38:24.8537769Z Name: tests to run (est. time: 0.0min) 2024-10-08T21:38:24.8538538Z Serial tests (1): 2024-10-08T21:38:24.8539160Z cpp/FileStoreTest 1/1 2024-10-08T21:38:24.8539821Z Parallel tests (0): 2024-10-08T21:38:24.8540447Z Name: excluded (est. time: 0.0min) 2024-10-08T21:38:24.8541158Z Serial tests (0): 2024-10-08T21:38:24.8541724Z Parallel tests (0): 2024-10-08T21:38:24.8597946Z Running cpp/FileStoreTest 1/1 ... [2024-10-08 21:38:24.859181] 2024-10-08T21:38:24.8599133Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:38:24.8603660Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-822fd9c8d9fd4ad3.xml', '-x', '--reruns=2'] ... [2024-10-08 21:38:24.859715] 2024-10-08T21:38:27.2790100Z 2024-10-08T21:38:27.2792594Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_da8feaa82f39f659_.log 2024-10-08T21:38:27.2794223Z 2024-10-08T21:38:29.1346855Z Running test batch 'tests to run' cost 4.28 seconds 2024-10-08T21:38:29.9007914Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-10-08T21:38:29.9840202Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T21:38:29.9842335Z import pkg_resources 2024-10-08T21:38:34.2556700Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-10-08T21:38:34.2657424Z Found test times from artifacts 2024-10-08T21:38:34.3079722Z Found test times from artifacts 2024-10-08T21:38:34.3092843Z Running all tests 2024-10-08T21:38:34.3096202Z Running parallel tests on 2 processes 2024-10-08T21:38:34.3097288Z Name: tests to run (est. time: 0.0min) 2024-10-08T21:38:34.3098069Z Serial tests (1): 2024-10-08T21:38:34.3098697Z cpp/HashStoreTest 1/1 2024-10-08T21:38:34.3099347Z Parallel tests (0): 2024-10-08T21:38:34.3099994Z Name: excluded (est. time: 0.0min) 2024-10-08T21:38:34.3100714Z Serial tests (0): 2024-10-08T21:38:34.3101288Z Parallel tests (0): 2024-10-08T21:38:34.3151216Z Running cpp/HashStoreTest 1/1 ... [2024-10-08 21:38:34.314656] 2024-10-08T21:38:34.3153355Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:38:34.3157505Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-74bd27d638e99155.xml', '-x', '--reruns=2'] ... [2024-10-08 21:38:34.315052] 2024-10-08T21:38:36.6850565Z 2024-10-08T21:38:36.6852943Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_5e915f06ac98a95c_.log 2024-10-08T21:38:36.6854609Z 2024-10-08T21:38:38.6508727Z Running test batch 'tests to run' cost 4.34 seconds 2024-10-08T21:38:39.4037891Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-10-08T21:38:39.4876506Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T21:38:39.4878617Z import pkg_resources 2024-10-08T21:38:43.7518579Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-10-08T21:38:43.7618789Z Found test times from artifacts 2024-10-08T21:38:43.8052149Z Found test times from artifacts 2024-10-08T21:38:43.8065191Z Running all tests 2024-10-08T21:38:43.8069149Z Running parallel tests on 2 processes 2024-10-08T21:38:43.8070779Z Name: tests to run (est. time: 0.0min) 2024-10-08T21:38:43.8071514Z Serial tests (1): 2024-10-08T21:38:43.8072405Z cpp/TCPStoreTest 1/1 2024-10-08T21:38:43.8073082Z Parallel tests (0): 2024-10-08T21:38:43.8073704Z Name: excluded (est. time: 0.0min) 2024-10-08T21:38:43.8074385Z Serial tests (0): 2024-10-08T21:38:43.8074951Z Parallel tests (0): 2024-10-08T21:38:43.8123963Z Running cpp/TCPStoreTest 1/1 ... [2024-10-08 21:38:43.811815] 2024-10-08T21:38:43.8125059Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:38:43.8131369Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b4cd52a2d575cc62.xml', '-x', '--reruns=2'] ... [2024-10-08 21:38:43.812235] 2024-10-08T21:38:57.6767067Z 2024-10-08T21:38:57.6769618Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_88b1e4b7f574e90f_.log 2024-10-08T21:38:57.6771253Z 2024-10-08T21:38:58.4639440Z Running test batch 'tests to run' cost 14.66 seconds 2024-10-08T21:38:59.2162422Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-10-08T21:38:59.2163292Z + [[ 1 == 1 ]] 2024-10-08T21:38:59.2163831Z + test_rpc 2024-10-08T21:38:59.2164431Z + echo 'Testing RPC C++ tests' 2024-10-08T21:38:59.2165164Z Testing RPC C++ tests 2024-10-08T21:38:59.2171657Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:59.2191793Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:59.2222933Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-10-08T21:38:59.2224505Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-10-08T21:38:59.3061111Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-10-08T21:38:59.3064063Z import pkg_resources 2024-10-08T21:39:03.5467347Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-10-08T21:39:03.5567696Z Found test times from artifacts 2024-10-08T21:39:03.5997410Z Found test times from artifacts 2024-10-08T21:39:03.6010531Z Running all tests 2024-10-08T21:39:03.6011284Z Running parallel tests on 2 processes 2024-10-08T21:39:03.6012132Z Name: tests to run (est. time: 0.0min) 2024-10-08T21:39:03.6012880Z Serial tests (1): 2024-10-08T21:39:03.6013464Z cpp/test_cpp_rpc 1/1 2024-10-08T21:39:03.6014079Z Parallel tests (0): 2024-10-08T21:39:03.6014720Z Name: excluded (est. time: 0.0min) 2024-10-08T21:39:03.6015591Z Serial tests (0): 2024-10-08T21:39:03.6016164Z Parallel tests (0): 2024-10-08T21:39:03.6066519Z Running cpp/test_cpp_rpc 1/1 ... [2024-10-08 21:39:03.606385] 2024-10-08T21:39:03.6067089Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-10-08T21:39:03.6071271Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '2', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-718cba9f729e0b2b.xml', '-x', '--reruns=2'] ... [2024-10-08 21:39:03.606785] 2024-10-08T21:39:13.9930882Z 2024-10-08T21:39:13.9932830Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_031af837f9e27be0_.log 2024-10-08T21:39:13.9938296Z 2024-10-08T21:39:14.7033146Z Running test batch 'tests to run' cost 11.1 seconds 2024-10-08T21:39:15.4599767Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-10-08T21:39:15.4600865Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-10-08T21:39:15.4602131Z docker exec -t "639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-10-08T21:39:15.4648261Z shell: /usr/bin/bash -e {0} 2024-10-08T21:39:15.4649037Z env: 2024-10-08T21:39:15.4649546Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:15.4650303Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:15.4651677Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:15.4652968Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:15.4653658Z AWS_REGION: us-east-1 2024-10-08T21:39:15.4654416Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:15.4655394Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:15.4666357Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:15.4666897Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:15.4667482Z ##[endgroup] 2024-10-08T21:39:15.5133870Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-10-08T21:39:15.5331557Z ##[group]Run cat test/**/*_toprint.log || true 2024-10-08T21:39:15.5331985Z cat test/**/*_toprint.log || true 2024-10-08T21:39:15.5358593Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:39:15.5358987Z env: 2024-10-08T21:39:15.5359211Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:15.5359535Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:15.5360107Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:15.5360646Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:15.5360955Z AWS_REGION: us-east-1 2024-10-08T21:39:15.5361300Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:15.5361686Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:15.5367278Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:15.5367719Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:15.5368194Z ##[endgroup] 2024-10-08T21:39:15.5474301Z cat: 'test/**/*_toprint.log': No such file or directory 2024-10-08T21:39:15.5519202Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-10-08T21:39:15.5520117Z kill "$MONITOR_SCRIPT_PID" 2024-10-08T21:39:15.5569890Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:39:15.5570799Z env: 2024-10-08T21:39:15.5571305Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:15.5572068Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:15.5573431Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:15.5574721Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:15.5575417Z AWS_REGION: us-east-1 2024-10-08T21:39:15.5576182Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:15.5577120Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:15.5587021Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:15.5587626Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:15.5588834Z MONITOR_SCRIPT_PID: 2519079 2024-10-08T21:39:15.5589638Z ##[endgroup] 2024-10-08T21:39:15.5971686Z Prepare all required actions 2024-10-08T21:39:15.5972571Z Getting action download info 2024-10-08T21:39:16.1425577Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-10-08T21:39:16.7642497Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-10-08T21:39:16.7642869Z with: 2024-10-08T21:39:16.7643076Z use-gha: true 2024-10-08T21:39:16.7643418Z file-suffix: test-distributed-1-1-linux.rocm.gpu_31258392692 2024-10-08T21:39:16.7643837Z s3-bucket: gha-artifacts 2024-10-08T21:39:16.7644093Z env: 2024-10-08T21:39:16.7644304Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:16.7644612Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:16.7645171Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:16.7645723Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:16.7646007Z AWS_REGION: us-east-1 2024-10-08T21:39:16.7646363Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:16.7646751Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:16.7652380Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:16.7652817Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:16.7653288Z ##[endgroup] 2024-10-08T21:39:16.7721878Z ##[group]Run actions/upload-artifact@v3 2024-10-08T21:39:16.7722196Z with: 2024-10-08T21:39:16.7722614Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip 2024-10-08T21:39:16.7723133Z retention-days: 14 2024-10-08T21:39:16.7723391Z if-no-files-found: warn 2024-10-08T21:39:16.7723660Z path: test/**/*.json 2024-10-08T21:39:16.7723917Z include-hidden-files: false 2024-10-08T21:39:16.7724182Z env: 2024-10-08T21:39:16.7724391Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:16.7724698Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:16.7725267Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:16.7725813Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:16.7726097Z AWS_REGION: us-east-1 2024-10-08T21:39:16.7726399Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:16.7726775Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:16.7732420Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:16.7732844Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:16.7733305Z ##[endgroup] 2024-10-08T21:39:17.1746153Z With the provided path, there will be 10 files uploaded 2024-10-08T21:39:17.1747630Z Starting artifact upload 2024-10-08T21:39:17.1750271Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-10-08T21:39:17.1752664Z Artifact name is valid! 2024-10-08T21:39:17.5315441Z Container for artifact "test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip" successfully created. Starting upload of file(s) 2024-10-08T21:39:18.4233362Z Total size of all the files uploaded is 41148 bytes 2024-10-08T21:39:18.4234630Z File upload process has finished. Finalizing the artifact upload 2024-10-08T21:39:18.5079687Z Artifact has been finalized. All files have been successfully uploaded! 2024-10-08T21:39:18.5080618Z 2024-10-08T21:39:18.5081224Z The raw size of all the files that were specified for upload is 361701 bytes 2024-10-08T21:39:18.5083436Z The size of all the files that were uploaded is 41148 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-10-08T21:39:18.5085019Z 2024-10-08T21:39:18.5086855Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-10-08T21:39:18.5088665Z 2024-10-08T21:39:18.5090136Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip has been successfully uploaded! 2024-10-08T21:39:18.5317788Z ##[group]Run actions/upload-artifact@v3 2024-10-08T21:39:18.5318575Z with: 2024-10-08T21:39:18.5319590Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip 2024-10-08T21:39:18.5320839Z retention-days: 14 2024-10-08T21:39:18.5321438Z if-no-files-found: ignore 2024-10-08T21:39:18.5322114Z path: test/**/*.xml test/**/*.csv 2024-10-08T21:39:18.5322848Z include-hidden-files: false 2024-10-08T21:39:18.5323464Z env: 2024-10-08T21:39:18.5323945Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:18.5324671Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:18.5326023Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:18.5327301Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:18.5327974Z AWS_REGION: us-east-1 2024-10-08T21:39:18.5328708Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:18.5329785Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:18.5344415Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:18.5345455Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:18.5346566Z ##[endgroup] 2024-10-08T21:39:18.9697759Z With the provided path, there will be 219 files uploaded 2024-10-08T21:39:18.9699212Z Starting artifact upload 2024-10-08T21:39:18.9701869Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-10-08T21:39:18.9704299Z Artifact name is valid! 2024-10-08T21:39:19.3369114Z Container for artifact "test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip" successfully created. Starting upload of file(s) 2024-10-08T21:39:29.3434208Z Total file count: 219 ---- Processed file #159 (72.6%) 2024-10-08T21:39:33.1108046Z Total size of all the files uploaded is 54350 bytes 2024-10-08T21:39:33.1109336Z File upload process has finished. Finalizing the artifact upload 2024-10-08T21:39:33.1905185Z Artifact has been finalized. All files have been successfully uploaded! 2024-10-08T21:39:33.1906124Z 2024-10-08T21:39:33.1906721Z The raw size of all the files that were specified for upload is 83177 bytes 2024-10-08T21:39:33.1908919Z The size of all the files that were uploaded is 54350 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-10-08T21:39:33.1910497Z 2024-10-08T21:39:33.1912340Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-10-08T21:39:33.1914150Z 2024-10-08T21:39:33.1915459Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip has been successfully uploaded! 2024-10-08T21:39:33.2136486Z ##[group]Run actions/upload-artifact@v3 2024-10-08T21:39:33.2137244Z with: 2024-10-08T21:39:33.2138198Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip 2024-10-08T21:39:33.2139844Z retention-days: 14 2024-10-08T21:39:33.2140448Z if-no-files-found: ignore 2024-10-08T21:39:33.2141131Z path: usage_log.txt test/**/*.log 2024-10-08T21:39:33.2141916Z include-hidden-files: false 2024-10-08T21:39:33.2142678Z env: 2024-10-08T21:39:33.2143253Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:33.2144125Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:33.2145745Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:33.2147301Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:33.2147826Z AWS_REGION: us-east-1 2024-10-08T21:39:33.2148268Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:33.2148741Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:33.2155732Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:33.2156283Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:33.2156869Z ##[endgroup] 2024-10-08T21:39:33.6461319Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-10-08T21:39:33.6464764Z The least common ancestor is /home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-10-08T21:39:33.6466597Z With the provided path, there will be 9 files uploaded 2024-10-08T21:39:33.6467972Z Starting artifact upload 2024-10-08T21:39:33.6470617Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-10-08T21:39:33.6473013Z Artifact name is valid! 2024-10-08T21:39:34.0007577Z Container for artifact "logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip" successfully created. Starting upload of file(s) 2024-10-08T21:39:34.8678368Z Total size of all the files uploaded is 192917 bytes 2024-10-08T21:39:34.8679692Z File upload process has finished. Finalizing the artifact upload 2024-10-08T21:39:34.9468656Z Artifact has been finalized. All files have been successfully uploaded! 2024-10-08T21:39:34.9469547Z 2024-10-08T21:39:34.9470178Z The raw size of all the files that were specified for upload is 5294306 bytes 2024-10-08T21:39:34.9472381Z The size of all the files that were uploaded is 192917 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-10-08T21:39:34.9473970Z 2024-10-08T21:39:34.9476005Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-10-08T21:39:34.9478142Z 2024-10-08T21:39:34.9479528Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_31258392692.zip has been successfully uploaded! 2024-10-08T21:39:34.9699109Z ##[group]Run # shellcheck disable=SC2156 2024-10-08T21:39:34.9700014Z # shellcheck disable=SC2156 2024-10-08T21:39:34.9701433Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-10-08T21:39:34.9756844Z shell: /usr/bin/bash -e {0} 2024-10-08T21:39:34.9757488Z env: 2024-10-08T21:39:34.9757986Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:34.9758745Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:34.9760130Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:34.9761421Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:34.9762108Z AWS_REGION: us-east-1 2024-10-08T21:39:34.9762969Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:34.9764046Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:34.9778462Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:34.9779535Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:34.9780663Z ##[endgroup] 2024-10-08T21:39:35.2513796Z Prepare all required actions 2024-10-08T21:39:35.2514800Z Getting action download info 2024-10-08T21:39:35.2582007Z ##[group]Run ./.github/actions/teardown-rocm 2024-10-08T21:39:35.2583219Z env: 2024-10-08T21:39:35.2583722Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:35.2584493Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:35.2585882Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:35.2587180Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:35.2587859Z AWS_REGION: us-east-1 2024-10-08T21:39:35.2588592Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:35.2589497Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:35.2603621Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:35.2604652Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:35.2605771Z ##[endgroup] 2024-10-08T21:39:35.2642388Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T21:39:35.2643775Z # ignore expansion of "docker ps -q" since it could be empty 2024-10-08T21:39:35.2644844Z # shellcheck disable=SC2046 2024-10-08T21:39:35.2645682Z docker stop $(docker ps -q) || true 2024-10-08T21:39:35.2646541Z # Prune all stopped containers. 2024-10-08T21:39:35.2647363Z docker container prune -f 2024-10-08T21:39:35.2702588Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:39:35.2703491Z env: 2024-10-08T21:39:35.2703999Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:35.2704746Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:35.2706096Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:35.2707388Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:35.2708070Z AWS_REGION: us-east-1 2024-10-08T21:39:35.2708800Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:35.2709709Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:35.2723852Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:35.2724881Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:35.2726001Z ##[endgroup] 2024-10-08T21:39:35.7300343Z 639ea538f71e 2024-10-08T21:39:52.7759782Z Deleted Containers: 2024-10-08T21:39:52.7760960Z 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:52.7761976Z 2024-10-08T21:39:52.7762319Z Total reclaimed space: 8.112GB 2024-10-08T21:39:52.7838089Z Prepare all required actions 2024-10-08T21:39:52.7898236Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-10-08T21:39:52.7898656Z with: 2024-10-08T21:39:52.7898922Z diskspace-cutoff: 70 2024-10-08T21:39:52.7899218Z env: 2024-10-08T21:39:52.7899476Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:52.7899858Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:52.7900560Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:52.7901244Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:52.7901601Z AWS_REGION: us-east-1 2024-10-08T21:39:52.7901991Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:52.7902484Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:52.7911241Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:52.7912269Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:52.7913381Z ##[endgroup] 2024-10-08T21:39:52.7941660Z ##[group]Run set -ex 2024-10-08T21:39:52.7941980Z set -ex 2024-10-08T21:39:52.7942286Z diskspace_cutoff=70 2024-10-08T21:39:52.7942759Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-10-08T21:39:52.7943553Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-10-08T21:39:52.7944726Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-10-08T21:39:52.7945995Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-10-08T21:39:52.7946498Z  docker system prune -af 2024-10-08T21:39:52.7947163Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-10-08T21:39:52.7948251Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-10-08T21:39:52.7949210Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-10-08T21:39:52.7950008Z  echo "$msg" 2024-10-08T21:39:52.7950349Z  exit 1 2024-10-08T21:39:52.7950661Z  else 2024-10-08T21:39:52.7951041Z  difference=$((diskspace - diskspace_new)) 2024-10-08T21:39:52.7951579Z  echo "Diskspace saved: $difference percent" 2024-10-08T21:39:52.7952029Z  fi 2024-10-08T21:39:52.7952295Z fi 2024-10-08T21:39:52.7995920Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-10-08T21:39:52.7996843Z env: 2024-10-08T21:39:52.7997333Z GIT_DEFAULT_BRANCH: main 2024-10-08T21:39:52.7998073Z DOCKER_HOST: unix:///run/user/1001/docker.sock 2024-10-08T21:39:52.7999475Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-10-08T21:39:52.8000758Z AWS_DEFAULT_REGION: us-east-1 2024-10-08T21:39:52.8001438Z AWS_REGION: us-east-1 2024-10-08T21:39:52.8002154Z AWS_ACCESS_KEY_ID: *** 2024-10-08T21:39:52.8003059Z AWS_SECRET_ACCESS_KEY: *** 2024-10-08T21:39:52.8016760Z AWS_SESSION_TOKEN: *** 2024-10-08T21:39:52.8017816Z CONTAINER_NAME: 639ea538f71e5db6fa0e834fcae4aba265218099174c1002d961005e5ce928a6 2024-10-08T21:39:52.8019126Z ##[endgroup] 2024-10-08T21:39:52.8081864Z + diskspace_cutoff=70 2024-10-08T21:39:52.8094389Z ++ docker info -f '{{.DockerRootDir}}' 2024-10-08T21:39:52.8585066Z + docker_root_dir=/home/pytorchci/.local/share/docker 2024-10-08T21:39:52.8591587Z ++ df -H --output=pcent /home/pytorchci/.local/share/docker 2024-10-08T21:39:52.8593168Z ++ sed -n 2p 2024-10-08T21:39:52.8593522Z ++ sed s/%// 2024-10-08T21:39:52.8594591Z ++ sed 's/ //' 2024-10-08T21:39:52.8613503Z + diskspace=47 2024-10-08T21:39:52.8614556Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-10-08T21:39:52.8615483Z + [[ 47 -ge 70 ]] 2024-10-08T21:39:52.8687032Z Post job cleanup. 2024-10-08T21:39:52.8746067Z Post job cleanup. 2024-10-08T21:39:52.9068993Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-10-08T21:39:52.9398010Z Post job cleanup. 2024-10-08T21:39:53.0613552Z Post job cleanup. 2024-10-08T21:39:53.0732854Z Post job cleanup. 2024-10-08T21:39:53.0832828Z Post job cleanup. 2024-10-08T21:39:53.1491683Z [command]/usr/bin/git version 2024-10-08T21:39:53.1523994Z git version 2.34.1 2024-10-08T21:39:53.1558774Z Temporarily overriding HOME='/home/pytorchci/actions-runner/_work/_temp/607ec4b5-83a4-4f25-88a0-3e4c5f5c1865' before making global git config changes 2024-10-08T21:39:53.1561201Z Adding repository directory to the temporary git global config as a safe directory 2024-10-08T21:39:53.1563289Z [command]/usr/bin/git config --global --add safe.directory /home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-10-08T21:39:53.1586941Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-10-08T21:39:53.1624382Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-10-08T21:39:53.1927794Z Entering 'android/libs/fbjni' 2024-10-08T21:39:53.1989060Z Entering 'third_party/FP16' 2024-10-08T21:39:53.2031127Z Entering 'third_party/FXdiv' 2024-10-08T21:39:53.2075356Z Entering 'third_party/NNPACK' 2024-10-08T21:39:53.2117301Z Entering 'third_party/NVTX' 2024-10-08T21:39:53.2165487Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T21:39:53.2210676Z Entering 'third_party/XNNPACK' 2024-10-08T21:39:53.2272430Z Entering 'third_party/benchmark' 2024-10-08T21:39:53.2314695Z Entering 'third_party/cpp-httplib' 2024-10-08T21:39:53.2356320Z Entering 'third_party/cpuinfo' 2024-10-08T21:39:53.2401268Z Entering 'third_party/cudnn_frontend' 2024-10-08T21:39:53.2454087Z Entering 'third_party/cutlass' 2024-10-08T21:39:53.2505251Z Entering 'third_party/eigen' 2024-10-08T21:39:53.2557699Z Entering 'third_party/fbgemm' 2024-10-08T21:39:53.2601710Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T21:39:53.2645070Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T21:39:53.2689355Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T21:39:53.2736324Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T21:39:53.2782965Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T21:39:53.2827007Z Entering 'third_party/flatbuffers' 2024-10-08T21:39:53.2874566Z Entering 'third_party/fmt' 2024-10-08T21:39:53.2915417Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T21:39:53.2964581Z Entering 'third_party/gloo' 2024-10-08T21:39:53.3008512Z Entering 'third_party/googletest' 2024-10-08T21:39:53.3050526Z Entering 'third_party/ideep' 2024-10-08T21:39:53.3090121Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T21:39:53.3138734Z Entering 'third_party/ittapi' 2024-10-08T21:39:53.3179224Z Entering 'third_party/kineto' 2024-10-08T21:39:53.3224221Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T21:39:53.3277561Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T21:39:53.3319534Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T21:39:53.3359450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T21:39:53.3402682Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T21:39:53.3440902Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T21:39:53.3500351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T21:39:53.3540629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T21:39:53.3579483Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T21:39:53.3637403Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T21:39:53.3694747Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T21:39:53.3735876Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T21:39:53.3777337Z Entering 'third_party/mimalloc' 2024-10-08T21:39:53.3819332Z Entering 'third_party/nccl/nccl' 2024-10-08T21:39:53.3859818Z Entering 'third_party/nlohmann' 2024-10-08T21:39:53.3902110Z Entering 'third_party/onnx' 2024-10-08T21:39:53.3956082Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T21:39:53.4005696Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T21:39:53.4055167Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T21:39:53.4097629Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T21:39:53.4142614Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T21:39:53.4187045Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T21:39:53.4224177Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T21:39:53.4262607Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T21:39:53.4304399Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T21:39:53.4347225Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T21:39:53.4383156Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T21:39:53.4422936Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T21:39:53.4462638Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T21:39:53.4521727Z Entering 'third_party/pocketfft' 2024-10-08T21:39:53.4565812Z Entering 'third_party/protobuf' 2024-10-08T21:39:53.4625889Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T21:39:53.4673776Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T21:39:53.4720603Z Entering 'third_party/psimd' 2024-10-08T21:39:53.4767089Z Entering 'third_party/pthreadpool' 2024-10-08T21:39:53.4816767Z Entering 'third_party/pybind11' 2024-10-08T21:39:53.4860785Z Entering 'third_party/python-peachpy' 2024-10-08T21:39:53.4908409Z Entering 'third_party/sleef' 2024-10-08T21:39:53.4954367Z Entering 'third_party/tensorpipe' 2024-10-08T21:39:53.4995854Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T21:39:53.5048449Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T21:39:53.5096510Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T21:39:53.5140339Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T21:39:53.5178399Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T21:39:53.5253071Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-10-08T21:39:53.5274007Z http.https://github.com/.extraheader 2024-10-08T21:39:53.5285242Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-10-08T21:39:53.5322213Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-10-08T21:39:53.5596799Z Entering 'android/libs/fbjni' 2024-10-08T21:39:53.5625045Z http.https://github.com/.extraheader 2024-10-08T21:39:53.5726327Z Entering 'third_party/FP16' 2024-10-08T21:39:53.5750064Z http.https://github.com/.extraheader 2024-10-08T21:39:53.5786948Z Entering 'third_party/FXdiv' 2024-10-08T21:39:53.5814410Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6025160Z Entering 'third_party/NNPACK' 2024-10-08T21:39:53.6054576Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6083502Z Entering 'third_party/NVTX' 2024-10-08T21:39:53.6104237Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6345414Z Entering 'third_party/VulkanMemoryAllocator' 2024-10-08T21:39:53.6369398Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6404272Z Entering 'third_party/XNNPACK' 2024-10-08T21:39:53.6426159Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6476186Z Entering 'third_party/benchmark' 2024-10-08T21:39:53.6497237Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6527928Z Entering 'third_party/cpp-httplib' 2024-10-08T21:39:53.6558786Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6593745Z Entering 'third_party/cpuinfo' 2024-10-08T21:39:53.6615421Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6647736Z Entering 'third_party/cudnn_frontend' 2024-10-08T21:39:53.6672161Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6702282Z Entering 'third_party/cutlass' 2024-10-08T21:39:53.6723489Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6759824Z Entering 'third_party/eigen' 2024-10-08T21:39:53.6781340Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6832862Z Entering 'third_party/fbgemm' 2024-10-08T21:39:53.6856218Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6886969Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-10-08T21:39:53.6919226Z http.https://github.com/.extraheader 2024-10-08T21:39:53.6954086Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-10-08T21:39:53.6973659Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7008875Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-10-08T21:39:53.7029591Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7066833Z Entering 'third_party/fbgemm/third_party/googletest' 2024-10-08T21:39:53.7090513Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7130162Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-10-08T21:39:53.7152536Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7191274Z Entering 'third_party/flatbuffers' 2024-10-08T21:39:53.7212934Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7252136Z Entering 'third_party/fmt' 2024-10-08T21:39:53.7276099Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7309794Z Entering 'third_party/gemmlowp/gemmlowp' 2024-10-08T21:39:53.7334836Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7369601Z Entering 'third_party/gloo' 2024-10-08T21:39:53.7390597Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7424309Z Entering 'third_party/googletest' 2024-10-08T21:39:53.7447772Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7486008Z Entering 'third_party/ideep' 2024-10-08T21:39:53.7512888Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7546165Z Entering 'third_party/ideep/mkl-dnn' 2024-10-08T21:39:53.7569546Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7618031Z Entering 'third_party/ittapi' 2024-10-08T21:39:53.7639621Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7672887Z Entering 'third_party/kineto' 2024-10-08T21:39:53.7694808Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7726119Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-10-08T21:39:53.7754311Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7791172Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-10-08T21:39:53.7815222Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7846227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-10-08T21:39:53.7868976Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7906396Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-10-08T21:39:53.7933871Z http.https://github.com/.extraheader 2024-10-08T21:39:53.7965334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-10-08T21:39:53.7988683Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8024795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-10-08T21:39:53.8050330Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8086559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-10-08T21:39:53.8108707Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8142825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-10-08T21:39:53.8162187Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8202829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-10-08T21:39:53.8228515Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8274791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-10-08T21:39:53.8295545Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8330454Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-10-08T21:39:53.8352569Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8389716Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-10-08T21:39:53.8418816Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8465359Z Entering 'third_party/mimalloc' 2024-10-08T21:39:53.8489805Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8534477Z Entering 'third_party/nccl/nccl' 2024-10-08T21:39:53.8558449Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8602557Z Entering 'third_party/nlohmann' 2024-10-08T21:39:53.8625166Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8660133Z Entering 'third_party/onnx' 2024-10-08T21:39:53.8681084Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8725572Z Entering 'third_party/onnx/third_party/benchmark' 2024-10-08T21:39:53.8759606Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8801651Z Entering 'third_party/onnx/third_party/pybind11' 2024-10-08T21:39:53.8835013Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8877838Z Entering 'third_party/opentelemetry-cpp' 2024-10-08T21:39:53.8899033Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8934760Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-10-08T21:39:53.8964816Z http.https://github.com/.extraheader 2024-10-08T21:39:53.8998343Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-10-08T21:39:53.9014930Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9047608Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-10-08T21:39:53.9078377Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9113430Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-10-08T21:39:53.9134790Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9166508Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-10-08T21:39:53.9186250Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9236463Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-10-08T21:39:53.9254367Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9286756Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-10-08T21:39:53.9315043Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9346315Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-10-08T21:39:53.9372669Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9406858Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-10-08T21:39:53.9434738Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9471520Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-10-08T21:39:53.9494800Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9544243Z Entering 'third_party/pocketfft' 2024-10-08T21:39:53.9565280Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9603409Z Entering 'third_party/protobuf' 2024-10-08T21:39:53.9635433Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9666889Z Entering 'third_party/protobuf/third_party/benchmark' 2024-10-08T21:39:53.9695258Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9728375Z Entering 'third_party/protobuf/third_party/googletest' 2024-10-08T21:39:53.9755006Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9788983Z Entering 'third_party/psimd' 2024-10-08T21:39:53.9814944Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9848743Z Entering 'third_party/pthreadpool' 2024-10-08T21:39:53.9876524Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9904646Z Entering 'third_party/pybind11' 2024-10-08T21:39:53.9934072Z http.https://github.com/.extraheader 2024-10-08T21:39:53.9973967Z Entering 'third_party/python-peachpy' 2024-10-08T21:39:53.9992925Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0026949Z Entering 'third_party/sleef' 2024-10-08T21:39:54.0054146Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0084394Z Entering 'third_party/tensorpipe' 2024-10-08T21:39:54.0109856Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0142242Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-10-08T21:39:54.0163576Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0196651Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-10-08T21:39:54.0217140Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0248691Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-10-08T21:39:54.0270804Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0304509Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-10-08T21:39:54.0326833Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0359539Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-10-08T21:39:54.0380040Z http.https://github.com/.extraheader 2024-10-08T21:39:54.0700556Z Cleaning up orphan processes