2024-11-01T16:44:50.0972760Z Current runner version: '2.320.0' 2024-11-01T16:44:50.0985117Z Runner name: 'gpu948c.jax.cs.cpe.ice.amd.com' 2024-11-01T16:44:50.0986873Z Runner group name: 'linux.rocm.gpu.group' 2024-11-01T16:44:50.0988723Z Machine name: 'gpu948c' 2024-11-01T16:44:50.1005803Z Testing runner upgrade compatibility 2024-11-01T16:44:50.2267327Z ##[group]GITHUB_TOKEN Permissions 2024-11-01T16:44:50.2269089Z Contents: read 2024-11-01T16:44:50.2269716Z Metadata: read 2024-11-01T16:44:50.2270103Z ##[endgroup] 2024-11-01T16:44:50.2273726Z Secret source: Actions 2024-11-01T16:44:50.2274529Z Prepare workflow directory 2024-11-01T16:44:50.8150295Z Prepare all required actions 2024-11-01T16:44:50.8416224Z Getting action download info 2024-11-01T16:44:51.2434686Z Download action repository 'pytorch/pytorch@main' (SHA:d79c5143d8e625947b93f8821bbe181419f47dee) 2024-11-01T16:44:57.0231650Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-11-01T16:44:57.5022854Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-11-01T16:44:57.9640402Z Download action repository 'pytorch/test-infra@main' (SHA:49fb39b5efb49007791d74b09885044cd64544bf) 2024-11-01T16:44:58.8620101Z Download action repository 'actions/upload-artifact@v3' (SHA:ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5) 2024-11-01T16:44:59.3281974Z Getting action download info 2024-11-01T16:44:59.4946109Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-11-01T16:44:59.9971012Z Getting action download info 2024-11-01T16:45:00.1295539Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-11-01T16:45:00.5758723Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/138766 (d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea) 2024-11-01T16:45:00.5761264Z ##[group] Inputs 2024-11-01T16:45:00.5761734Z build-environment: linux-focal-rocm6.2-py3.10 2024-11-01T16:45:00.5763016Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-11-01T16:45:00.5764654Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:45:00.5765474Z sync-tag: 2024-11-01T16:45:00.5766258Z timeout-minutes: 300 2024-11-01T16:45:00.5767250Z tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-11-01T16:45:00.5768263Z ##[endgroup] 2024-11-01T16:45:00.5769015Z Complete job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:45:00.8067336Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-11-01T16:45:00.8067888Z with: 2024-11-01T16:45:00.8068141Z no-sudo: true 2024-11-01T16:45:00.8068425Z submodules: recursive 2024-11-01T16:45:00.8068723Z fetch-depth: 0 2024-11-01T16:45:00.8068998Z env: 2024-11-01T16:45:00.8069252Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:00.8069559Z ##[endgroup] 2024-11-01T16:45:00.8291343Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:00.8292627Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-11-01T16:45:00.8320535Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:00.8320989Z env: 2024-11-01T16:45:00.8321248Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:00.8321563Z ##[endgroup] 2024-11-01T16:45:00.8879618Z ##[group]Run retry () { 2024-11-01T16:45:00.8880419Z retry () { 2024-11-01T16:45:00.8881775Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-11-01T16:45:00.8882739Z } 2024-11-01T16:45:00.8883263Z echo "${GITHUB_WORKSPACE}" 2024-11-01T16:45:00.8883989Z if [ -z "${NO_SUDO}" ]; then 2024-11-01T16:45:00.8884837Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:45:00.8885634Z else 2024-11-01T16:45:00.8886205Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-11-01T16:45:00.8886939Z fi 2024-11-01T16:45:00.8887461Z mkdir "${GITHUB_WORKSPACE}" 2024-11-01T16:45:00.8926629Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:45:00.8927506Z env: 2024-11-01T16:45:00.8927981Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:00.8928560Z NO_SUDO: true 2024-11-01T16:45:00.8929041Z ##[endgroup] 2024-11-01T16:45:00.9060836Z /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T16:45:05.6776680Z ##[group]Run malfet/checkout@silent-checkout 2024-11-01T16:45:05.6777579Z with: 2024-11-01T16:45:05.6778144Z ref: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:45:05.6778912Z fetch-depth: 0 2024-11-01T16:45:05.6779439Z submodules: recursive 2024-11-01T16:45:05.6779999Z quiet-checkout: true 2024-11-01T16:45:05.6780580Z repository: pytorch/pytorch 2024-11-01T16:45:05.6781450Z token: *** 2024-11-01T16:45:05.6781960Z ssh-strict: true 2024-11-01T16:45:05.6782513Z persist-credentials: true 2024-11-01T16:45:05.6783104Z clean: true 2024-11-01T16:45:05.6783665Z sparse-checkout-cone-mode: true 2024-11-01T16:45:05.6784324Z lfs: false 2024-11-01T16:45:05.6784811Z set-safe-directory: true 2024-11-01T16:45:05.6785386Z env: 2024-11-01T16:45:05.6785841Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:45:05.6786396Z ##[endgroup] 2024-11-01T16:45:05.8261991Z Syncing repository: pytorch/pytorch 2024-11-01T16:45:05.8264774Z ##[group]Getting Git version info 2024-11-01T16:45:05.8266049Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:45:05.8267472Z [command]/usr/bin/git version 2024-11-01T16:45:05.8268012Z git version 2.34.1 2024-11-01T16:45:05.8274913Z ##[endgroup] 2024-11-01T16:45:05.8306906Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/c374704f-daaf-4228-8324-3edd6b377c94' before making global git config changes 2024-11-01T16:45:05.8309373Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T16:45:05.8311512Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T16:45:05.8366324Z Deleting the contents of '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-11-01T16:45:05.8372374Z ##[group]Initializing the repository 2024-11-01T16:45:05.8376872Z [command]/usr/bin/git init /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T16:45:05.8439245Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-11-01T16:45:05.8440911Z hint: is subject to change. To configure the initial branch name to use in all 2024-11-01T16:45:05.8442422Z hint: of your new repositories, which will suppress this warning, call: 2024-11-01T16:45:05.8443435Z hint: 2024-11-01T16:45:05.8444253Z hint: git config --global init.defaultBranch 2024-11-01T16:45:05.8445139Z hint: 2024-11-01T16:45:05.8446053Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-11-01T16:45:05.8447597Z hint: 'development'. The just-created branch can be renamed via this command: 2024-11-01T16:45:05.8448667Z hint: 2024-11-01T16:45:05.8449220Z hint: git branch -m 2024-11-01T16:45:05.8453096Z Initialized empty Git repository in /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-11-01T16:45:05.8469320Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-11-01T16:45:05.8542005Z ##[endgroup] 2024-11-01T16:45:05.8543268Z ##[group]Disabling automatic garbage collection 2024-11-01T16:45:05.8544684Z [command]/usr/bin/git config --local gc.auto 0 2024-11-01T16:45:05.8604640Z ##[endgroup] 2024-11-01T16:45:05.8605562Z ##[group]Setting up auth 2024-11-01T16:45:05.8613353Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T16:45:05.8669718Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T16:45:05.9259743Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T16:45:05.9317636Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T16:45:05.9896874Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:45:05.9968717Z ##[endgroup] 2024-11-01T16:45:05.9969761Z ##[group]Fetching the repository 2024-11-01T16:45:05.9981881Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-11-01T16:45:09.7057492Z remote: Enumerating objects: 1053138 2024-11-01T16:45:09.7058450Z remote: Enumerating objects: 1053739, done. 2024-11-01T16:45:09.7059405Z remote: Counting objects: 0% (1/601) 2024-11-01T16:45:09.7060259Z remote: Counting objects: 1% (7/601) 2024-11-01T16:45:09.7061087Z remote: Counting objects: 2% (13/601) 2024-11-01T16:45:09.7061954Z remote: Counting objects: 3% (19/601) 2024-11-01T16:45:09.7062783Z remote: Counting objects: 4% (25/601) 2024-11-01T16:45:09.7063588Z remote: Counting objects: 5% (31/601) 2024-11-01T16:45:09.7064459Z remote: Counting objects: 6% (37/601) 2024-11-01T16:45:09.7065293Z remote: Counting objects: 7% (43/601) 2024-11-01T16:45:09.7066177Z remote: Counting objects: 8% (49/601) 2024-11-01T16:45:09.7067109Z remote: Counting objects: 9% (55/601) 2024-11-01T16:45:09.7333445Z remote: Counting objects: 10% (61/601) 2024-11-01T16:45:09.7334479Z remote: Counting objects: 11% (67/601) 2024-11-01T16:45:09.7335460Z remote: Counting objects: 12% (73/601) 2024-11-01T16:45:09.7336596Z remote: Counting objects: 13% (79/601) 2024-11-01T16:45:09.7337453Z remote: Counting objects: 14% (85/601) 2024-11-01T16:45:09.7338285Z remote: Counting objects: 15% (91/601) 2024-11-01T16:45:09.7339097Z remote: Counting objects: 16% (97/601) 2024-11-01T16:45:09.7339900Z remote: Counting objects: 17% (103/601) 2024-11-01T16:45:09.7340748Z remote: Counting objects: 18% (109/601) 2024-11-01T16:45:09.7341579Z remote: Counting objects: 19% (115/601) 2024-11-01T16:45:09.7343160Z remote: Counting objects: 20% (121/601) 2024-11-01T16:45:09.7344030Z remote: Counting objects: 21% (127/601) 2024-11-01T16:45:09.7344860Z remote: Counting objects: 22% (133/601) 2024-11-01T16:45:09.7345673Z remote: Counting objects: 23% (139/601) 2024-11-01T16:45:09.7346488Z remote: Counting objects: 24% (145/601) 2024-11-01T16:45:09.7347293Z remote: Counting objects: 25% (151/601) 2024-11-01T16:45:09.7348104Z remote: Counting objects: 26% (157/601) 2024-11-01T16:45:09.7348886Z remote: Counting objects: 27% (163/601) 2024-11-01T16:45:09.7349687Z remote: Counting objects: 28% (169/601) 2024-11-01T16:45:09.7350490Z remote: Counting objects: 29% (175/601) 2024-11-01T16:45:09.7351286Z remote: Counting objects: 30% (181/601) 2024-11-01T16:45:09.7352076Z remote: Counting objects: 31% (187/601) 2024-11-01T16:45:09.7352875Z remote: Counting objects: 32% (193/601) 2024-11-01T16:45:09.7353670Z remote: Counting objects: 33% (199/601) 2024-11-01T16:45:09.7354556Z remote: Counting objects: 34% (205/601) 2024-11-01T16:45:09.7355707Z remote: Counting objects: 35% (211/601) 2024-11-01T16:45:09.7356495Z remote: Counting objects: 36% (217/601) 2024-11-01T16:45:09.7357281Z remote: Counting objects: 37% (223/601) 2024-11-01T16:45:09.7358071Z remote: Counting objects: 38% (229/601) 2024-11-01T16:45:09.7358859Z remote: Counting objects: 39% (235/601) 2024-11-01T16:45:09.7359680Z remote: Counting objects: 40% (241/601) 2024-11-01T16:45:09.7360521Z remote: Counting objects: 41% (247/601) 2024-11-01T16:45:09.7361351Z remote: Counting objects: 42% (253/601) 2024-11-01T16:45:09.7362149Z remote: Counting objects: 43% (259/601) 2024-11-01T16:45:09.7362969Z remote: Counting objects: 44% (265/601) 2024-11-01T16:45:09.7363779Z remote: Counting objects: 45% (271/601) 2024-11-01T16:45:09.7364569Z remote: Counting objects: 46% (277/601) 2024-11-01T16:45:09.7365359Z remote: Counting objects: 47% (283/601) 2024-11-01T16:45:09.7366162Z remote: Counting objects: 48% (289/601) 2024-11-01T16:45:09.7366946Z remote: Counting objects: 49% (295/601) 2024-11-01T16:45:09.7367733Z remote: Counting objects: 50% (301/601) 2024-11-01T16:45:09.7368532Z remote: Counting objects: 51% (307/601) 2024-11-01T16:45:09.7369334Z remote: Counting objects: 52% (313/601) 2024-11-01T16:45:09.7370116Z remote: Counting objects: 53% (319/601) 2024-11-01T16:45:09.7370914Z remote: Counting objects: 54% (325/601) 2024-11-01T16:45:09.7371704Z remote: Counting objects: 55% (331/601) 2024-11-01T16:45:09.7372495Z remote: Counting objects: 56% (337/601) 2024-11-01T16:45:09.7373294Z remote: Counting objects: 57% (343/601) 2024-11-01T16:45:09.7374090Z remote: Counting objects: 58% (349/601) 2024-11-01T16:45:09.7374887Z remote: Counting objects: 59% (355/601) 2024-11-01T16:45:09.7375681Z remote: Counting objects: 60% (361/601) 2024-11-01T16:45:09.7376626Z remote: Counting objects: 61% (367/601) 2024-11-01T16:45:09.7377404Z remote: Counting objects: 62% (373/601) 2024-11-01T16:45:09.7378227Z remote: Counting objects: 63% (379/601) 2024-11-01T16:45:09.7379022Z remote: Counting objects: 64% (385/601) 2024-11-01T16:45:09.7379813Z remote: Counting objects: 65% (391/601) 2024-11-01T16:45:09.7380589Z remote: Counting objects: 66% (397/601) 2024-11-01T16:45:09.7381381Z remote: Counting objects: 67% (403/601) 2024-11-01T16:45:09.7382190Z remote: Counting objects: 68% (409/601) 2024-11-01T16:45:09.7382976Z remote: Counting objects: 69% (415/601) 2024-11-01T16:45:09.7383773Z remote: Counting objects: 70% (421/601) 2024-11-01T16:45:09.7384574Z remote: Counting objects: 71% (427/601) 2024-11-01T16:45:09.7385673Z remote: Counting objects: 72% (433/601) 2024-11-01T16:45:09.7386488Z remote: Counting objects: 73% (439/601) 2024-11-01T16:45:09.7387292Z remote: Counting objects: 74% (445/601) 2024-11-01T16:45:09.7388071Z remote: Counting objects: 75% (451/601) 2024-11-01T16:45:09.7388863Z remote: Counting objects: 76% (457/601) 2024-11-01T16:45:09.7389655Z remote: Counting objects: 77% (463/601) 2024-11-01T16:45:09.7390433Z remote: Counting objects: 78% (469/601) 2024-11-01T16:45:09.7391224Z remote: Counting objects: 79% (475/601) 2024-11-01T16:45:09.7392017Z remote: Counting objects: 80% (481/601) 2024-11-01T16:45:09.7392810Z remote: Counting objects: 81% (487/601) 2024-11-01T16:45:09.7393598Z remote: Counting objects: 82% (493/601) 2024-11-01T16:45:09.7394389Z remote: Counting objects: 83% (499/601) 2024-11-01T16:45:09.7395181Z remote: Counting objects: 84% (505/601) 2024-11-01T16:45:09.7395975Z remote: Counting objects: 85% (511/601) 2024-11-01T16:45:09.7396764Z remote: Counting objects: 86% (517/601) 2024-11-01T16:45:09.7397875Z remote: Counting objects: 87% (523/601) 2024-11-01T16:45:09.7398647Z remote: Counting objects: 88% (529/601) 2024-11-01T16:45:09.7399436Z remote: Counting objects: 89% (535/601) 2024-11-01T16:45:09.7400229Z remote: Counting objects: 90% (541/601) 2024-11-01T16:45:09.7401006Z remote: Counting objects: 91% (547/601) 2024-11-01T16:45:09.7401810Z remote: Counting objects: 92% (553/601) 2024-11-01T16:45:09.7402598Z remote: Counting objects: 93% (559/601) 2024-11-01T16:45:09.7403374Z remote: Counting objects: 94% (565/601) 2024-11-01T16:45:09.7404162Z remote: Counting objects: 95% (571/601) 2024-11-01T16:45:09.7404950Z remote: Counting objects: 96% (577/601) 2024-11-01T16:45:09.7405723Z remote: Counting objects: 97% (583/601) 2024-11-01T16:45:09.7406512Z remote: Counting objects: 98% (589/601) 2024-11-01T16:45:09.7407307Z remote: Counting objects: 99% (595/601) 2024-11-01T16:45:09.7408092Z remote: Counting objects: 100% (601/601) 2024-11-01T16:45:09.7408959Z remote: Counting objects: 100% (601/601), done. 2024-11-01T16:45:09.7409890Z remote: Compressing objects: 0% (1/287) 2024-11-01T16:45:09.7410757Z remote: Compressing objects: 1% (3/287) 2024-11-01T16:45:09.7411607Z remote: Compressing objects: 2% (6/287) 2024-11-01T16:45:09.7412468Z remote: Compressing objects: 3% (9/287) 2024-11-01T16:45:09.7413337Z remote: Compressing objects: 4% (12/287) 2024-11-01T16:45:09.7414206Z remote: Compressing objects: 5% (15/287) 2024-11-01T16:45:09.7415078Z remote: Compressing objects: 6% (18/287) 2024-11-01T16:45:09.7484849Z remote: Compressing objects: 7% (21/287) 2024-11-01T16:45:09.7638681Z remote: Compressing objects: 8% (23/287) 2024-11-01T16:45:09.7738326Z remote: Compressing objects: 9% (26/287) 2024-11-01T16:45:09.7770097Z remote: Compressing objects: 10% (29/287) 2024-11-01T16:45:09.7771013Z remote: Compressing objects: 11% (32/287) 2024-11-01T16:45:09.7771900Z remote: Compressing objects: 12% (35/287) 2024-11-01T16:45:09.7772773Z remote: Compressing objects: 13% (38/287) 2024-11-01T16:45:09.7773657Z remote: Compressing objects: 14% (41/287) 2024-11-01T16:45:09.7774519Z remote: Compressing objects: 15% (44/287) 2024-11-01T16:45:09.7775373Z remote: Compressing objects: 16% (46/287) 2024-11-01T16:45:09.7776228Z remote: Compressing objects: 17% (49/287) 2024-11-01T16:45:09.7777234Z remote: Compressing objects: 18% (52/287) 2024-11-01T16:45:09.7778101Z remote: Compressing objects: 19% (55/287) 2024-11-01T16:45:09.7778962Z remote: Compressing objects: 20% (58/287) 2024-11-01T16:45:09.7779814Z remote: Compressing objects: 21% (61/287) 2024-11-01T16:45:09.7787030Z remote: Compressing objects: 22% (64/287) 2024-11-01T16:45:09.7787934Z remote: Compressing objects: 23% (67/287) 2024-11-01T16:45:09.7788825Z remote: Compressing objects: 24% (69/287) 2024-11-01T16:45:09.7789657Z remote: Compressing objects: 25% (72/287) 2024-11-01T16:45:09.7790513Z remote: Compressing objects: 26% (75/287) 2024-11-01T16:45:09.7791352Z remote: Compressing objects: 27% (78/287) 2024-11-01T16:45:09.7792172Z remote: Compressing objects: 28% (81/287) 2024-11-01T16:45:09.7793022Z remote: Compressing objects: 29% (84/287) 2024-11-01T16:45:09.7793871Z remote: Compressing objects: 30% (87/287) 2024-11-01T16:45:09.7800306Z remote: Compressing objects: 31% (89/287) 2024-11-01T16:45:09.7803548Z remote: Compressing objects: 32% (92/287) 2024-11-01T16:45:09.7807727Z remote: Compressing objects: 33% (95/287) 2024-11-01T16:45:09.7810923Z remote: Compressing objects: 34% (98/287) 2024-11-01T16:45:09.7819747Z remote: Compressing objects: 35% (101/287) 2024-11-01T16:45:09.7823889Z remote: Compressing objects: 36% (104/287) 2024-11-01T16:45:09.7829205Z remote: Compressing objects: 37% (107/287) 2024-11-01T16:45:09.7835101Z remote: Compressing objects: 38% (110/287) 2024-11-01T16:45:09.7835994Z remote: Compressing objects: 39% (112/287) 2024-11-01T16:45:09.7842307Z remote: Compressing objects: 40% (115/287) 2024-11-01T16:45:09.7843176Z remote: Compressing objects: 41% (118/287) 2024-11-01T16:45:09.7846870Z remote: Compressing objects: 42% (121/287) 2024-11-01T16:45:09.7847745Z remote: Compressing objects: 43% (124/287) 2024-11-01T16:45:09.7848624Z remote: Compressing objects: 44% (127/287) 2024-11-01T16:45:09.7849474Z remote: Compressing objects: 45% (130/287) 2024-11-01T16:45:09.7854625Z remote: Compressing objects: 46% (133/287) 2024-11-01T16:45:09.7860597Z remote: Compressing objects: 47% (135/287) 2024-11-01T16:45:09.7871849Z remote: Compressing objects: 48% (138/287) 2024-11-01T16:45:09.7872720Z remote: Compressing objects: 49% (141/287) 2024-11-01T16:45:09.7873637Z remote: Compressing objects: 50% (144/287) 2024-11-01T16:45:09.7874492Z remote: Compressing objects: 51% (147/287) 2024-11-01T16:45:09.7875334Z remote: Compressing objects: 52% (150/287) 2024-11-01T16:45:09.7876196Z remote: Compressing objects: 53% (153/287) 2024-11-01T16:45:09.7877047Z remote: Compressing objects: 54% (155/287) 2024-11-01T16:45:09.7877886Z remote: Compressing objects: 55% (158/287) 2024-11-01T16:45:09.7878748Z remote: Compressing objects: 56% (161/287) 2024-11-01T16:45:09.7879592Z remote: Compressing objects: 57% (164/287) 2024-11-01T16:45:09.7880429Z remote: Compressing objects: 58% (167/287) 2024-11-01T16:45:09.7881277Z remote: Compressing objects: 59% (170/287) 2024-11-01T16:45:09.7882129Z remote: Compressing objects: 60% (173/287) 2024-11-01T16:45:09.7882977Z remote: Compressing objects: 61% (176/287) 2024-11-01T16:45:09.7883825Z remote: Compressing objects: 62% (178/287) 2024-11-01T16:45:09.7884685Z remote: Compressing objects: 63% (181/287) 2024-11-01T16:45:09.7885566Z remote: Compressing objects: 64% (184/287) 2024-11-01T16:45:09.7886432Z remote: Compressing objects: 65% (187/287) 2024-11-01T16:45:09.7887307Z remote: Compressing objects: 66% (190/287) 2024-11-01T16:45:09.7888176Z remote: Compressing objects: 67% (193/287) 2024-11-01T16:45:09.7889023Z remote: Compressing objects: 68% (196/287) 2024-11-01T16:45:09.7889888Z remote: Compressing objects: 69% (199/287) 2024-11-01T16:45:09.7890746Z remote: Compressing objects: 70% (201/287) 2024-11-01T16:45:09.7891591Z remote: Compressing objects: 71% (204/287) 2024-11-01T16:45:09.7892484Z remote: Compressing objects: 72% (207/287) 2024-11-01T16:45:09.7893695Z remote: Compressing objects: 73% (210/287) 2024-11-01T16:45:09.7894588Z remote: Compressing objects: 74% (213/287) 2024-11-01T16:45:09.7895443Z remote: Compressing objects: 75% (216/287) 2024-11-01T16:45:09.7896443Z remote: Compressing objects: 76% (219/287) 2024-11-01T16:45:09.7897320Z remote: Compressing objects: 77% (221/287) 2024-11-01T16:45:09.7898156Z remote: Compressing objects: 78% (224/287) 2024-11-01T16:45:09.7899007Z remote: Compressing objects: 79% (227/287) 2024-11-01T16:45:09.7899863Z remote: Compressing objects: 80% (230/287) 2024-11-01T16:45:09.7900701Z remote: Compressing objects: 81% (233/287) 2024-11-01T16:45:09.7901553Z remote: Compressing objects: 82% (236/287) 2024-11-01T16:45:09.7902405Z remote: Compressing objects: 83% (239/287) 2024-11-01T16:45:09.7903246Z remote: Compressing objects: 84% (242/287) 2024-11-01T16:45:09.7904092Z remote: Compressing objects: 85% (244/287) 2024-11-01T16:45:09.7904956Z remote: Compressing objects: 86% (247/287) 2024-11-01T16:45:09.7906221Z remote: Compressing objects: 87% (250/287) 2024-11-01T16:45:09.7907105Z remote: Compressing objects: 88% (253/287) 2024-11-01T16:45:09.7908005Z remote: Compressing objects: 89% (256/287) 2024-11-01T16:45:09.7908883Z remote: Compressing objects: 90% (259/287) 2024-11-01T16:45:09.7909777Z remote: Compressing objects: 91% (262/287) 2024-11-01T16:45:09.7910631Z remote: Compressing objects: 92% (265/287) 2024-11-01T16:45:09.7911488Z remote: Compressing objects: 93% (267/287) 2024-11-01T16:45:09.7912357Z remote: Compressing objects: 94% (270/287) 2024-11-01T16:45:09.7913192Z remote: Compressing objects: 95% (273/287) 2024-11-01T16:45:09.7914037Z remote: Compressing objects: 96% (276/287) 2024-11-01T16:45:09.7914889Z remote: Compressing objects: 97% (279/287) 2024-11-01T16:45:09.7915737Z remote: Compressing objects: 98% (282/287) 2024-11-01T16:45:09.7916607Z remote: Compressing objects: 99% (285/287) 2024-11-01T16:45:09.7917492Z remote: Compressing objects: 100% (287/287) 2024-11-01T16:45:09.7918434Z remote: Compressing objects: 100% (287/287), done. 2024-11-01T16:45:28.7806862Z remote: Total 1053739 (delta 369), reused 520 (delta 313), pack-reused 1053138 (from 1) 2024-11-01T16:45:41.2622666Z [command]/usr/bin/git rev-parse --verify --quiet d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea^{object} 2024-11-01T16:45:41.2679100Z d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:45:41.2690773Z ##[endgroup] 2024-11-01T16:45:41.2691775Z ##[group]Determining the checkout info 2024-11-01T16:45:41.2692960Z ##[endgroup] 2024-11-01T16:45:41.2697165Z ##[group]Checking out the ref 2024-11-01T16:45:41.2700327Z [command]/usr/bin/git checkout --quiet --force d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:45:43.0186333Z ##[endgroup] 2024-11-01T16:45:43.0187469Z ##[group]Setting up auth for fetching submodules 2024-11-01T16:45:43.0191713Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-11-01T16:45:43.0276826Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-11-01T16:45:43.0341283Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-11-01T16:45:43.0410463Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-11-01T16:45:43.0479405Z ##[endgroup] 2024-11-01T16:45:43.0480336Z ##[group]Fetching submodules 2024-11-01T16:45:43.0484848Z [command]/usr/bin/git submodule sync --recursive 2024-11-01T16:45:43.1114925Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-11-01T16:45:43.1740443Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-11-01T16:45:43.1752243Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-11-01T16:45:43.1757100Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-11-01T16:45:43.1765330Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-11-01T16:45:43.1773887Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-11-01T16:45:43.1784219Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-11-01T16:45:43.1792232Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-11-01T16:45:43.1801443Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-11-01T16:45:43.1811099Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-11-01T16:45:43.1820624Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-11-01T16:45:43.1830281Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-11-01T16:45:43.1840057Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-11-01T16:45:43.1849744Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-11-01T16:45:43.1859849Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-11-01T16:45:43.1870125Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-11-01T16:45:43.1881650Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-11-01T16:45:43.1891939Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-11-01T16:45:43.1903681Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:45:43.1912668Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-11-01T16:45:43.1921945Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-11-01T16:45:43.1930970Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-11-01T16:45:43.1941510Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-11-01T16:45:43.1951958Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-11-01T16:45:43.1962554Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-11-01T16:45:43.1972330Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-11-01T16:45:43.1982754Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-11-01T16:45:43.1992818Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-11-01T16:45:43.2003514Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-11-01T16:45:43.2013816Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-11-01T16:45:43.2024496Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-11-01T16:45:43.2035128Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-11-01T16:45:43.2046034Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-11-01T16:45:43.2056701Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-11-01T16:45:43.2069051Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-11-01T16:45:43.2079898Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-11-01T16:45:43.2091154Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-11-01T16:45:43.2260099Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-11-01T16:45:43.8312594Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-11-01T16:45:44.2663840Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-11-01T16:45:44.6669111Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-11-01T16:45:45.2156102Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-11-01T16:45:45.8795252Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-11-01T16:45:47.6820698Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-11-01T16:45:54.9168684Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-11-01T16:45:55.6743919Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-11-01T16:45:57.3352224Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-11-01T16:45:58.0667858Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-11-01T16:45:58.9426980Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-11-01T16:46:00.2169886Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-11-01T16:46:02.1900688Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-11-01T16:46:06.3461841Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-11-01T16:46:07.9346706Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-11-01T16:46:09.6973843Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-11-01T16:46:10.8272440Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-11-01T16:46:11.5536703Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-11-01T16:46:12.5569108Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-11-01T16:46:13.6562500Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-11-01T16:46:14.2772549Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-11-01T16:46:14.8402215Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-11-01T16:46:16.4882429Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-11-01T16:46:17.4813220Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-11-01T16:46:18.1431257Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-11-01T16:46:22.6280502Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-11-01T16:46:24.6769909Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-11-01T16:46:30.6811771Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-11-01T16:46:31.2744811Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-11-01T16:46:38.7310435Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-11-01T16:46:39.1041043Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-11-01T16:46:39.6136630Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-11-01T16:46:40.8845946Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-11-01T16:46:42.0772031Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-11-01T16:46:43.0859583Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-11-01T16:46:43.8362543Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-11-01T16:46:43.8996879Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-11-01T16:46:43.9548144Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-11-01T16:46:44.0492008Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-11-01T16:46:44.1512983Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-11-01T16:46:44.2711411Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-11-01T16:46:45.3920145Z Submodule path 'third_party/XNNPACK': checked out '87ee0b46b834f67bad9025d4a82ed5654f3403d3' 2024-11-01T16:46:45.4881068Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-11-01T16:46:45.8134289Z Submodule path 'third_party/composable_kernel': checked out 'cedccd59c94cb0c74e7ec0d0f6c791aed081febc' 2024-11-01T16:46:45.9533845Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-11-01T16:46:46.1309211Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-11-01T16:46:46.2431129Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-11-01T16:46:46.9484026Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-11-01T16:46:47.3000956Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-11-01T16:46:47.4829543Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-11-01T16:46:47.4944796Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:46:47.4953328Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:46:47.4961746Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:46:47.4970143Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:46:47.4979746Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:46:47.5092136Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-11-01T16:46:48.7191461Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-11-01T16:46:49.5964891Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-11-01T16:46:51.5671279Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-11-01T16:46:52.6951312Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-11-01T16:46:53.4654041Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-11-01T16:46:53.6476808Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-11-01T16:46:54.2201796Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-11-01T16:46:54.3770281Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-11-01T16:46:54.4347608Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-11-01T16:46:54.6337321Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-11-01T16:46:54.7475984Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-11-01T16:46:54.8653656Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-11-01T16:46:54.9587861Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-11-01T16:46:55.0785735Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:46:55.1434674Z Submodule path 'third_party/ideep': checked out '41d636c2bbcea6bff0faf97cdb65a48cdde987af' 2024-11-01T16:46:55.1551041Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-11-01T16:46:55.1661184Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-11-01T16:47:05.0811169Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-11-01T16:47:05.1565528Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-11-01T16:47:05.3314648Z Submodule path 'third_party/kineto': checked out 'ed052ea024b9468908d558b15cd3f7584fb0f492' 2024-11-01T16:47:05.3427016Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:47:05.3435207Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:47:05.3443956Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:47:05.3552140Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-11-01T16:47:06.1812407Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-11-01T16:47:07.3629783Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-11-01T16:47:08.9021111Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-11-01T16:47:08.9118300Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:47:08.9125300Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:47:08.9128459Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:47:08.9131558Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:47:08.9134648Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:47:08.9138971Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:47:08.9143303Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:47:08.9148110Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:47:08.9225521Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-11-01T16:47:09.9443295Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-11-01T16:47:10.7282445Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-11-01T16:47:11.8997324Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-11-01T16:47:12.6079952Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-11-01T16:47:13.6631497Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-11-01T16:47:15.0477061Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-11-01T16:47:19.5234670Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-11-01T16:47:20.3017496Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-11-01T16:47:20.3775011Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-11-01T16:47:20.4877492Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-11-01T16:47:20.5495186Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-11-01T16:47:20.5616863Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:47:20.5728669Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-11-01T16:47:22.2203366Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-11-01T16:47:22.2975798Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-11-01T16:47:22.4074894Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-11-01T16:47:22.6071218Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-11-01T16:47:22.6766123Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-11-01T16:47:22.7910913Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-11-01T16:47:22.9293323Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-11-01T16:47:23.0493414Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-11-01T16:47:23.1543929Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-11-01T16:47:23.3486950Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-11-01T16:47:23.9333780Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-11-01T16:47:23.9488458Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-11-01T16:47:23.9644349Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-11-01T16:47:25.2518026Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-11-01T16:47:25.4112045Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-11-01T16:47:25.4233264Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:47:25.4241515Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:47:25.4249885Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:47:25.4258431Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:47:25.4267418Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:47:25.4276105Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:47:25.4284924Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:47:25.4293598Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:47:25.4410930Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-11-01T16:47:26.7147349Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-11-01T16:47:28.1005482Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-11-01T16:47:28.7458961Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-11-01T16:47:33.2478985Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-11-01T16:47:33.9050702Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-11-01T16:47:34.4108923Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-11-01T16:47:35.0208289Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-11-01T16:47:42.9590269Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-11-01T16:47:43.0639492Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-11-01T16:47:43.1324422Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-11-01T16:47:43.3240764Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-11-01T16:47:43.3865044Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-11-01T16:47:43.4518959Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-11-01T16:47:43.5199775Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-11-01T16:47:43.5316271Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:47:43.5324251Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:47:43.5434004Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-11-01T16:47:45.2384246Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-11-01T16:47:46.9849165Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-11-01T16:47:47.0806867Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-11-01T16:47:47.7755072Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-11-01T16:47:47.8444284Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-11-01T16:47:48.2720638Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-11-01T16:47:48.2845870Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:47:48.2854262Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-11-01T16:47:48.2978242Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-11-01T16:47:49.0173981Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-11-01T16:47:50.4153306Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-11-01T16:47:50.5786040Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-11-01T16:47:50.6336830Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-11-01T16:47:50.6975192Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-11-01T16:47:50.8137765Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-11-01T16:47:50.9199481Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-11-01T16:47:51.0400158Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-11-01T16:47:51.1371111Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-11-01T16:47:51.1478649Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:47:51.1487224Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:47:51.1495249Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:47:51.1504045Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:47:51.1620598Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-11-01T16:47:52.3301295Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-11-01T16:47:52.8832786Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-11-01T16:47:54.3922080Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-11-01T16:47:55.7540053Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-11-01T16:47:55.8206192Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-11-01T16:47:55.9699832Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-11-01T16:47:56.0680726Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-11-01T16:47:56.0803638Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:47:56.0918102Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-11-01T16:47:56.5939117Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-11-01T16:47:56.6159647Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-11-01T16:47:56.6798347Z Entering 'android/libs/fbjni' 2024-11-01T16:47:56.6904979Z Entering 'third_party/FP16' 2024-11-01T16:47:56.7011855Z Entering 'third_party/FXdiv' 2024-11-01T16:47:56.7116549Z Entering 'third_party/NNPACK' 2024-11-01T16:47:56.7219699Z Entering 'third_party/NVTX' 2024-11-01T16:47:56.7326284Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:47:56.7430816Z Entering 'third_party/XNNPACK' 2024-11-01T16:47:56.7572413Z Entering 'third_party/benchmark' 2024-11-01T16:47:56.7674542Z Entering 'third_party/composable_kernel' 2024-11-01T16:47:56.7790768Z Entering 'third_party/cpp-httplib' 2024-11-01T16:47:56.7892615Z Entering 'third_party/cpuinfo' 2024-11-01T16:47:56.7998488Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:47:56.8098879Z Entering 'third_party/cutlass' 2024-11-01T16:47:56.8222015Z Entering 'third_party/eigen' 2024-11-01T16:47:56.8331805Z Entering 'third_party/fbgemm' 2024-11-01T16:47:56.8432982Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:47:56.8529135Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:47:56.8628946Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:47:56.8742683Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:47:56.8839006Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:47:56.8941561Z Entering 'third_party/flatbuffers' 2024-11-01T16:47:56.9051427Z Entering 'third_party/fmt' 2024-11-01T16:47:56.9156022Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:47:56.9265632Z Entering 'third_party/gloo' 2024-11-01T16:47:56.9369589Z Entering 'third_party/googletest' 2024-11-01T16:47:56.9473536Z Entering 'third_party/ideep' 2024-11-01T16:47:56.9574232Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:47:56.9693816Z Entering 'third_party/ittapi' 2024-11-01T16:47:56.9797711Z Entering 'third_party/kineto' 2024-11-01T16:47:56.9897044Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:47:56.9993544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:47:57.0096156Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:47:57.0195972Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:47:57.0295613Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:47:57.0390730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:47:57.0497406Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:47:57.0598664Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:47:57.0696130Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:47:57.0799658Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:47:57.0909437Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:47:57.1005791Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:47:57.1107859Z Entering 'third_party/mimalloc' 2024-11-01T16:47:57.1213098Z Entering 'third_party/nccl/nccl' 2024-11-01T16:47:57.1312362Z Entering 'third_party/nlohmann' 2024-11-01T16:47:57.1417085Z Entering 'third_party/onnx' 2024-11-01T16:47:57.1552225Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:47:57.1662632Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:47:57.1766138Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:47:57.1868315Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:47:57.1963964Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:47:57.2064244Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:47:57.2166601Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:47:57.2264464Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:47:57.2362589Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:47:57.2453806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:47:57.2559522Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:47:57.2664089Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:47:57.2810188Z Entering 'third_party/pocketfft' 2024-11-01T16:47:57.2913853Z Entering 'third_party/protobuf' 2024-11-01T16:47:57.3019090Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:47:57.3118265Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:47:57.3227072Z Entering 'third_party/psimd' 2024-11-01T16:47:57.3331948Z Entering 'third_party/pthreadpool' 2024-11-01T16:47:57.3436460Z Entering 'third_party/pybind11' 2024-11-01T16:47:57.3542478Z Entering 'third_party/python-peachpy' 2024-11-01T16:47:57.3645153Z Entering 'third_party/sleef' 2024-11-01T16:47:57.3749294Z Entering 'third_party/tensorpipe' 2024-11-01T16:47:57.3849657Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:47:57.3948543Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:47:57.4047114Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:47:57.4141807Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:47:57.4237533Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:47:57.4378127Z ##[endgroup] 2024-11-01T16:47:57.4382807Z ##[group]Persisting credentials for submodules 2024-11-01T16:47:57.4386007Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-11-01T16:47:57.4999841Z Entering 'android/libs/fbjni' 2024-11-01T16:47:57.5117462Z Entering 'third_party/FP16' 2024-11-01T16:47:57.5234056Z Entering 'third_party/FXdiv' 2024-11-01T16:47:57.5349584Z Entering 'third_party/NNPACK' 2024-11-01T16:47:57.5466406Z Entering 'third_party/NVTX' 2024-11-01T16:47:57.5580986Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:47:57.5701719Z Entering 'third_party/XNNPACK' 2024-11-01T16:47:57.5856231Z Entering 'third_party/benchmark' 2024-11-01T16:47:57.5970290Z Entering 'third_party/composable_kernel' 2024-11-01T16:47:57.6101056Z Entering 'third_party/cpp-httplib' 2024-11-01T16:47:57.6217166Z Entering 'third_party/cpuinfo' 2024-11-01T16:47:57.6333313Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:47:57.6447574Z Entering 'third_party/cutlass' 2024-11-01T16:47:57.6581804Z Entering 'third_party/eigen' 2024-11-01T16:47:57.6700476Z Entering 'third_party/fbgemm' 2024-11-01T16:47:57.6809223Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:47:57.6921670Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:47:57.7034860Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:47:57.7158519Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:47:57.7267371Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:47:57.7384426Z Entering 'third_party/flatbuffers' 2024-11-01T16:47:57.7504944Z Entering 'third_party/fmt' 2024-11-01T16:47:57.7620282Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:47:57.7739356Z Entering 'third_party/gloo' 2024-11-01T16:47:57.7855827Z Entering 'third_party/googletest' 2024-11-01T16:47:57.7969957Z Entering 'third_party/ideep' 2024-11-01T16:47:57.8081982Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:47:57.8213468Z Entering 'third_party/ittapi' 2024-11-01T16:47:57.8329044Z Entering 'third_party/kineto' 2024-11-01T16:47:57.8442768Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:47:57.8552315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:47:57.8665901Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:47:57.8775328Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:47:57.8887762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:47:57.8992940Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:47:57.9107505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:47:57.9219491Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:47:57.9329312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:47:57.9441350Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:47:57.9556172Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:47:57.9667558Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:47:57.9787888Z Entering 'third_party/mimalloc' 2024-11-01T16:47:57.9903135Z Entering 'third_party/nccl/nccl' 2024-11-01T16:47:58.0017920Z Entering 'third_party/nlohmann' 2024-11-01T16:47:58.0138301Z Entering 'third_party/onnx' 2024-11-01T16:47:58.0280499Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:47:58.0406253Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:47:58.0521248Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:47:58.0631820Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:47:58.0742359Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:47:58.0852246Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:47:58.0964738Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:47:58.1075488Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:47:58.1186508Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:47:58.1291152Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:47:58.1406992Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:47:58.1524195Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:47:58.1681424Z Entering 'third_party/pocketfft' 2024-11-01T16:47:58.1796650Z Entering 'third_party/protobuf' 2024-11-01T16:47:58.1915884Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:47:58.2029033Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:47:58.2152440Z Entering 'third_party/psimd' 2024-11-01T16:47:58.2268995Z Entering 'third_party/pthreadpool' 2024-11-01T16:47:58.2387251Z Entering 'third_party/pybind11' 2024-11-01T16:47:58.2504906Z Entering 'third_party/python-peachpy' 2024-11-01T16:47:58.2619518Z Entering 'third_party/sleef' 2024-11-01T16:47:58.2733474Z Entering 'third_party/tensorpipe' 2024-11-01T16:47:58.2844342Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:47:58.2956408Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:47:58.3068053Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:47:58.3179225Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:47:58.3283643Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:47:58.3441187Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-11-01T16:47:58.4060194Z Entering 'android/libs/fbjni' 2024-11-01T16:47:58.4157361Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-11-01T16:47:58.4211206Z Entering 'third_party/FP16' 2024-11-01T16:47:58.4306782Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-11-01T16:47:58.4359721Z Entering 'third_party/FXdiv' 2024-11-01T16:47:58.4461489Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-11-01T16:47:58.4514094Z Entering 'third_party/NNPACK' 2024-11-01T16:47:58.4611001Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-11-01T16:47:58.4664526Z Entering 'third_party/NVTX' 2024-11-01T16:47:58.4761321Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-11-01T16:47:58.4815988Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:47:58.4914351Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-11-01T16:47:58.4967617Z Entering 'third_party/XNNPACK' 2024-11-01T16:47:58.5067874Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-11-01T16:47:58.5155766Z Entering 'third_party/benchmark' 2024-11-01T16:47:58.5251350Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:47:58.5304696Z Entering 'third_party/composable_kernel' 2024-11-01T16:47:58.5400486Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-11-01T16:47:58.5466516Z Entering 'third_party/cpp-httplib' 2024-11-01T16:47:58.5561892Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-11-01T16:47:58.5616525Z Entering 'third_party/cpuinfo' 2024-11-01T16:47:58.5712711Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:47:58.5766470Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:47:58.5862537Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-11-01T16:47:58.5916365Z Entering 'third_party/cutlass' 2024-11-01T16:47:58.6013331Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:47:58.6086829Z Entering 'third_party/eigen' 2024-11-01T16:47:58.6182883Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-11-01T16:47:58.6241108Z Entering 'third_party/fbgemm' 2024-11-01T16:47:58.6337748Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-11-01T16:47:58.6388986Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:47:58.6485065Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-11-01T16:47:58.6537069Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:47:58.6631083Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-11-01T16:47:58.6682192Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:47:58.6774050Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-11-01T16:47:58.6841427Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:47:58.6934194Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:58.6984632Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:47:58.7080567Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-11-01T16:47:58.7138559Z Entering 'third_party/flatbuffers' 2024-11-01T16:47:58.7234795Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-11-01T16:47:58.7293631Z Entering 'third_party/fmt' 2024-11-01T16:47:58.7389336Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:47:58.7442314Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:47:58.7536784Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-11-01T16:47:58.7589851Z Entering 'third_party/gloo' 2024-11-01T16:47:58.7684525Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-11-01T16:47:58.7738229Z Entering 'third_party/googletest' 2024-11-01T16:47:58.7837382Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:58.7889554Z Entering 'third_party/ideep' 2024-11-01T16:47:58.7986011Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-11-01T16:47:58.8034082Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:47:58.8128615Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-11-01T16:47:58.8197970Z Entering 'third_party/ittapi' 2024-11-01T16:47:58.8293563Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-11-01T16:47:58.8346739Z Entering 'third_party/kineto' 2024-11-01T16:47:58.8440750Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-11-01T16:47:58.8490904Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:47:58.8587140Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-11-01T16:47:58.8635389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:47:58.8730862Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-11-01T16:47:58.8785324Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:47:58.8880391Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-11-01T16:47:58.8933188Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:47:58.9029586Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-11-01T16:47:58.9082133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:47:58.9177972Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-11-01T16:47:58.9226450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:47:58.9324817Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-11-01T16:47:58.9382140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:47:58.9476676Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-11-01T16:47:58.9527277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:47:58.9623638Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:58.9674307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:47:58.9770642Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-11-01T16:47:58.9824128Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:47:58.9917136Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-11-01T16:47:58.9976255Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:47:59.0069410Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-11-01T16:47:59.0119513Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:47:59.0218209Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-11-01T16:47:59.0277561Z Entering 'third_party/mimalloc' 2024-11-01T16:47:59.0373482Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-11-01T16:47:59.0429391Z Entering 'third_party/nccl/nccl' 2024-11-01T16:47:59.0524240Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-11-01T16:47:59.0579274Z Entering 'third_party/nlohmann' 2024-11-01T16:47:59.0673520Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-11-01T16:47:59.0728459Z Entering 'third_party/onnx' 2024-11-01T16:47:59.0824715Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-11-01T16:47:59.0907172Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:47:59.1001307Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:47:59.1063754Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:47:59.1158551Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-11-01T16:47:59.1211504Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:47:59.1305926Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:47:59.1356999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:47:59.1451112Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:59.1501358Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:47:59.1594241Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-11-01T16:47:59.1646027Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:47:59.1740765Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-11-01T16:47:59.1794609Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:47:59.1891298Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-11-01T16:47:59.1940792Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:47:59.2038360Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-11-01T16:47:59.2087883Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:47:59.2181957Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-11-01T16:47:59.2228455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:47:59.2325972Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-11-01T16:47:59.2380597Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:47:59.2475777Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-11-01T16:47:59.2532901Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:47:59.2628300Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-11-01T16:47:59.2729225Z Entering 'third_party/pocketfft' 2024-11-01T16:47:59.2825402Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-11-01T16:47:59.2879918Z Entering 'third_party/protobuf' 2024-11-01T16:47:59.2974603Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-11-01T16:47:59.3032635Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:47:59.3126026Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-11-01T16:47:59.3175565Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:47:59.3270926Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:59.3330053Z Entering 'third_party/psimd' 2024-11-01T16:47:59.3426835Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-11-01T16:47:59.3480740Z Entering 'third_party/pthreadpool' 2024-11-01T16:47:59.3576091Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-11-01T16:47:59.3628616Z Entering 'third_party/pybind11' 2024-11-01T16:47:59.3723415Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:47:59.3777214Z Entering 'third_party/python-peachpy' 2024-11-01T16:47:59.3872279Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-11-01T16:47:59.3925146Z Entering 'third_party/sleef' 2024-11-01T16:47:59.4019709Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-11-01T16:47:59.4077249Z Entering 'third_party/tensorpipe' 2024-11-01T16:47:59.4174581Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-11-01T16:47:59.4224635Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:47:59.4317804Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-11-01T16:47:59.4369434Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:47:59.4462311Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-11-01T16:47:59.4512290Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:47:59.4605605Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-11-01T16:47:59.4656437Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:47:59.4748349Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-11-01T16:47:59.4794488Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:47:59.4890840Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-11-01T16:47:59.5416063Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-11-01T16:47:59.6035398Z Entering 'android/libs/fbjni' 2024-11-01T16:47:59.6140385Z Entering 'third_party/FP16' 2024-11-01T16:47:59.6242734Z Entering 'third_party/FXdiv' 2024-11-01T16:47:59.6345676Z Entering 'third_party/NNPACK' 2024-11-01T16:47:59.6447340Z Entering 'third_party/NVTX' 2024-11-01T16:47:59.6554459Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:47:59.6656611Z Entering 'third_party/XNNPACK' 2024-11-01T16:47:59.6796566Z Entering 'third_party/benchmark' 2024-11-01T16:47:59.6903981Z Entering 'third_party/composable_kernel' 2024-11-01T16:47:59.7018339Z Entering 'third_party/cpp-httplib' 2024-11-01T16:47:59.7119488Z Entering 'third_party/cpuinfo' 2024-11-01T16:47:59.7224422Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:47:59.7327213Z Entering 'third_party/cutlass' 2024-11-01T16:47:59.7447177Z Entering 'third_party/eigen' 2024-11-01T16:47:59.7557990Z Entering 'third_party/fbgemm' 2024-11-01T16:47:59.7661434Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:47:59.7757133Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:47:59.7856203Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:47:59.7966938Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:47:59.8062485Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:47:59.8165517Z Entering 'third_party/flatbuffers' 2024-11-01T16:47:59.8275446Z Entering 'third_party/fmt' 2024-11-01T16:47:59.8376722Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:47:59.8479346Z Entering 'third_party/gloo' 2024-11-01T16:47:59.8582536Z Entering 'third_party/googletest' 2024-11-01T16:47:59.8684063Z Entering 'third_party/ideep' 2024-11-01T16:47:59.8780413Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:47:59.8897366Z Entering 'third_party/ittapi' 2024-11-01T16:47:59.8999744Z Entering 'third_party/kineto' 2024-11-01T16:47:59.9099651Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:47:59.9195045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:47:59.9298896Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:47:59.9398434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:47:59.9497374Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:47:59.9590766Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:47:59.9698270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:47:59.9795333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:47:59.9895281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:47:59.9996517Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:48:00.0103197Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:48:00.0200526Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:48:00.0304612Z Entering 'third_party/mimalloc' 2024-11-01T16:48:00.0409175Z Entering 'third_party/nccl/nccl' 2024-11-01T16:48:00.0512479Z Entering 'third_party/nlohmann' 2024-11-01T16:48:00.0616880Z Entering 'third_party/onnx' 2024-11-01T16:48:00.0752383Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:48:00.0864720Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:48:00.0968326Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:48:00.1064747Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:48:00.1167887Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:48:00.1264511Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:48:00.1363684Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:48:00.1463264Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:48:00.1562559Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:48:00.1655544Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:48:00.1761082Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:48:00.1867913Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:48:00.2012062Z Entering 'third_party/pocketfft' 2024-11-01T16:48:00.2115475Z Entering 'third_party/protobuf' 2024-11-01T16:48:00.2226151Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:48:00.2327077Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:48:00.2434374Z Entering 'third_party/psimd' 2024-11-01T16:48:00.2539258Z Entering 'third_party/pthreadpool' 2024-11-01T16:48:00.2642706Z Entering 'third_party/pybind11' 2024-11-01T16:48:00.2749064Z Entering 'third_party/python-peachpy' 2024-11-01T16:48:00.2852306Z Entering 'third_party/sleef' 2024-11-01T16:48:00.2955410Z Entering 'third_party/tensorpipe' 2024-11-01T16:48:00.3056235Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:48:00.3152970Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:48:00.3251431Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:48:00.3350375Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:48:00.3443666Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:48:00.3588809Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-11-01T16:48:00.4205298Z Entering 'android/libs/fbjni' 2024-11-01T16:48:00.4309778Z Entering 'third_party/FP16' 2024-11-01T16:48:00.4414835Z Entering 'third_party/FXdiv' 2024-11-01T16:48:00.4515936Z Entering 'third_party/NNPACK' 2024-11-01T16:48:00.4619381Z Entering 'third_party/NVTX' 2024-11-01T16:48:00.4725846Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T16:48:00.4828368Z Entering 'third_party/XNNPACK' 2024-11-01T16:48:00.4967522Z Entering 'third_party/benchmark' 2024-11-01T16:48:00.5073096Z Entering 'third_party/composable_kernel' 2024-11-01T16:48:00.5191165Z Entering 'third_party/cpp-httplib' 2024-11-01T16:48:00.5294225Z Entering 'third_party/cpuinfo' 2024-11-01T16:48:00.5399787Z Entering 'third_party/cudnn_frontend' 2024-11-01T16:48:00.5501652Z Entering 'third_party/cutlass' 2024-11-01T16:48:00.5624191Z Entering 'third_party/eigen' 2024-11-01T16:48:00.5732050Z Entering 'third_party/fbgemm' 2024-11-01T16:48:00.5835257Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T16:48:00.5934551Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T16:48:00.6031783Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T16:48:00.6143192Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T16:48:00.6242179Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T16:48:00.6343165Z Entering 'third_party/flatbuffers' 2024-11-01T16:48:00.6457392Z Entering 'third_party/fmt' 2024-11-01T16:48:00.6561831Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T16:48:00.6665650Z Entering 'third_party/gloo' 2024-11-01T16:48:00.6766786Z Entering 'third_party/googletest' 2024-11-01T16:48:00.6870368Z Entering 'third_party/ideep' 2024-11-01T16:48:00.6970394Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T16:48:00.7087407Z Entering 'third_party/ittapi' 2024-11-01T16:48:00.7190490Z Entering 'third_party/kineto' 2024-11-01T16:48:00.7289488Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T16:48:00.7387095Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T16:48:00.7487149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T16:48:00.7586298Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T16:48:00.7685597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T16:48:00.7781058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T16:48:00.7888462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T16:48:00.7989122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T16:48:00.8088637Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T16:48:00.8191503Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T16:48:00.8295138Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T16:48:00.8393953Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T16:48:00.8495667Z Entering 'third_party/mimalloc' 2024-11-01T16:48:00.8599905Z Entering 'third_party/nccl/nccl' 2024-11-01T16:48:00.8703120Z Entering 'third_party/nlohmann' 2024-11-01T16:48:00.8807343Z Entering 'third_party/onnx' 2024-11-01T16:48:00.8939269Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T16:48:00.9049473Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T16:48:00.9154572Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T16:48:00.9251840Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T16:48:00.9348837Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T16:48:00.9445822Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T16:48:00.9546404Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T16:48:00.9641976Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T16:48:00.9737603Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T16:48:00.9830186Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T16:48:00.9931517Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T16:48:01.0034946Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T16:48:01.0178901Z Entering 'third_party/pocketfft' 2024-11-01T16:48:01.0279796Z Entering 'third_party/protobuf' 2024-11-01T16:48:01.0388338Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T16:48:01.0486118Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T16:48:01.0592476Z Entering 'third_party/psimd' 2024-11-01T16:48:01.0693331Z Entering 'third_party/pthreadpool' 2024-11-01T16:48:01.0797378Z Entering 'third_party/pybind11' 2024-11-01T16:48:01.0901986Z Entering 'third_party/python-peachpy' 2024-11-01T16:48:01.1005672Z Entering 'third_party/sleef' 2024-11-01T16:48:01.1108409Z Entering 'third_party/tensorpipe' 2024-11-01T16:48:01.1209444Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T16:48:01.1306531Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T16:48:01.1402549Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T16:48:01.1499187Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T16:48:01.1591666Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T16:48:01.1729131Z ##[endgroup] 2024-11-01T16:48:01.1818214Z [command]/usr/bin/git log -1 --format='%H' 2024-11-01T16:48:01.1875254Z 'd1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea' 2024-11-01T16:48:01.2267101Z Prepare all required actions 2024-11-01T16:48:01.2267897Z Getting action download info 2024-11-01T16:48:01.2401601Z ##[group]Run ./.github/actions/setup-rocm 2024-11-01T16:48:01.2402224Z env: 2024-11-01T16:48:01.2402618Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.2403136Z ##[endgroup] 2024-11-01T16:48:01.2442544Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-11-01T16:48:01.2443922Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-11-01T16:48:01.2481732Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.2482493Z env: 2024-11-01T16:48:01.2482908Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.2483409Z ##[endgroup] 2024-11-01T16:48:01.2640218Z ##[group]Run set -ex 2024-11-01T16:48:01.2640813Z set -ex 2024-11-01T16:48:01.2641293Z  2024-11-01T16:48:01.2641832Z cat ~/.docker/config.json || true 2024-11-01T16:48:01.2643700Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-11-01T16:48:01.2645467Z rm -f ~/.docker/config.json 2024-11-01T16:48:01.2680991Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.2681856Z env: 2024-11-01T16:48:01.2682290Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.2683017Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.2683796Z ##[endgroup] 2024-11-01T16:48:01.2761684Z + cat /var/home/pytorchci/.docker/config.json 2024-11-01T16:48:01.2785035Z { 2024-11-01T16:48:01.2785491Z "auths": {} 2024-11-01T16:48:01.2788706Z + rm -f /var/home/pytorchci/.docker/config.json 2024-11-01T16:48:01.2823972Z } 2024-11-01T16:48:01.2871991Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:48:01.2873358Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T16:48:01.2874358Z # shellcheck disable=SC2046 2024-11-01T16:48:01.2875152Z docker stop $(docker ps -q) || true 2024-11-01T16:48:01.2875946Z # Prune all stopped containers. 2024-11-01T16:48:01.2876707Z docker container prune -f 2024-11-01T16:48:01.2920148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.2921019Z env: 2024-11-01T16:48:01.2921463Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.2922196Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.2922972Z ##[endgroup] 2024-11-01T16:48:01.3642935Z "docker stop" requires at least 1 argument. 2024-11-01T16:48:01.3644095Z See 'docker stop --help'. 2024-11-01T16:48:01.3649487Z 2024-11-01T16:48:01.3650223Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-11-01T16:48:01.3650943Z 2024-11-01T16:48:01.3651219Z Stop one or more running containers 2024-11-01T16:48:01.3996140Z Total reclaimed space: 0B 2024-11-01T16:48:01.4096153Z ##[group]Run cat /etc/os-release || true 2024-11-01T16:48:01.4097537Z cat /etc/os-release || true 2024-11-01T16:48:01.4098388Z cat /etc/apt/sources.list.d/rocm.list || true 2024-11-01T16:48:01.4099289Z cat /opt/rocm/.info/version || true 2024-11-01T16:48:01.4100010Z whoami 2024-11-01T16:48:01.4141853Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.4142700Z env: 2024-11-01T16:48:01.4143155Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.4143877Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.4144650Z ##[endgroup] 2024-11-01T16:48:01.4248210Z PRETTY_NAME="Ubuntu 22.04.4 LTS" 2024-11-01T16:48:01.4248877Z NAME="Ubuntu" 2024-11-01T16:48:01.4249371Z VERSION_ID="22.04" 2024-11-01T16:48:01.4249919Z VERSION="22.04.4 LTS (Jammy Jellyfish)" 2024-11-01T16:48:01.4254889Z VERSION_CODENAME=jammy 2024-11-01T16:48:01.4255543Z ID=ubuntu 2024-11-01T16:48:01.4256007Z ID_LIKE=debian 2024-11-01T16:48:01.4256723Z HOME_URL="https://www.ubuntu.com/" 2024-11-01T16:48:01.4258243Z SUPPORT_URL="https://help.ubuntu.com/" 2024-11-01T16:48:01.4259197Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-11-01T16:48:01.4260776Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-11-01T16:48:01.4261956Z UBUNTU_CODENAME=jammy 2024-11-01T16:48:01.4275904Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.2.1 jammy main 2024-11-01T16:48:01.4301830Z 6.2.1-112 2024-11-01T16:48:01.4343687Z pytorchci 2024-11-01T16:48:01.4393301Z ##[group]Run rocm-smi 2024-11-01T16:48:01.4393866Z rocm-smi 2024-11-01T16:48:01.4431843Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.4432689Z env: 2024-11-01T16:48:01.4433145Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.4433871Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.4434674Z ##[endgroup] 2024-11-01T16:48:01.6513313Z 2024-11-01T16:48:01.6513336Z 2024-11-01T16:48:01.6514485Z ========================================= ROCm System Management Interface ========================================= 2024-11-01T16:48:01.6515922Z =================================================== Concise Info =================================================== 2024-11-01T16:48:01.6517452Z Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-11-01T16:48:01.6519458Z  (DID, GUID) (Edge) (Avg) (Mem, Compute, ID)  2024-11-01T16:48:01.6520747Z ==================================================================================================================== 2024-11-01T16:48:01.6524206Z 0 4 0x740c, 57586 43.0°C 95.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-11-01T16:48:01.6526050Z 1 5 0x740c, 45873 44.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-11-01T16:48:01.6527856Z 2 2 0x740c, 51627 41.0°C 90.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-11-01T16:48:01.6529589Z 3 3 0x740c, 64489 37.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-11-01T16:48:01.6531311Z 4 8 0x740c, 30939 43.0°C 99.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-11-01T16:48:01.6533014Z 5 9 0x740c, 8466 41.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-11-01T16:48:01.6534718Z 6 6 0x740c, 41154 40.0°C 96.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-11-01T16:48:01.6536623Z 7 7 0x740c, 63755 33.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-11-01T16:48:01.6537890Z ==================================================================================================================== 2024-11-01T16:48:01.6539040Z =============================================== End of ROCm SMI Log ================================================ 2024-11-01T16:48:01.6700558Z ##[group]Run rocminfo 2024-11-01T16:48:01.6701137Z rocminfo 2024-11-01T16:48:01.6739457Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.6740326Z env: 2024-11-01T16:48:01.6740774Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.6741481Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.6742248Z ##[endgroup] 2024-11-01T16:48:01.8404896Z ROCk module version 6.8.5 is loaded 2024-11-01T16:48:01.8405434Z ===================== 2024-11-01T16:48:01.8405905Z HSA System Attributes 2024-11-01T16:48:01.8406494Z ===================== 2024-11-01T16:48:01.8407053Z Runtime Version: 1.14 2024-11-01T16:48:01.8407657Z Runtime Ext Version: 1.6 2024-11-01T16:48:01.8408312Z System Timestamp Freq.: 1000.000000MHz 2024-11-01T16:48:01.8409454Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-11-01T16:48:01.8418869Z Machine Model: LARGE 2024-11-01T16:48:01.8419977Z System Endianness: LITTLE 2024-11-01T16:48:01.8420836Z Mwaitx: DISABLED 2024-11-01T16:48:01.8421549Z DMAbuf Support: YES 2024-11-01T16:48:01.8421959Z 2024-11-01T16:48:01.8422141Z ========== 2024-11-01T16:48:01.8422698Z HSA Agents 2024-11-01T16:48:01.8423218Z ========== 2024-11-01T16:48:01.8423715Z ******* 2024-11-01T16:48:01.8424237Z Agent 1 2024-11-01T16:48:01.8424776Z ******* 2024-11-01T16:48:01.8425611Z Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:48:01.8426656Z Uuid: CPU-XX 2024-11-01T16:48:01.8427732Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:48:01.8428717Z Vendor Name: CPU 2024-11-01T16:48:01.8429690Z Feature: None specified 2024-11-01T16:48:01.8430638Z Profile: FULL_PROFILE 2024-11-01T16:48:01.8431595Z Float Round Mode: NEAR 2024-11-01T16:48:01.8432572Z Max Queue Number: 0(0x0) 2024-11-01T16:48:01.8433543Z Queue Min Size: 0(0x0) 2024-11-01T16:48:01.8434463Z Queue Max Size: 0(0x0) 2024-11-01T16:48:01.8435396Z Queue Type: MULTI 2024-11-01T16:48:01.8436552Z Node: 0 2024-11-01T16:48:01.8437085Z Device Type: CPU 2024-11-01T16:48:01.8437558Z Cache Info: 2024-11-01T16:48:01.8438001Z L1: 32768(0x8000) KB 2024-11-01T16:48:01.8438563Z Chip ID: 0(0x0) 2024-11-01T16:48:01.8439117Z ASIC Revision: 0(0x0) 2024-11-01T16:48:01.8439694Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8440280Z Max Clock Freq. (MHz): 2000 2024-11-01T16:48:01.8440829Z BDFID: 0 2024-11-01T16:48:01.8441378Z Internal Node ID: 0 2024-11-01T16:48:01.8441948Z Compute Unit: 64 2024-11-01T16:48:01.8442493Z SIMDs per CU: 0 2024-11-01T16:48:01.8443093Z Shader Engines: 0 2024-11-01T16:48:01.8444008Z Shader Arrs. per Eng.: 0 2024-11-01T16:48:01.8444620Z WatchPts on Addr. Ranges:1 2024-11-01T16:48:01.8445120Z Memory Properties: 2024-11-01T16:48:01.8445495Z Features: None 2024-11-01T16:48:01.8445871Z Pool Info: 2024-11-01T16:48:01.8446212Z Pool 1 2024-11-01T16:48:01.8446687Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8447273Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:48:01.8447852Z Allocatable: TRUE 2024-11-01T16:48:01.8448437Z Alloc Granule: 4KB 2024-11-01T16:48:01.8449053Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8449678Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8450278Z Accessible by all: TRUE 2024-11-01T16:48:01.8450771Z Pool 2 2024-11-01T16:48:01.8451227Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-11-01T16:48:01.8452032Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:48:01.8452614Z Allocatable: TRUE 2024-11-01T16:48:01.8453194Z Alloc Granule: 4KB 2024-11-01T16:48:01.8453814Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8454424Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8455038Z Accessible by all: TRUE 2024-11-01T16:48:01.8455525Z Pool 3 2024-11-01T16:48:01.8455990Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8456663Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:48:01.8457224Z Allocatable: TRUE 2024-11-01T16:48:01.8457792Z Alloc Granule: 4KB 2024-11-01T16:48:01.8458407Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8459022Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8459628Z Accessible by all: TRUE 2024-11-01T16:48:01.8460119Z ISA Info: 2024-11-01T16:48:01.8460449Z ******* 2024-11-01T16:48:01.8460769Z Agent 2 2024-11-01T16:48:01.8461084Z ******* 2024-11-01T16:48:01.8461563Z Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:48:01.8462406Z Uuid: CPU-XX 2024-11-01T16:48:01.8463051Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:48:01.8463629Z Vendor Name: CPU 2024-11-01T16:48:01.8464179Z Feature: None specified 2024-11-01T16:48:01.8464741Z Profile: FULL_PROFILE 2024-11-01T16:48:01.8465323Z Float Round Mode: NEAR 2024-11-01T16:48:01.8465903Z Max Queue Number: 0(0x0) 2024-11-01T16:48:01.8466482Z Queue Min Size: 0(0x0) 2024-11-01T16:48:01.8467034Z Queue Max Size: 0(0x0) 2024-11-01T16:48:01.8467572Z Queue Type: MULTI 2024-11-01T16:48:01.8468101Z Node: 1 2024-11-01T16:48:01.8468623Z Device Type: CPU 2024-11-01T16:48:01.8469093Z Cache Info: 2024-11-01T16:48:01.8469510Z L1: 32768(0x8000) KB 2024-11-01T16:48:01.8470050Z Chip ID: 0(0x0) 2024-11-01T16:48:01.8470582Z ASIC Revision: 0(0x0) 2024-11-01T16:48:01.8471143Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8471717Z Max Clock Freq. (MHz): 2000 2024-11-01T16:48:01.8472247Z BDFID: 0 2024-11-01T16:48:01.8472785Z Internal Node ID: 1 2024-11-01T16:48:01.8473370Z Compute Unit: 64 2024-11-01T16:48:01.8473942Z SIMDs per CU: 0 2024-11-01T16:48:01.8474497Z Shader Engines: 0 2024-11-01T16:48:01.8475095Z Shader Arrs. per Eng.: 0 2024-11-01T16:48:01.8475746Z WatchPts on Addr. Ranges:1 2024-11-01T16:48:01.8476506Z Memory Properties: 2024-11-01T16:48:01.8477216Z Features: None 2024-11-01T16:48:01.8477922Z Pool Info: 2024-11-01T16:48:01.8478518Z Pool 1 2024-11-01T16:48:01.8479635Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8480616Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:48:01.8481546Z Allocatable: TRUE 2024-11-01T16:48:01.8482520Z Alloc Granule: 4KB 2024-11-01T16:48:01.8483534Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8484544Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8485533Z Accessible by all: TRUE 2024-11-01T16:48:01.8486351Z Pool 2 2024-11-01T16:48:01.8487104Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-11-01T16:48:01.8488065Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:48:01.8488984Z Allocatable: TRUE 2024-11-01T16:48:01.8489961Z Alloc Granule: 4KB 2024-11-01T16:48:01.8490957Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8491977Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8492933Z Accessible by all: TRUE 2024-11-01T16:48:01.8493732Z Pool 3 2024-11-01T16:48:01.8494468Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8495379Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:48:01.8496892Z Allocatable: TRUE 2024-11-01T16:48:01.8497843Z Alloc Granule: 4KB 2024-11-01T16:48:01.8498826Z Alloc Recommended Granule:4KB 2024-11-01T16:48:01.8499817Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8500800Z Accessible by all: TRUE 2024-11-01T16:48:01.8501597Z ISA Info: 2024-11-01T16:48:01.8502146Z ******* 2024-11-01T16:48:01.8502687Z Agent 3 2024-11-01T16:48:01.8503206Z ******* 2024-11-01T16:48:01.8503884Z Name: gfx90a 2024-11-01T16:48:01.8505050Z Uuid: GPU-8f375f9eb6d0d924 2024-11-01T16:48:01.8506173Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8507554Z Vendor Name: AMD 2024-11-01T16:48:01.8508658Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8509682Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8510315Z Float Round Mode: NEAR 2024-11-01T16:48:01.8510894Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8511459Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8512028Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8512631Z Queue Type: MULTI 2024-11-01T16:48:01.8513229Z Node: 2 2024-11-01T16:48:01.8513794Z Device Type: GPU 2024-11-01T16:48:01.8514256Z Cache Info: 2024-11-01T16:48:01.8514679Z L1: 16(0x10) KB 2024-11-01T16:48:01.8515218Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8515794Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8516634Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8517734Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8519122Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8520035Z BDFID: 12800 2024-11-01T16:48:01.8520937Z Internal Node ID: 2 2024-11-01T16:48:01.8521854Z Compute Unit: 104 2024-11-01T16:48:01.8522770Z SIMDs per CU: 4 2024-11-01T16:48:01.8523743Z Shader Engines: 8 2024-11-01T16:48:01.8524697Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8525676Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8526665Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8527496Z Memory Properties: 2024-11-01T16:48:01.8528142Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8529028Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8529994Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8530925Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8531763Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8532555Z x 1024(0x400) 2024-11-01T16:48:01.8533361Z y 1024(0x400) 2024-11-01T16:48:01.8534157Z z 1024(0x400) 2024-11-01T16:48:01.8535071Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8536699Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8537675Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8538506Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8539202Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8540016Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8540852Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8541772Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8562834Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8563980Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8564983Z IOMMU Support:: None 2024-11-01T16:48:01.8565799Z Pool Info: 2024-11-01T16:48:01.8566386Z Pool 1 2024-11-01T16:48:01.8567190Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8568192Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8569147Z Allocatable: TRUE 2024-11-01T16:48:01.8570132Z Alloc Granule: 4KB 2024-11-01T16:48:01.8571170Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8572196Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8573199Z Accessible by all: FALSE 2024-11-01T16:48:01.8574040Z Pool 2 2024-11-01T16:48:01.8574804Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8575774Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8577157Z Allocatable: TRUE 2024-11-01T16:48:01.8578140Z Alloc Granule: 4KB 2024-11-01T16:48:01.8579150Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8580175Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8581161Z Accessible by all: FALSE 2024-11-01T16:48:01.8581994Z Pool 3 2024-11-01T16:48:01.8583203Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8584160Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8585071Z Allocatable: TRUE 2024-11-01T16:48:01.8586093Z Alloc Granule: 4KB 2024-11-01T16:48:01.8587262Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8588447Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8589608Z Accessible by all: FALSE 2024-11-01T16:48:01.8590224Z Pool 4 2024-11-01T16:48:01.8590664Z Segment: GROUP 2024-11-01T16:48:01.8591223Z Size: 64(0x40) KB 2024-11-01T16:48:01.8591781Z Allocatable: FALSE 2024-11-01T16:48:01.8592371Z Alloc Granule: 0KB 2024-11-01T16:48:01.8592981Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8593592Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8594281Z Accessible by all: FALSE 2024-11-01T16:48:01.8594765Z ISA Info: 2024-11-01T16:48:01.8595115Z ISA 1 2024-11-01T16:48:01.8595717Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8596909Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8597937Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8598913Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8599918Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8600876Z Fast f16: TRUE 2024-11-01T16:48:01.8601818Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8602678Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8603470Z x 1024(0x400) 2024-11-01T16:48:01.8604317Z y 1024(0x400) 2024-11-01T16:48:01.8605171Z z 1024(0x400) 2024-11-01T16:48:01.8606180Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8607219Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8608155Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8609055Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8609919Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8610861Z FBarrier Max Size: 32 2024-11-01T16:48:01.8611660Z ******* 2024-11-01T16:48:01.8612191Z Agent 4 2024-11-01T16:48:01.8612712Z ******* 2024-11-01T16:48:01.8613341Z Name: gfx90a 2024-11-01T16:48:01.8614375Z Uuid: GPU-47e65c59488469bd 2024-11-01T16:48:01.8615351Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8616421Z Vendor Name: AMD 2024-11-01T16:48:01.8617369Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8618286Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8619199Z Float Round Mode: NEAR 2024-11-01T16:48:01.8620189Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8621147Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8622393Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8623330Z Queue Type: MULTI 2024-11-01T16:48:01.8624211Z Node: 3 2024-11-01T16:48:01.8625088Z Device Type: GPU 2024-11-01T16:48:01.8625891Z Cache Info: 2024-11-01T16:48:01.8626705Z L1: 16(0x10) KB 2024-11-01T16:48:01.8627703Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8628750Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8629823Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8630815Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8631741Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8632660Z BDFID: 13568 2024-11-01T16:48:01.8633555Z Internal Node ID: 3 2024-11-01T16:48:01.8634473Z Compute Unit: 104 2024-11-01T16:48:01.8635361Z SIMDs per CU: 4 2024-11-01T16:48:01.8636360Z Shader Engines: 8 2024-11-01T16:48:01.8637460Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8638611Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8639961Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8640792Z Memory Properties: 2024-11-01T16:48:01.8641434Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8642312Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8643256Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8644193Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8645037Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8645800Z x 1024(0x400) 2024-11-01T16:48:01.8646623Z y 1024(0x400) 2024-11-01T16:48:01.8647423Z z 1024(0x400) 2024-11-01T16:48:01.8648325Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8649424Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8650396Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8651223Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8651892Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8652708Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8653518Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8654426Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8655435Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8656501Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8657427Z IOMMU Support:: None 2024-11-01T16:48:01.8658193Z Pool Info: 2024-11-01T16:48:01.8658764Z Pool 1 2024-11-01T16:48:01.8659529Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8660498Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8661428Z Allocatable: TRUE 2024-11-01T16:48:01.8662398Z Alloc Granule: 4KB 2024-11-01T16:48:01.8663400Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8664757Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8665754Z Accessible by all: FALSE 2024-11-01T16:48:01.8666700Z Pool 2 2024-11-01T16:48:01.8667598Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8668719Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8669809Z Allocatable: TRUE 2024-11-01T16:48:01.8670746Z Alloc Granule: 4KB 2024-11-01T16:48:01.8671631Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8672526Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8673374Z Accessible by all: FALSE 2024-11-01T16:48:01.8674082Z Pool 3 2024-11-01T16:48:01.8674727Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8675535Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8676331Z Allocatable: TRUE 2024-11-01T16:48:01.8677162Z Alloc Granule: 4KB 2024-11-01T16:48:01.8678013Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8678894Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8679741Z Accessible by all: FALSE 2024-11-01T16:48:01.8680749Z Pool 4 2024-11-01T16:48:01.8681364Z Segment: GROUP 2024-11-01T16:48:01.8682134Z Size: 64(0x40) KB 2024-11-01T16:48:01.8682914Z Allocatable: FALSE 2024-11-01T16:48:01.8683756Z Alloc Granule: 0KB 2024-11-01T16:48:01.8684630Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8685518Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8686361Z Accessible by all: FALSE 2024-11-01T16:48:01.8687069Z ISA Info: 2024-11-01T16:48:01.8687565Z ISA 1 2024-11-01T16:48:01.8688358Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8689271Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8690149Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8691003Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8691889Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8692725Z Fast f16: TRUE 2024-11-01T16:48:01.8693557Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8694317Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8695010Z x 1024(0x400) 2024-11-01T16:48:01.8695752Z y 1024(0x400) 2024-11-01T16:48:01.8696997Z z 1024(0x400) 2024-11-01T16:48:01.8697960Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8698824Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8699455Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8700191Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8700936Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8701741Z FBarrier Max Size: 32 2024-11-01T16:48:01.8702440Z ******* 2024-11-01T16:48:01.8703274Z Agent 5 2024-11-01T16:48:01.8703747Z ******* 2024-11-01T16:48:01.8704324Z Name: gfx90a 2024-11-01T16:48:01.8705235Z Uuid: GPU-7a90e0d9d56cf49c 2024-11-01T16:48:01.8706108Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8707118Z Vendor Name: AMD 2024-11-01T16:48:01.8708086Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8709033Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8710012Z Float Round Mode: NEAR 2024-11-01T16:48:01.8710913Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8711727Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8712533Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8713346Z Queue Type: MULTI 2024-11-01T16:48:01.8714112Z Node: 4 2024-11-01T16:48:01.8714865Z Device Type: GPU 2024-11-01T16:48:01.8715543Z Cache Info: 2024-11-01T16:48:01.8716192Z L1: 16(0x10) KB 2024-11-01T16:48:01.8717080Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8718370Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8719206Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8720024Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8720860Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8721655Z BDFID: 4352 2024-11-01T16:48:01.8722453Z Internal Node ID: 4 2024-11-01T16:48:01.8723265Z Compute Unit: 104 2024-11-01T16:48:01.8724056Z SIMDs per CU: 4 2024-11-01T16:48:01.8724854Z Shader Engines: 8 2024-11-01T16:48:01.8725684Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8726537Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8727409Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8728123Z Memory Properties: 2024-11-01T16:48:01.8728693Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8729463Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8730282Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8731109Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8731853Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8732512Z x 1024(0x400) 2024-11-01T16:48:01.8733231Z y 1024(0x400) 2024-11-01T16:48:01.8733940Z z 1024(0x400) 2024-11-01T16:48:01.8734712Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8735667Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8736637Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8737350Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8737947Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8738669Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8739374Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8740464Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8741358Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8742210Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8743089Z IOMMU Support:: None 2024-11-01T16:48:01.8743897Z Pool Info: 2024-11-01T16:48:01.8744417Z Pool 1 2024-11-01T16:48:01.8745100Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8745941Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8746906Z Allocatable: TRUE 2024-11-01T16:48:01.8747921Z Alloc Granule: 4KB 2024-11-01T16:48:01.8748965Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8750033Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8750978Z Accessible by all: FALSE 2024-11-01T16:48:01.8751687Z Pool 2 2024-11-01T16:48:01.8752359Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8753199Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8754000Z Allocatable: TRUE 2024-11-01T16:48:01.8754849Z Alloc Granule: 4KB 2024-11-01T16:48:01.8755719Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8757116Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8758130Z Accessible by all: FALSE 2024-11-01T16:48:01.8758940Z Pool 3 2024-11-01T16:48:01.8759584Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8760394Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8761188Z Allocatable: TRUE 2024-11-01T16:48:01.8762024Z Alloc Granule: 4KB 2024-11-01T16:48:01.8762882Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8763775Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8764624Z Accessible by all: FALSE 2024-11-01T16:48:01.8765323Z Pool 4 2024-11-01T16:48:01.8765957Z Segment: GROUP 2024-11-01T16:48:01.8766726Z Size: 64(0x40) KB 2024-11-01T16:48:01.8767510Z Allocatable: FALSE 2024-11-01T16:48:01.8768337Z Alloc Granule: 0KB 2024-11-01T16:48:01.8769203Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8770092Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8770930Z Accessible by all: FALSE 2024-11-01T16:48:01.8771633Z ISA Info: 2024-11-01T16:48:01.8772134Z ISA 1 2024-11-01T16:48:01.8772919Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8773826Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8774705Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8775559Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8776571Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8777408Z Fast f16: TRUE 2024-11-01T16:48:01.8778230Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8779293Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8779995Z x 1024(0x400) 2024-11-01T16:48:01.8780734Z y 1024(0x400) 2024-11-01T16:48:01.8781478Z z 1024(0x400) 2024-11-01T16:48:01.8782280Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8783022Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8783646Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8784393Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8785144Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8785945Z FBarrier Max Size: 32 2024-11-01T16:48:01.8786765Z ******* 2024-11-01T16:48:01.8787314Z Agent 6 2024-11-01T16:48:01.8787860Z ******* 2024-11-01T16:48:01.8788543Z Name: gfx90a 2024-11-01T16:48:01.8789599Z Uuid: GPU-8ad9228e7f5394ba 2024-11-01T16:48:01.8790588Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8791462Z Vendor Name: AMD 2024-11-01T16:48:01.8792274Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8793065Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8794199Z Float Round Mode: NEAR 2024-11-01T16:48:01.8795039Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8795852Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8797183Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8798139Z Queue Type: MULTI 2024-11-01T16:48:01.8799021Z Node: 5 2024-11-01T16:48:01.8799772Z Device Type: GPU 2024-11-01T16:48:01.8800450Z Cache Info: 2024-11-01T16:48:01.8801046Z L1: 16(0x10) KB 2024-11-01T16:48:01.8801784Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8802557Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8803357Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8804163Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8804994Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8805780Z BDFID: 5120 2024-11-01T16:48:01.8806559Z Internal Node ID: 5 2024-11-01T16:48:01.8807374Z Compute Unit: 104 2024-11-01T16:48:01.8808165Z SIMDs per CU: 4 2024-11-01T16:48:01.8808972Z Shader Engines: 8 2024-11-01T16:48:01.8809794Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8810646Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8811502Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8812238Z Memory Properties: 2024-11-01T16:48:01.8812801Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8813567Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8814393Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8815222Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8815955Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8817150Z x 1024(0x400) 2024-11-01T16:48:01.8817880Z y 1024(0x400) 2024-11-01T16:48:01.8818589Z z 1024(0x400) 2024-11-01T16:48:01.8819356Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8820320Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8821172Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8821887Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8822494Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8823203Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8823914Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8824691Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8825580Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8826488Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8827443Z IOMMU Support:: None 2024-11-01T16:48:01.8828263Z Pool Info: 2024-11-01T16:48:01.8828852Z Pool 1 2024-11-01T16:48:01.8829637Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8830641Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8831787Z Allocatable: TRUE 2024-11-01T16:48:01.8832635Z Alloc Granule: 4KB 2024-11-01T16:48:01.8833523Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8834426Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8835287Z Accessible by all: FALSE 2024-11-01T16:48:01.8836016Z Pool 2 2024-11-01T16:48:01.8836778Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8837776Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8838711Z Allocatable: TRUE 2024-11-01T16:48:01.8839579Z Alloc Granule: 4KB 2024-11-01T16:48:01.8840445Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8841333Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8842179Z Accessible by all: FALSE 2024-11-01T16:48:01.8842885Z Pool 3 2024-11-01T16:48:01.8843520Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8844315Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8845120Z Allocatable: TRUE 2024-11-01T16:48:01.8845940Z Alloc Granule: 4KB 2024-11-01T16:48:01.8846824Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8847700Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8848543Z Accessible by all: FALSE 2024-11-01T16:48:01.8849246Z Pool 4 2024-11-01T16:48:01.8849862Z Segment: GROUP 2024-11-01T16:48:01.8850633Z Size: 64(0x40) KB 2024-11-01T16:48:01.8851412Z Allocatable: FALSE 2024-11-01T16:48:01.8852236Z Alloc Granule: 0KB 2024-11-01T16:48:01.8853088Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8854260Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8855128Z Accessible by all: FALSE 2024-11-01T16:48:01.8855830Z ISA Info: 2024-11-01T16:48:01.8856413Z ISA 1 2024-11-01T16:48:01.8857216Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8858121Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8858976Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8859848Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8860739Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8861564Z Fast f16: TRUE 2024-11-01T16:48:01.8862414Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8863171Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8863866Z x 1024(0x400) 2024-11-01T16:48:01.8864601Z y 1024(0x400) 2024-11-01T16:48:01.8865334Z z 1024(0x400) 2024-11-01T16:48:01.8866142Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8867007Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8867761Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8869006Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8869882Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8870833Z FBarrier Max Size: 32 2024-11-01T16:48:01.8871538Z ******* 2024-11-01T16:48:01.8871998Z Agent 7 2024-11-01T16:48:01.8872471Z ******* 2024-11-01T16:48:01.8873085Z Name: gfx90a 2024-11-01T16:48:01.8873998Z Uuid: GPU-bdbf1ce15a4e2cd8 2024-11-01T16:48:01.8874855Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8875681Z Vendor Name: AMD 2024-11-01T16:48:01.8876536Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8877503Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8878455Z Float Round Mode: NEAR 2024-11-01T16:48:01.8879386Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8880200Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8881002Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8881806Z Queue Type: MULTI 2024-11-01T16:48:01.8882564Z Node: 6 2024-11-01T16:48:01.8883316Z Device Type: GPU 2024-11-01T16:48:01.8883993Z Cache Info: 2024-11-01T16:48:01.8884597Z L1: 16(0x10) KB 2024-11-01T16:48:01.8885334Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8886106Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8886898Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8887717Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8888526Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8889316Z BDFID: 44544 2024-11-01T16:48:01.8890102Z Internal Node ID: 6 2024-11-01T16:48:01.8890905Z Compute Unit: 104 2024-11-01T16:48:01.8891983Z SIMDs per CU: 4 2024-11-01T16:48:01.8892805Z Shader Engines: 8 2024-11-01T16:48:01.8893623Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8894481Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8895339Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8896054Z Memory Properties: 2024-11-01T16:48:01.8896906Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8897682Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8898506Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8899332Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8900065Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8900725Z x 1024(0x400) 2024-11-01T16:48:01.8901452Z y 1024(0x400) 2024-11-01T16:48:01.8902158Z z 1024(0x400) 2024-11-01T16:48:01.8902952Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8903910Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8904760Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8905480Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8906446Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8907297Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8908140Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8909070Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8910106Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8910753Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8911222Z IOMMU Support:: None 2024-11-01T16:48:01.8911606Z Pool Info: 2024-11-01T16:48:01.8911891Z Pool 1 2024-11-01T16:48:01.8912258Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8912728Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8913189Z Allocatable: TRUE 2024-11-01T16:48:01.8913661Z Alloc Granule: 4KB 2024-11-01T16:48:01.8914144Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8914646Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8915120Z Accessible by all: FALSE 2024-11-01T16:48:01.8915513Z Pool 2 2024-11-01T16:48:01.8915885Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8916351Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8916794Z Allocatable: TRUE 2024-11-01T16:48:01.8917255Z Alloc Granule: 4KB 2024-11-01T16:48:01.8917735Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8918222Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8918710Z Accessible by all: FALSE 2024-11-01T16:48:01.8919103Z Pool 3 2024-11-01T16:48:01.8919452Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8919896Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8920330Z Allocatable: TRUE 2024-11-01T16:48:01.8920972Z Alloc Granule: 4KB 2024-11-01T16:48:01.8921462Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8921955Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8922424Z Accessible by all: FALSE 2024-11-01T16:48:01.8922811Z Pool 4 2024-11-01T16:48:01.8923156Z Segment: GROUP 2024-11-01T16:48:01.8923586Z Size: 64(0x40) KB 2024-11-01T16:48:01.8924021Z Allocatable: FALSE 2024-11-01T16:48:01.8924479Z Alloc Granule: 0KB 2024-11-01T16:48:01.8924960Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8925445Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8925926Z Accessible by all: FALSE 2024-11-01T16:48:01.8926318Z ISA Info: 2024-11-01T16:48:01.8926587Z ISA 1 2024-11-01T16:48:01.8927032Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8927536Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8928017Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8928488Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8929126Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8929591Z Fast f16: TRUE 2024-11-01T16:48:01.8930041Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8930458Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8930841Z x 1024(0x400) 2024-11-01T16:48:01.8931250Z y 1024(0x400) 2024-11-01T16:48:01.8931656Z z 1024(0x400) 2024-11-01T16:48:01.8932107Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8932515Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8932866Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8933273Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8933689Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8934140Z FBarrier Max Size: 32 2024-11-01T16:48:01.8934531Z ******* 2024-11-01T16:48:01.8934794Z Agent 8 2024-11-01T16:48:01.8935043Z ******* 2024-11-01T16:48:01.8935358Z Name: gfx90a 2024-11-01T16:48:01.8935853Z Uuid: GPU-fbca43c355c8b7ae 2024-11-01T16:48:01.8936400Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8936873Z Vendor Name: AMD 2024-11-01T16:48:01.8937323Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8937759Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8938209Z Float Round Mode: NEAR 2024-11-01T16:48:01.8938681Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8939125Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8939574Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8940021Z Queue Type: MULTI 2024-11-01T16:48:01.8940435Z Node: 7 2024-11-01T16:48:01.8941038Z Device Type: GPU 2024-11-01T16:48:01.8941426Z Cache Info: 2024-11-01T16:48:01.8941756Z L1: 16(0x10) KB 2024-11-01T16:48:01.8942169Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8942604Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8943041Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8943482Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8943948Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8944384Z BDFID: 45824 2024-11-01T16:48:01.8944813Z Internal Node ID: 7 2024-11-01T16:48:01.8945278Z Compute Unit: 104 2024-11-01T16:48:01.8945718Z SIMDs per CU: 4 2024-11-01T16:48:01.8946157Z Shader Engines: 8 2024-11-01T16:48:01.8946619Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8947101Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8947566Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8947967Z Memory Properties: 2024-11-01T16:48:01.8948282Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8948905Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8949367Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8949827Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8950235Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8950596Z x 1024(0x400) 2024-11-01T16:48:01.8950995Z y 1024(0x400) 2024-11-01T16:48:01.8951394Z z 1024(0x400) 2024-11-01T16:48:01.8951814Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8952347Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8952819Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8953212Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8953548Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8953945Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8954334Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8954783Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.8955258Z Packet Processor uCode:: 83 2024-11-01T16:48:01.8955720Z SDMA engine uCode:: 8 2024-11-01T16:48:01.8956179Z IOMMU Support:: None 2024-11-01T16:48:01.8956569Z Pool Info: 2024-11-01T16:48:01.8956847Z Pool 1 2024-11-01T16:48:01.8957213Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.8957678Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8958134Z Allocatable: TRUE 2024-11-01T16:48:01.8958599Z Alloc Granule: 4KB 2024-11-01T16:48:01.8959090Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8959584Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8960058Z Accessible by all: FALSE 2024-11-01T16:48:01.8960449Z Pool 2 2024-11-01T16:48:01.8960823Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.8961427Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8961882Z Allocatable: TRUE 2024-11-01T16:48:01.8962351Z Alloc Granule: 4KB 2024-11-01T16:48:01.8962834Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8963332Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8963804Z Accessible by all: FALSE 2024-11-01T16:48:01.8964202Z Pool 3 2024-11-01T16:48:01.8964553Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.8964997Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.8965435Z Allocatable: TRUE 2024-11-01T16:48:01.8965892Z Alloc Granule: 4KB 2024-11-01T16:48:01.8966378Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.8966867Z Alloc Alignment: 4KB 2024-11-01T16:48:01.8967333Z Accessible by all: FALSE 2024-11-01T16:48:01.8967723Z Pool 4 2024-11-01T16:48:01.8968068Z Segment: GROUP 2024-11-01T16:48:01.8968499Z Size: 64(0x40) KB 2024-11-01T16:48:01.8969075Z Allocatable: FALSE 2024-11-01T16:48:01.8969534Z Alloc Granule: 0KB 2024-11-01T16:48:01.8970007Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.8970500Z Alloc Alignment: 0KB 2024-11-01T16:48:01.8970971Z Accessible by all: FALSE 2024-11-01T16:48:01.8971367Z ISA Info: 2024-11-01T16:48:01.8971640Z ISA 1 2024-11-01T16:48:01.8972075Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.8972576Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.8973048Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.8973525Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8974017Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.8974480Z Fast f16: TRUE 2024-11-01T16:48:01.8974933Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8975358Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8975737Z x 1024(0x400) 2024-11-01T16:48:01.8976154Z y 1024(0x400) 2024-11-01T16:48:01.8976659Z z 1024(0x400) 2024-11-01T16:48:01.8977091Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8977500Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8977852Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8978263Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8978673Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8979125Z FBarrier Max Size: 32 2024-11-01T16:48:01.8979516Z ******* 2024-11-01T16:48:01.8979769Z Agent 9 2024-11-01T16:48:01.8980024Z ******* 2024-11-01T16:48:01.8980340Z Name: gfx90a 2024-11-01T16:48:01.8980826Z Uuid: GPU-224bf7aab4354d06 2024-11-01T16:48:01.8981455Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.8981932Z Vendor Name: AMD 2024-11-01T16:48:01.8982373Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.8982827Z Profile: BASE_PROFILE 2024-11-01T16:48:01.8983280Z Float Round Mode: NEAR 2024-11-01T16:48:01.8983737Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.8984194Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.8984643Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.8985080Z Queue Type: MULTI 2024-11-01T16:48:01.8985497Z Node: 8 2024-11-01T16:48:01.8985918Z Device Type: GPU 2024-11-01T16:48:01.8986303Z Cache Info: 2024-11-01T16:48:01.8986628Z L1: 16(0x10) KB 2024-11-01T16:48:01.8987040Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.8987464Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.8987896Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.8988346Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.8988798Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.8989392Z BDFID: 36352 2024-11-01T16:48:01.8989837Z Internal Node ID: 8 2024-11-01T16:48:01.8990284Z Compute Unit: 104 2024-11-01T16:48:01.8990715Z SIMDs per CU: 4 2024-11-01T16:48:01.8991165Z Shader Engines: 8 2024-11-01T16:48:01.8991625Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.8992091Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.8992566Z Coherent Host Access: FALSE 2024-11-01T16:48:01.8992968Z Memory Properties: 2024-11-01T16:48:01.8993279Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.8993708Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.8994171Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.8994622Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.8995022Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.8995388Z x 1024(0x400) 2024-11-01T16:48:01.8995784Z y 1024(0x400) 2024-11-01T16:48:01.8996178Z z 1024(0x400) 2024-11-01T16:48:01.8996616Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.8997152Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.8997621Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.8998021Z Grid Max Size per Dimension: 2024-11-01T16:48:01.8998354Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.8998750Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.8999154Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.8999593Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.9000072Z Packet Processor uCode:: 83 2024-11-01T16:48:01.9000538Z SDMA engine uCode:: 8 2024-11-01T16:48:01.9000995Z IOMMU Support:: None 2024-11-01T16:48:01.9001516Z Pool Info: 2024-11-01T16:48:01.9001800Z Pool 1 2024-11-01T16:48:01.9002178Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.9002643Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9003094Z Allocatable: TRUE 2024-11-01T16:48:01.9003562Z Alloc Granule: 4KB 2024-11-01T16:48:01.9004050Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9004549Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9005021Z Accessible by all: FALSE 2024-11-01T16:48:01.9005415Z Pool 2 2024-11-01T16:48:01.9005779Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.9006246Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9006693Z Allocatable: TRUE 2024-11-01T16:48:01.9007152Z Alloc Granule: 4KB 2024-11-01T16:48:01.9007645Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9008138Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9008612Z Accessible by all: FALSE 2024-11-01T16:48:01.9009012Z Pool 3 2024-11-01T16:48:01.9009519Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.9009967Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9010407Z Allocatable: TRUE 2024-11-01T16:48:01.9010867Z Alloc Granule: 4KB 2024-11-01T16:48:01.9011349Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9011840Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9012311Z Accessible by all: FALSE 2024-11-01T16:48:01.9012705Z Pool 4 2024-11-01T16:48:01.9013049Z Segment: GROUP 2024-11-01T16:48:01.9013473Z Size: 64(0x40) KB 2024-11-01T16:48:01.9013906Z Allocatable: FALSE 2024-11-01T16:48:01.9014363Z Alloc Granule: 0KB 2024-11-01T16:48:01.9014851Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.9015339Z Alloc Alignment: 0KB 2024-11-01T16:48:01.9015804Z Accessible by all: FALSE 2024-11-01T16:48:01.9016192Z ISA Info: 2024-11-01T16:48:01.9016623Z ISA 1 2024-11-01T16:48:01.9017069Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.9017571Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.9018049Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.9018526Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.9019014Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.9019480Z Fast f16: TRUE 2024-11-01T16:48:01.9019936Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.9020348Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.9020732Z x 1024(0x400) 2024-11-01T16:48:01.9021150Z y 1024(0x400) 2024-11-01T16:48:01.9021720Z z 1024(0x400) 2024-11-01T16:48:01.9022176Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.9022592Z Grid Max Size per Dimension: 2024-11-01T16:48:01.9022945Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.9023364Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.9023781Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.9024228Z FBarrier Max Size: 32 2024-11-01T16:48:01.9024614Z ******* 2024-11-01T16:48:01.9024874Z Agent 10 2024-11-01T16:48:01.9025132Z ******* 2024-11-01T16:48:01.9025438Z Name: gfx90a 2024-11-01T16:48:01.9025940Z Uuid: GPU-8473d8ace24d57eb 2024-11-01T16:48:01.9026408Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:48:01.9026859Z Vendor Name: AMD 2024-11-01T16:48:01.9027314Z Feature: KERNEL_DISPATCH 2024-11-01T16:48:01.9027761Z Profile: BASE_PROFILE 2024-11-01T16:48:01.9028206Z Float Round Mode: NEAR 2024-11-01T16:48:01.9028671Z Max Queue Number: 128(0x80) 2024-11-01T16:48:01.9029124Z Queue Min Size: 64(0x40) 2024-11-01T16:48:01.9029767Z Queue Max Size: 131072(0x20000) 2024-11-01T16:48:01.9030210Z Queue Type: MULTI 2024-11-01T16:48:01.9030633Z Node: 9 2024-11-01T16:48:01.9031058Z Device Type: GPU 2024-11-01T16:48:01.9031430Z Cache Info: 2024-11-01T16:48:01.9031769Z L1: 16(0x10) KB 2024-11-01T16:48:01.9032183Z L2: 8192(0x2000) KB 2024-11-01T16:48:01.9032603Z Chip ID: 29708(0x740c) 2024-11-01T16:48:01.9033043Z ASIC Revision: 1(0x1) 2024-11-01T16:48:01.9033493Z Cacheline Size: 64(0x40) 2024-11-01T16:48:01.9033946Z Max Clock Freq. (MHz): 1700 2024-11-01T16:48:01.9034391Z BDFID: 37632 2024-11-01T16:48:01.9034825Z Internal Node ID: 9 2024-11-01T16:48:01.9035269Z Compute Unit: 104 2024-11-01T16:48:01.9035704Z SIMDs per CU: 4 2024-11-01T16:48:01.9036147Z Shader Engines: 8 2024-11-01T16:48:01.9036603Z Shader Arrs. per Eng.: 1 2024-11-01T16:48:01.9037070Z WatchPts on Addr. Ranges:4 2024-11-01T16:48:01.9037546Z Coherent Host Access: FALSE 2024-11-01T16:48:01.9037948Z Memory Properties: 2024-11-01T16:48:01.9038257Z Features: KERNEL_DISPATCH 2024-11-01T16:48:01.9038679Z Fast F16 Operation: TRUE 2024-11-01T16:48:01.9039138Z Wavefront Size: 64(0x40) 2024-11-01T16:48:01.9039599Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.9040006Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.9040373Z x 1024(0x400) 2024-11-01T16:48:01.9040764Z y 1024(0x400) 2024-11-01T16:48:01.9041153Z z 1024(0x400) 2024-11-01T16:48:01.9041725Z Max Waves Per CU: 32(0x20) 2024-11-01T16:48:01.9042264Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:48:01.9042732Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.9043137Z Grid Max Size per Dimension: 2024-11-01T16:48:01.9043466Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.9043865Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.9044261Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.9044698Z Max fbarriers/Workgrp: 32 2024-11-01T16:48:01.9045181Z Packet Processor uCode:: 83 2024-11-01T16:48:01.9045651Z SDMA engine uCode:: 8 2024-11-01T16:48:01.9046101Z IOMMU Support:: None 2024-11-01T16:48:01.9046476Z Pool Info: 2024-11-01T16:48:01.9046762Z Pool 1 2024-11-01T16:48:01.9047147Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:48:01.9047611Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9048068Z Allocatable: TRUE 2024-11-01T16:48:01.9048536Z Alloc Granule: 4KB 2024-11-01T16:48:01.9049024Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9049662Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9050138Z Accessible by all: FALSE 2024-11-01T16:48:01.9050528Z Pool 2 2024-11-01T16:48:01.9050902Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:48:01.9051361Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9051802Z Allocatable: TRUE 2024-11-01T16:48:01.9052267Z Alloc Granule: 4KB 2024-11-01T16:48:01.9052748Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9053241Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9053709Z Accessible by all: FALSE 2024-11-01T16:48:01.9054099Z Pool 3 2024-11-01T16:48:01.9054453Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:48:01.9054897Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:48:01.9055340Z Allocatable: TRUE 2024-11-01T16:48:01.9055799Z Alloc Granule: 4KB 2024-11-01T16:48:01.9056332Z Alloc Recommended Granule:2048KB 2024-11-01T16:48:01.9056822Z Alloc Alignment: 4KB 2024-11-01T16:48:01.9057292Z Accessible by all: FALSE 2024-11-01T16:48:01.9057672Z Pool 4 2024-11-01T16:48:01.9058018Z Segment: GROUP 2024-11-01T16:48:01.9058442Z Size: 64(0x40) KB 2024-11-01T16:48:01.9058864Z Allocatable: FALSE 2024-11-01T16:48:01.9059333Z Alloc Granule: 0KB 2024-11-01T16:48:01.9059813Z Alloc Recommended Granule:0KB 2024-11-01T16:48:01.9060305Z Alloc Alignment: 0KB 2024-11-01T16:48:01.9060767Z Accessible by all: FALSE 2024-11-01T16:48:01.9061159Z ISA Info: 2024-11-01T16:48:01.9061433Z ISA 1 2024-11-01T16:48:01.9062024Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:48:01.9062535Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:48:01.9063011Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:48:01.9063485Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.9063970Z Default Rounding Mode: NEAR 2024-11-01T16:48:01.9064428Z Fast f16: TRUE 2024-11-01T16:48:01.9064884Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:48:01.9065300Z Workgroup Max Size per Dimension: 2024-11-01T16:48:01.9065682Z x 1024(0x400) 2024-11-01T16:48:01.9066089Z y 1024(0x400) 2024-11-01T16:48:01.9066497Z z 1024(0x400) 2024-11-01T16:48:01.9066945Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:48:01.9067354Z Grid Max Size per Dimension: 2024-11-01T16:48:01.9067698Z x 4294967295(0xffffffff) 2024-11-01T16:48:01.9068106Z y 4294967295(0xffffffff) 2024-11-01T16:48:01.9068520Z z 4294967295(0xffffffff) 2024-11-01T16:48:01.9068964Z FBarrier Max Size: 32 2024-11-01T16:48:01.9069548Z *** Done *** 2024-11-01T16:48:01.9090455Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-11-01T16:48:01.9090978Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-11-01T16:48:01.9091872Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-11-01T16:48:01.9092691Z if [[ $ngpu -eq 0 ]]; then 2024-11-01T16:48:01.9093142Z  echo "Error: Failed to detect any GPUs on the runner" 2024-11-01T16:48:01.9093576Z  echo "$msg" 2024-11-01T16:48:01.9093846Z  exit 1 2024-11-01T16:48:01.9094092Z fi 2024-11-01T16:48:01.9094326Z if [[ $ngpu -eq 1 ]]; then 2024-11-01T16:48:01.9094881Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-11-01T16:48:01.9095432Z  echo "$msg" 2024-11-01T16:48:01.9095692Z  exit 1 2024-11-01T16:48:01.9095938Z fi 2024-11-01T16:48:01.9115793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:01.9116218Z env: 2024-11-01T16:48:01.9116440Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:01.9116781Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:01.9117169Z ##[endgroup] 2024-11-01T16:48:02.1073045Z Prepare all required actions 2024-11-01T16:48:02.1145691Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-11-01T16:48:02.1146517Z with: 2024-11-01T16:48:02.1147000Z diskspace-cutoff: 70 2024-11-01T16:48:02.1147555Z env: 2024-11-01T16:48:02.1148064Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:02.1148826Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:02.1149672Z ##[endgroup] 2024-11-01T16:48:02.1269084Z ##[group]Run set -ex 2024-11-01T16:48:02.1269619Z set -ex 2024-11-01T16:48:02.1270090Z diskspace_cutoff=70 2024-11-01T16:48:02.1270879Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-11-01T16:48:02.1272412Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-11-01T16:48:02.1274498Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-11-01T16:48:02.1276136Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-11-01T16:48:02.1276920Z  docker system prune -af 2024-11-01T16:48:02.1278341Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-11-01T16:48:02.1279618Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-11-01T16:48:02.1280951Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-11-01T16:48:02.1282097Z  echo "$msg" 2024-11-01T16:48:02.1282621Z  exit 1 2024-11-01T16:48:02.1283089Z  else 2024-11-01T16:48:02.1283697Z  difference=$((diskspace - diskspace_new)) 2024-11-01T16:48:02.1284570Z  echo "Diskspace saved: $difference percent" 2024-11-01T16:48:02.1285283Z  fi 2024-11-01T16:48:02.1285692Z fi 2024-11-01T16:48:02.1320192Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:02.1320962Z env: 2024-11-01T16:48:02.1321361Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:02.1322006Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:02.1322687Z ##[endgroup] 2024-11-01T16:48:02.1395486Z + diskspace_cutoff=70 2024-11-01T16:48:02.1403750Z ++ docker info -f '{{.DockerRootDir}}' 2024-11-01T16:48:02.2319305Z + docker_root_dir=/media/4TB/docker-rootless 2024-11-01T16:48:02.2334664Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-11-01T16:48:02.2337963Z ++ sed -n 2p 2024-11-01T16:48:02.2341456Z ++ sed s/%// 2024-11-01T16:48:02.2345326Z ++ sed 's/ //' 2024-11-01T16:48:02.2387113Z + diskspace=26 2024-11-01T16:48:02.2389664Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-11-01T16:48:02.2391538Z + [[ 26 -ge 70 ]] 2024-11-01T16:48:02.2454621Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:48:02.2455910Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:48:02.2457285Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-11-01T16:48:02.2498131Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:02.2499021Z env: 2024-11-01T16:48:02.2499481Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:02.2500237Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:02.2501049Z ##[endgroup] 2024-11-01T16:48:02.2712090Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-11-01T16:48:02.2713909Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-11-01T16:48:02.2715957Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-11-01T16:48:02.2755796Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:02.2756642Z env: 2024-11-01T16:48:02.2757088Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:02.2757816Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:02.2758629Z ##[endgroup] 2024-11-01T16:48:02.2986168Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-11-01T16:48:02.2987187Z with: 2024-11-01T16:48:02.2988165Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-11-01T16:48:02.2989396Z aws-region: us-east-1 2024-11-01T16:48:02.2990080Z role-duration-seconds: 18000 2024-11-01T16:48:02.2990841Z audience: sts.amazonaws.com 2024-11-01T16:48:02.2991498Z env: 2024-11-01T16:48:02.2991997Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:02.2992874Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:02.2994441Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:02.2995803Z ##[endgroup] 2024-11-01T16:48:02.7786996Z Assuming role with OIDC 2024-11-01T16:48:02.9565841Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-11-01T16:48:03.0294244Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-11-01T16:48:03.0295184Z with: 2024-11-01T16:48:03.0295760Z mask-password: true 2024-11-01T16:48:03.0296829Z registry-type: private 2024-11-01T16:48:03.0297538Z skip-logout: false 2024-11-01T16:48:03.0298139Z env: 2024-11-01T16:48:03.0298689Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:03.0299583Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:03.0301203Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:03.0302677Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:03.0303470Z AWS_REGION: us-east-1 2024-11-01T16:48:03.0304381Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:03.0305465Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:03.0320960Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:03.0321718Z ##[endgroup] 2024-11-01T16:48:03.3628855Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7007967Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-11-01T16:48:03.7009108Z with: 2024-11-01T16:48:03.7010661Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7012443Z docker-build-dir: .ci/docker 2024-11-01T16:48:03.7013130Z working-directory: . 2024-11-01T16:48:03.7013983Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7014926Z force-push: false 2024-11-01T16:48:03.7015459Z env: 2024-11-01T16:48:03.7015955Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:03.7017893Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:03.7019575Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:03.7021097Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:03.7022139Z AWS_REGION: us-east-1 2024-11-01T16:48:03.7023340Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:03.7024668Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:03.7037641Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:03.7038587Z ##[endgroup] 2024-11-01T16:48:03.7093838Z ##[group]Run set -ex 2024-11-01T16:48:03.7095012Z set -ex 2024-11-01T16:48:03.7096151Z  2024-11-01T16:48:03.7097858Z # If the docker build directory or the build script doesn't exist, the action will 2024-11-01T16:48:03.7100223Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-11-01T16:48:03.7102912Z # job could then download the pre-built image as usual 2024-11-01T16:48:03.7104830Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-11-01T16:48:03.7106847Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7108496Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7109976Z  2024-11-01T16:48:03.7111521Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-11-01T16:48:03.7113137Z  exit 0 2024-11-01T16:48:03.7114230Z else 2024-11-01T16:48:03.7115290Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7116530Z fi 2024-11-01T16:48:03.7117494Z  2024-11-01T16:48:03.7118789Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-11-01T16:48:03.7120957Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-11-01T16:48:03.7123054Z  # use it as it is, but first let's extract the tag 2024-11-01T16:48:03.7124878Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-11-01T16:48:03.7126814Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7128787Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7130281Z else 2024-11-01T16:48:03.7131661Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-11-01T16:48:03.7133392Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7135820Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7137851Z fi 2024-11-01T16:48:03.7180811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:03.7182235Z env: 2024-11-01T16:48:03.7183278Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:03.7184520Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:03.7186557Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:03.7188664Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:03.7189795Z AWS_REGION: us-east-1 2024-11-01T16:48:03.7191104Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:03.7193129Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:03.7208551Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:03.7209828Z REPO_NAME: pytorch 2024-11-01T16:48:03.7212031Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7214370Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:48:03.7215921Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7217426Z ##[endgroup] 2024-11-01T16:48:03.7307763Z + [[ ! -d .ci/docker ]] 2024-11-01T16:48:03.7308750Z + [[ ! -f .ci/docker/build.sh ]] 2024-11-01T16:48:03.7309861Z + echo skip=false 2024-11-01T16:48:03.7313296Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-11-01T16:48:03.7325936Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7328558Z ++ awk -F '[:,]' '{print $2}' 2024-11-01T16:48:03.7382435Z + DOCKER_TAG=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7384487Z + echo docker-tag=bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7387355Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7478903Z ##[group]Run set +e 2024-11-01T16:48:03.7480110Z set +e 2024-11-01T16:48:03.7481054Z set -x 2024-11-01T16:48:03.7482006Z  2024-11-01T16:48:03.7482779Z login() { 2024-11-01T16:48:03.7484285Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:48:03.7485957Z } 2024-11-01T16:48:03.7486682Z  2024-11-01T16:48:03.7487455Z retry () { 2024-11-01T16:48:03.7488578Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:48:03.7489665Z } 2024-11-01T16:48:03.7490537Z  2024-11-01T16:48:03.7491337Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:48:03.7492494Z  2024-11-01T16:48:03.7493801Z # Check if image already exists, if it does then skip building it 2024-11-01T16:48:03.7495321Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-11-01T16:48:03.7496996Z  exit 0 2024-11-01T16:48:03.7497822Z fi 2024-11-01T16:48:03.7498537Z  2024-11-01T16:48:03.7499800Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-11-01T16:48:03.7501555Z # be empty. The default action would be to continue rebuild the image 2024-11-01T16:48:03.7503091Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-11-01T16:48:03.7504728Z  # if we're on the base branch then use the parent commit 2024-11-01T16:48:03.7506153Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-11-01T16:48:03.7507292Z else 2024-11-01T16:48:03.7508409Z  # otherwise we're on a PR, so use the most recent base commit 2024-11-01T16:48:03.7509857Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-11-01T16:48:03.7511167Z fi 2024-11-01T16:48:03.7511912Z  2024-11-01T16:48:03.7512726Z if [[ -z "${MERGE_BASE}" ]]; then 2024-11-01T16:48:03.7514055Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7515104Z  2024-11-01T16:48:03.7516690Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-11-01T16:48:03.7518425Z  exit 0 2024-11-01T16:48:03.7519212Z fi 2024-11-01T16:48:03.7520127Z  2024-11-01T16:48:03.7521236Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-11-01T16:48:03.7523856Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-11-01T16:48:03.7525573Z  exit 1 2024-11-01T16:48:03.7526411Z fi 2024-11-01T16:48:03.7527302Z  2024-11-01T16:48:03.7528425Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-11-01T16:48:03.7530385Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-11-01T16:48:03.7532440Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-11-01T16:48:03.7534445Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-11-01T16:48:03.7537355Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-11-01T16:48:03.7538779Z fi 2024-11-01T16:48:03.7539506Z  2024-11-01T16:48:03.7540511Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:03.7584586Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:03.7585626Z env: 2024-11-01T16:48:03.7586470Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:03.7587523Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:03.7588990Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:03.7590503Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:03.7591307Z AWS_REGION: us-east-1 2024-11-01T16:48:03.7592282Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:03.7593558Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:03.7604893Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:03.7605816Z DOCKER_BUILD_DIR: .ci/docker 2024-11-01T16:48:03.7606789Z BASE_REVISION: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:48:03.7608816Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7610802Z DOCKER_TAG: bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:03.7611996Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7613036Z ##[endgroup] 2024-11-01T16:48:03.7697184Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7698463Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:03.7705184Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:48:03.7708884Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:04.9219321Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2024-11-01T16:48:04.9220904Z Configure a credential helper to remove this warning. See 2024-11-01T16:48:04.9222585Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2024-11-01T16:48:04.9223563Z 2024-11-01T16:48:04.9223820Z Login Succeeded 2024-11-01T16:48:04.9260317Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:05.5972239Z { 2024-11-01T16:48:05.5972881Z "schemaVersion": 2, 2024-11-01T16:48:05.5974005Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-11-01T16:48:05.5975374Z "config": { 2024-11-01T16:48:05.5976534Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-11-01T16:48:05.5977588Z "size": 24320, 2024-11-01T16:48:05.5978664Z "digest": "sha256:4d076369ac6bdd477f80b455e0eafad40dbf413bb6bca80e5a8acefa29995600" 2024-11-01T16:48:05.5979865Z }, 2024-11-01T16:48:05.5980404Z "layers": [ 2024-11-01T16:48:05.5980924Z { 2024-11-01T16:48:05.5981802Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.5982869Z "size": 28583948, 2024-11-01T16:48:05.5983897Z "digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1" 2024-11-01T16:48:05.5985044Z }, 2024-11-01T16:48:05.5985504Z { 2024-11-01T16:48:05.5986960Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.5988058Z "size": 1822, 2024-11-01T16:48:05.5989115Z "digest": "sha256:162b6138f2b03f11bc7d166adcec6d7edc9543aff23647976a7ccf9c9eb42e73" 2024-11-01T16:48:05.5990281Z }, 2024-11-01T16:48:05.5990731Z { 2024-11-01T16:48:05.5991535Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.5992576Z "size": 312559684, 2024-11-01T16:48:05.5993621Z "digest": "sha256:f2772a057b93184d18e0a4ce2e0c7df9490978287bbaff68e6240a5de81d8827" 2024-11-01T16:48:05.5994765Z }, 2024-11-01T16:48:05.5995219Z { 2024-11-01T16:48:05.5996016Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.5997450Z "size": 864, 2024-11-01T16:48:05.5998477Z "digest": "sha256:a07e96d4579e0cbb485fcdb25d0777d240f0cd0fd8222ce7cea1056c9fdeec1d" 2024-11-01T16:48:05.5999644Z }, 2024-11-01T16:48:05.6000094Z { 2024-11-01T16:48:05.6000892Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6001912Z "size": 106, 2024-11-01T16:48:05.6002915Z "digest": "sha256:fc4226a98863f023397c2c581c98b11b40e1b116bc177f3c1fe9faf0dad9a7e7" 2024-11-01T16:48:05.6004053Z }, 2024-11-01T16:48:05.6004502Z { 2024-11-01T16:48:05.6005293Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6006304Z "size": 704, 2024-11-01T16:48:05.6007310Z "digest": "sha256:98e488266c9caf9e952b8351e89c1c082371ecd2ed67df55f5c7ccb59180e23b" 2024-11-01T16:48:05.6008454Z }, 2024-11-01T16:48:05.6008901Z { 2024-11-01T16:48:05.6009693Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6010722Z "size": 1260, 2024-11-01T16:48:05.6011775Z "digest": "sha256:631b0545c1dc2d77efadfcdfae1fe1fa0edb9ea6be6f7233885e79aab09802a6" 2024-11-01T16:48:05.6012954Z }, 2024-11-01T16:48:05.6013407Z { 2024-11-01T16:48:05.6014183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6015196Z "size": 3678, 2024-11-01T16:48:05.6016196Z "digest": "sha256:33613f42c2e6eec7e082c2c1119e142e3f9f04266c95cfb6f0b96baf51b99175" 2024-11-01T16:48:05.6017472Z }, 2024-11-01T16:48:05.6017916Z { 2024-11-01T16:48:05.6018689Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6019705Z "size": 1801, 2024-11-01T16:48:05.6020714Z "digest": "sha256:4f9ae8abe33756cfd7b80b90bd7b1af316c0394e58bf41e7c1586fa021e4509a" 2024-11-01T16:48:05.6021868Z }, 2024-11-01T16:48:05.6022313Z { 2024-11-01T16:48:05.6023095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6024107Z "size": 702, 2024-11-01T16:48:05.6025138Z "digest": "sha256:f45ff9cb5ae341e9958b6a54478fa78323dcab64bcecf414e207f196a31530f2" 2024-11-01T16:48:05.6026300Z }, 2024-11-01T16:48:05.6026741Z { 2024-11-01T16:48:05.6027517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6028564Z "size": 2684244768, 2024-11-01T16:48:05.6029901Z "digest": "sha256:79f90e3b35e3589cc7a225f3644dc07793ce58a89ee9d3930e5380de85858c8e" 2024-11-01T16:48:05.6031023Z }, 2024-11-01T16:48:05.6031471Z { 2024-11-01T16:48:05.6032256Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6033264Z "size": 380, 2024-11-01T16:48:05.6034258Z "digest": "sha256:26a1748cd9372ed1db6b0199fd5e837823c3d9b34e863ab4d8ff816af972f4e0" 2024-11-01T16:48:05.6035389Z }, 2024-11-01T16:48:05.6035842Z { 2024-11-01T16:48:05.6036620Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6037628Z "size": 12113, 2024-11-01T16:48:05.6038625Z "digest": "sha256:46077999b29ee2cd69577fb7695cf83271c7e559726ec7c6f5a2d4c9f93eae42" 2024-11-01T16:48:05.6039743Z }, 2024-11-01T16:48:05.6040187Z { 2024-11-01T16:48:05.6040967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6041965Z "size": 504, 2024-11-01T16:48:05.6043330Z "digest": "sha256:298f94ba17808fb4fa481ebed4e657c49d2f8d6e2de4d6c64f4ce4a5ddac5839" 2024-11-01T16:48:05.6044662Z }, 2024-11-01T16:48:05.6045117Z { 2024-11-01T16:48:05.6045895Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6046912Z "size": 121477455, 2024-11-01T16:48:05.6047972Z "digest": "sha256:b5de01ce3c49d8ea251ba514ecd44d5a1990afd04c6ae45b9a7ab35edad56038" 2024-11-01T16:48:05.6049147Z }, 2024-11-01T16:48:05.6049605Z { 2024-11-01T16:48:05.6050400Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6051426Z "size": 109, 2024-11-01T16:48:05.6052405Z "digest": "sha256:545e959ed0e880eef71900f4d98644d203829af6579697320009d114c075969a" 2024-11-01T16:48:05.6053880Z }, 2024-11-01T16:48:05.6054339Z { 2024-11-01T16:48:05.6055126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6056150Z "size": 490, 2024-11-01T16:48:05.6057489Z "digest": "sha256:0d6cd48338a2b75af7392982c4550ec831a096d07be2cc4b53e9f7a6fdcd04cc" 2024-11-01T16:48:05.6058633Z }, 2024-11-01T16:48:05.6059085Z { 2024-11-01T16:48:05.6059872Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6060880Z "size": 297, 2024-11-01T16:48:05.6061862Z "digest": "sha256:e189d4f6677473e231661cf42501a78bfc04fd8a55a429e6330fa7223d8fd8f2" 2024-11-01T16:48:05.6062977Z }, 2024-11-01T16:48:05.6063422Z { 2024-11-01T16:48:05.6064212Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6065226Z "size": 103, 2024-11-01T16:48:05.6066222Z "digest": "sha256:d6802c7f20ed5257008c8dc9d18c79c1196d1b1b390ccb307c9a0508b833b105" 2024-11-01T16:48:05.6067360Z }, 2024-11-01T16:48:05.6067808Z { 2024-11-01T16:48:05.6068604Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6069631Z "size": 1473, 2024-11-01T16:48:05.6070643Z "digest": "sha256:56a128ca4cdc55990dc1622c87d1c6b7de79c31829cc25560f9f020e903900c4" 2024-11-01T16:48:05.6071781Z }, 2024-11-01T16:48:05.6072240Z { 2024-11-01T16:48:05.6073035Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6074053Z "size": 426816767, 2024-11-01T16:48:05.6075093Z "digest": "sha256:777a3fad188d9b3d7842e6408fefb18952ba4c9cfd7b6d1615c6907804577114" 2024-11-01T16:48:05.6076218Z }, 2024-11-01T16:48:05.6076662Z { 2024-11-01T16:48:05.6077441Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6078448Z "size": 163, 2024-11-01T16:48:05.6079434Z "digest": "sha256:7bd376e7570dd04b940d25fa18e82cede2981f05cbc3772fc0041165174ca2d2" 2024-11-01T16:48:05.6080553Z }, 2024-11-01T16:48:05.6081010Z { 2024-11-01T16:48:05.6081792Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6082809Z "size": 1640, 2024-11-01T16:48:05.6083845Z "digest": "sha256:7beadf9babec249728f9f1f97764bab920f82b6dbfd6216150ab34aa6524f875" 2024-11-01T16:48:05.6084993Z }, 2024-11-01T16:48:05.6085436Z { 2024-11-01T16:48:05.6086218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6087239Z "size": 7842049879, 2024-11-01T16:48:05.6088268Z "digest": "sha256:d09605f5be3506ee239f9b2fd1080b1a94c9395e1d26592335204b510ba32987" 2024-11-01T16:48:05.6089392Z }, 2024-11-01T16:48:05.6089834Z { 2024-11-01T16:48:05.6090601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6091608Z "size": 105, 2024-11-01T16:48:05.6092600Z "digest": "sha256:23d9b59fc2d6ba444e4e9dcda52e784e75d092b74bcd343fe6449963eb93d0a6" 2024-11-01T16:48:05.6093746Z }, 2024-11-01T16:48:05.6094188Z { 2024-11-01T16:48:05.6094965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6095970Z "size": 974, 2024-11-01T16:48:05.6097061Z "digest": "sha256:32757d535f12983953ed7fa3e6c9105c523cc59f9de46009d99d2006effd9095" 2024-11-01T16:48:05.6098167Z }, 2024-11-01T16:48:05.6098613Z { 2024-11-01T16:48:05.6099847Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6100892Z "size": 318526641, 2024-11-01T16:48:05.6101897Z "digest": "sha256:176796974e42836ae83a90fd8a36f1c55314712beb7305e1b61fe62d90c9461a" 2024-11-01T16:48:05.6103004Z }, 2024-11-01T16:48:05.6103450Z { 2024-11-01T16:48:05.6104218Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6105220Z "size": 111, 2024-11-01T16:48:05.6106202Z "digest": "sha256:a1fb76efb5786598c6e880d39d27ce17817e0dcd5906512cb018e2892964d682" 2024-11-01T16:48:05.6107326Z }, 2024-11-01T16:48:05.6107771Z { 2024-11-01T16:48:05.6108542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6109887Z "size": 1829, 2024-11-01T16:48:05.6110876Z "digest": "sha256:fb133b23798c65c183654eb1d111cbe9d143d29870b86f8e43580b5b69e2caa9" 2024-11-01T16:48:05.6111988Z }, 2024-11-01T16:48:05.6112433Z { 2024-11-01T16:48:05.6113213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6114224Z "size": 619478538, 2024-11-01T16:48:05.6115216Z "digest": "sha256:0569484aa5b4385d29ef9490a19b583b672100a675957ac38746fb077b759849" 2024-11-01T16:48:05.6116308Z }, 2024-11-01T16:48:05.6116760Z { 2024-11-01T16:48:05.6117539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6118544Z "size": 167, 2024-11-01T16:48:05.6119522Z "digest": "sha256:eb69f898c8797ad483631ea108e76e54259cc907a09784dc8857a06a3ffe0e57" 2024-11-01T16:48:05.6120644Z }, 2024-11-01T16:48:05.6121098Z { 2024-11-01T16:48:05.6121861Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6122878Z "size": 2854540, 2024-11-01T16:48:05.6123893Z "digest": "sha256:283525ceebdf5dc275b9425fc557b2b8d424b8c3a9e2a3c073ff335160103d01" 2024-11-01T16:48:05.6125003Z }, 2024-11-01T16:48:05.6125444Z { 2024-11-01T16:48:05.6126220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6127213Z "size": 107, 2024-11-01T16:48:05.6128231Z "digest": "sha256:0d1abd111f4dd717dd3dcad5c3952dfed808be7bf1ef590c0678aa94b46c1c14" 2024-11-01T16:48:05.6129399Z }, 2024-11-01T16:48:05.6129843Z { 2024-11-01T16:48:05.6130624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6131630Z "size": 566, 2024-11-01T16:48:05.6132606Z "digest": "sha256:b4868ebe0605763563ea0bfae50d1c902ab28491ff1d79d6b52f5f3ae35cf910" 2024-11-01T16:48:05.6133734Z }, 2024-11-01T16:48:05.6134176Z { 2024-11-01T16:48:05.6134941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6135958Z "size": 43194130, 2024-11-01T16:48:05.6137135Z "digest": "sha256:9fc532b0af2f61d2359cd94f813d79a2adcdee9abf19791554de15d0af60f989" 2024-11-01T16:48:05.6138267Z }, 2024-11-01T16:48:05.6138707Z { 2024-11-01T16:48:05.6139478Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6140470Z "size": 106, 2024-11-01T16:48:05.6141475Z "digest": "sha256:90c3c714af3689cefd8bcc0b6d59b5c573a87b2c9795e2084ef418c8b3404c38" 2024-11-01T16:48:05.6142627Z }, 2024-11-01T16:48:05.6143067Z { 2024-11-01T16:48:05.6143843Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6144847Z "size": 294, 2024-11-01T16:48:05.6145809Z "digest": "sha256:6417650b4dec807ec6d46758c24117f43e85f3a9e1028cc354df4824245109c5" 2024-11-01T16:48:05.6146914Z }, 2024-11-01T16:48:05.6147354Z { 2024-11-01T16:48:05.6148126Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6149121Z "size": 88286, 2024-11-01T16:48:05.6150162Z "digest": "sha256:6fc4dbdf12d4bce3c6e44af38173e6e22944b6eb8dd8e6fc21e752640db877ec" 2024-11-01T16:48:05.6151333Z }, 2024-11-01T16:48:05.6151771Z { 2024-11-01T16:48:05.6152542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6153537Z "size": 106, 2024-11-01T16:48:05.6154869Z "digest": "sha256:15bc9b64c61392bff9a10be3fe8d69c90559b10d607eaf23c747c9da5772b8b0" 2024-11-01T16:48:05.6156046Z }, 2024-11-01T16:48:05.6156495Z { 2024-11-01T16:48:05.6157271Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6158279Z "size": 1402, 2024-11-01T16:48:05.6159300Z "digest": "sha256:9bbd022454bb3dc9fb6690e011a9aa0784f8ea91ff4eda4c642ff112568b2b03" 2024-11-01T16:48:05.6160443Z }, 2024-11-01T16:48:05.6160878Z { 2024-11-01T16:48:05.6161651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6162649Z "size": 702, 2024-11-01T16:48:05.6163651Z "digest": "sha256:f45ff9cb5ae341e9958b6a54478fa78323dcab64bcecf414e207f196a31530f2" 2024-11-01T16:48:05.6165144Z }, 2024-11-01T16:48:05.6165590Z { 2024-11-01T16:48:05.6166375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6167384Z "size": 136, 2024-11-01T16:48:05.6168376Z "digest": "sha256:614bea1c0a739f0f037eb961dd827afa53e913e8de6588bb66907080f5ffa175" 2024-11-01T16:48:05.6169524Z }, 2024-11-01T16:48:05.6169968Z { 2024-11-01T16:48:05.6170742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6171746Z "size": 120, 2024-11-01T16:48:05.6172735Z "digest": "sha256:9dce8d080268c9c364e311cfacd73d963991e3dc2684d910c3b5e2588fe30676" 2024-11-01T16:48:05.6173863Z }, 2024-11-01T16:48:05.6174312Z { 2024-11-01T16:48:05.6175092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6176118Z "size": 4434541507, 2024-11-01T16:48:05.6177498Z "digest": "sha256:7f3712a06f75e7c6131cebf428acf3d45011d22d83864168c9eb376a8cc792b4" 2024-11-01T16:48:05.6178645Z }, 2024-11-01T16:48:05.6179095Z { 2024-11-01T16:48:05.6179863Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6180864Z "size": 176, 2024-11-01T16:48:05.6181866Z "digest": "sha256:5ca64a90cefdf08096b2c7732e5e85b972d5e501cd85258dea602f90e9183435" 2024-11-01T16:48:05.6182998Z }, 2024-11-01T16:48:05.6183452Z { 2024-11-01T16:48:05.6184229Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6185223Z "size": 209, 2024-11-01T16:48:05.6186242Z "digest": "sha256:ee728b2ce5abe76b5f8b1f1128107ea80d50f82ea0c464de1aeea4ae9f66c575" 2024-11-01T16:48:05.6187382Z }, 2024-11-01T16:48:05.6187754Z { 2024-11-01T16:48:05.6188402Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6189242Z "size": 702, 2024-11-01T16:48:05.6190079Z "digest": "sha256:f45ff9cb5ae341e9958b6a54478fa78323dcab64bcecf414e207f196a31530f2" 2024-11-01T16:48:05.6191043Z }, 2024-11-01T16:48:05.6191426Z { 2024-11-01T16:48:05.6192076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6192920Z "size": 633, 2024-11-01T16:48:05.6193731Z "digest": "sha256:725423b7ca2ca9294a492201aef72a0f9c0604ccf17d0d03a2133a0488b82f4a" 2024-11-01T16:48:05.6194677Z }, 2024-11-01T16:48:05.6195045Z { 2024-11-01T16:48:05.6195695Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6196553Z "size": 652359953, 2024-11-01T16:48:05.6197415Z "digest": "sha256:e5f2017e4c0250718d09dd7cc301e8304ec9bba476ae08ef5822ea315f37c0c7" 2024-11-01T16:48:05.6198364Z }, 2024-11-01T16:48:05.6198731Z { 2024-11-01T16:48:05.6199371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6200201Z "size": 2047, 2024-11-01T16:48:05.6201028Z "digest": "sha256:34af416b9973175fefc767b712903fdc05ba8d9e689811a83e5dfd226d27be4d" 2024-11-01T16:48:05.6201965Z }, 2024-11-01T16:48:05.6202332Z { 2024-11-01T16:48:05.6202982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6203822Z "size": 15983068, 2024-11-01T16:48:05.6204675Z "digest": "sha256:9a372539a5bdeb26050d5f5b507262e52cb86c8bc4965568f3dfbe6e7f1f5d15" 2024-11-01T16:48:05.6205594Z }, 2024-11-01T16:48:05.6205958Z { 2024-11-01T16:48:05.6206909Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-11-01T16:48:05.6207759Z "size": 54145664, 2024-11-01T16:48:05.6208621Z "digest": "sha256:f47e4fa885c56af74fac500b96a440d217a07080565c57cda42f0ae5b2891f4f" 2024-11-01T16:48:05.6209556Z } 2024-11-01T16:48:05.6209916Z ] 2024-11-01T16:48:05.6210283Z } 2024-11-01T16:48:05.6210675Z + exit 0 2024-11-01T16:48:05.6476429Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-11-01T16:48:05.6476974Z with: 2024-11-01T16:48:05.6477721Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:05.6478914Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:05.6479362Z env: 2024-11-01T16:48:05.6479608Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:05.6480000Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:05.6480681Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:05.6481324Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:05.6481668Z AWS_REGION: us-east-1 2024-11-01T16:48:05.6482209Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:05.6482681Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:05.6488752Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:05.6489120Z ##[endgroup] 2024-11-01T16:48:05.6507508Z ##[group]Run set -x 2024-11-01T16:48:05.6507812Z set -x 2024-11-01T16:48:05.6508088Z set +e 2024-11-01T16:48:05.6508338Z  2024-11-01T16:48:05.6508586Z login() { 2024-11-01T16:48:05.6509180Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-11-01T16:48:05.6509815Z } 2024-11-01T16:48:05.6510059Z  2024-11-01T16:48:05.6510315Z retry () { 2024-11-01T16:48:05.6510667Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-11-01T16:48:05.6511046Z } 2024-11-01T16:48:05.6511292Z  2024-11-01T16:48:05.6511582Z retry login "${DOCKER_REGISTRY}" 2024-11-01T16:48:05.6511972Z  2024-11-01T16:48:05.6512216Z set -e 2024-11-01T16:48:05.6512643Z # ignore output since only exit code is used for conditional 2024-11-01T16:48:05.6513261Z # only pull docker image if it's not available locally 2024-11-01T16:48:05.6513930Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-11-01T16:48:05.6514523Z  retry docker pull "${DOCKER_IMAGE}" 2024-11-01T16:48:05.6514895Z fi 2024-11-01T16:48:05.6537721Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:05.6538178Z env: 2024-11-01T16:48:05.6538426Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:05.6538814Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:05.6539489Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:05.6540109Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:05.6540445Z AWS_REGION: us-east-1 2024-11-01T16:48:05.6540818Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:05.6541273Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:05.6547311Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:05.6548127Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:05.6549051Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:05.6549504Z ##[endgroup] 2024-11-01T16:48:05.6609390Z + set +e 2024-11-01T16:48:05.6610401Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:05.6611417Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:05.6619144Z + aws ecr get-login-password --region us-east-1 2024-11-01T16:48:05.6623435Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T16:48:06.8086091Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2024-11-01T16:48:06.8087679Z Configure a credential helper to remove this warning. See 2024-11-01T16:48:06.8089424Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2024-11-01T16:48:06.8090374Z 2024-11-01T16:48:06.8090792Z Login Succeeded 2024-11-01T16:48:06.8143048Z + set -e 2024-11-01T16:48:06.8145045Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:06.8599170Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:48:06.8600695Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-11-01T16:48:06.8602470Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-11-01T16:48:06.8603683Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:06.8648989Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:06.8649938Z env: 2024-11-01T16:48:06.8650465Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:06.8651291Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:06.8652779Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:06.8654117Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:06.8654820Z AWS_REGION: us-east-1 2024-11-01T16:48:06.8655638Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:06.8656759Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:06.8670069Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:06.8670830Z ##[endgroup] 2024-11-01T16:48:07.2001268Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:48:07.3719487Z Requirement already satisfied: psutil==5.9.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (5.9.1) 2024-11-01T16:48:07.3724756Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /var/home/pytorchci/.local/lib/python3.10/site-packages (11.525.84) 2024-11-01T16:48:07.5718398Z Prepare all required actions 2024-11-01T16:48:07.5719363Z Getting action download info 2024-11-01T16:48:07.7376241Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-11-01T16:48:08.3112854Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-11-01T16:48:08.7347077Z ##[group]Run ./.github/actions/download-build-artifacts 2024-11-01T16:48:08.7347957Z with: 2024-11-01T16:48:08.7348512Z name: linux-focal-rocm6.2-py3.10 2024-11-01T16:48:08.7349249Z s3-bucket: gha-artifacts 2024-11-01T16:48:08.7349859Z env: 2024-11-01T16:48:08.7350374Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:08.7351133Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:08.7352523Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:08.7353786Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:08.7354534Z AWS_REGION: us-east-1 2024-11-01T16:48:08.7355311Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:08.7356420Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:08.7369097Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:08.7369735Z ##[endgroup] 2024-11-01T16:48:08.7475291Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:48:08.7476066Z with: 2024-11-01T16:48:08.7476556Z name: linux-focal-rocm6.2-py3.10 2024-11-01T16:48:08.7477214Z s3-bucket: gha-artifacts 2024-11-01T16:48:08.7477785Z region: us-east-1 2024-11-01T16:48:08.7478256Z env: 2024-11-01T16:48:08.7478684Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:08.7479370Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:08.7480621Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:08.7481741Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:08.7482354Z AWS_REGION: us-east-1 2024-11-01T16:48:08.7483007Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:08.7483880Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:08.7494987Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:08.7495543Z ##[endgroup] 2024-11-01T16:48:09.2635666Z (node:2123042) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:48:09.2636981Z 2024-11-01T16:48:09.2637482Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:48:09.2638798Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:48:09.2640383Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:48:09.4182467Z Found 1 objects with prefix pytorch/pytorch/11632515612/linux-focal-rocm6.2-py3.10/ 2024-11-01T16:48:09.4185167Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:48:28.4235268Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-11-01T16:48:28.4245839Z Artifact download has finished successfully 2024-11-01T16:48:28.4806721Z ##[group]Run unzip -o artifacts.zip 2024-11-01T16:48:28.4807516Z unzip -o artifacts.zip 2024-11-01T16:48:28.4850804Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:28.4851709Z env: 2024-11-01T16:48:28.4852216Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:28.4853012Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:28.4854391Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:28.4855641Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:28.4856655Z AWS_REGION: us-east-1 2024-11-01T16:48:28.4857456Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:28.4858392Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:28.4872059Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:28.4872784Z ##[endgroup] 2024-11-01T16:48:28.4992471Z Archive: artifacts.zip 2024-11-01T16:48:28.4994953Z creating: dist/ 2024-11-01T16:48:34.0600744Z inflating: dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:48:34.0749458Z inflating: dist/.ninja_log 2024-11-01T16:48:34.0750340Z creating: build/custom_test_artifacts/ 2024-11-01T16:48:34.0751456Z creating: build/custom_test_artifacts/custom-op-build/ 2024-11-01T16:48:34.0752739Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-11-01T16:48:34.0754251Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:48:34.0756063Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:48:34.0757801Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-11-01T16:48:34.0759549Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:48:34.0761332Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:48:34.0763085Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:48:34.0765080Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:48:34.0767064Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:48:34.0768845Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:48:34.0770609Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:48:34.0772716Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:48:34.0774792Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:48:34.0777050Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:48:34.0779108Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:48:34.0781203Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:48:34.0783278Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:48:34.0785052Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:48:34.0786597Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:48:34.0788207Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-11-01T16:48:34.0789854Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-11-01T16:48:34.0792235Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-11-01T16:48:34.0794315Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-11-01T16:48:34.0796306Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-11-01T16:48:34.0798165Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-11-01T16:48:34.0800085Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-11-01T16:48:34.0802006Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-11-01T16:48:34.0803944Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-11-01T16:48:34.0805851Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-11-01T16:48:34.0807765Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-11-01T16:48:34.0809644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-11-01T16:48:34.0963244Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-11-01T16:48:34.0965115Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-11-01T16:48:34.0967092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-11-01T16:48:34.0969278Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-11-01T16:48:34.0971358Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-11-01T16:48:34.0973313Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-11-01T16:48:34.0975349Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-11-01T16:48:34.0977545Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-11-01T16:48:34.0979589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-11-01T16:48:34.0981641Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-11-01T16:48:34.0983644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-11-01T16:48:34.0992402Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-11-01T16:48:34.1084012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-11-01T16:48:34.1086177Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:48:34.1088115Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:48:34.1089873Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-11-01T16:48:34.1091511Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-11-01T16:48:34.1093131Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-11-01T16:48:34.1094703Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-11-01T16:48:34.1096172Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-11-01T16:48:34.1097733Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-11-01T16:48:34.1099187Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-11-01T16:48:34.1222463Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-11-01T16:48:34.1290137Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-11-01T16:48:34.1291486Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-11-01T16:48:34.1292740Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-11-01T16:48:34.1294209Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:48:34.1295940Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:48:34.1297739Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-11-01T16:48:34.1299428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:48:34.1301163Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:48:34.1302874Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:48:34.1304841Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:48:34.1306815Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:48:34.1308964Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:48:34.1310741Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:48:34.1312791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:48:34.1314853Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:48:34.1316861Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:48:34.1318865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:48:34.1320898Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:48:34.1322942Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:48:34.1324658Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:48:34.1326162Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:48:34.1327737Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-11-01T16:48:34.1329402Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-11-01T16:48:34.1331294Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-11-01T16:48:34.1333397Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-11-01T16:48:34.1335435Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-11-01T16:48:34.1337667Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-11-01T16:48:34.1339632Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-11-01T16:48:34.1341633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-11-01T16:48:34.1343633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-11-01T16:48:34.1345599Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-11-01T16:48:34.1347540Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-11-01T16:48:34.1349642Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-11-01T16:48:34.1412911Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-11-01T16:48:34.1415068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:48:34.1417339Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:48:34.1419110Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-11-01T16:48:34.1420741Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-11-01T16:48:34.1422320Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-11-01T16:48:34.1423863Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-11-01T16:48:34.1425313Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-11-01T16:48:34.1426684Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-11-01T16:48:34.1428125Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-11-01T16:48:34.1474336Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-11-01T16:48:34.1476078Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-11-01T16:48:34.1477471Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-11-01T16:48:34.1479049Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-11-01T16:48:34.1480888Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-11-01T16:48:34.1482624Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-11-01T16:48:34.1484381Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-11-01T16:48:34.1486341Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-11-01T16:48:34.1488206Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-11-01T16:48:34.1490292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-11-01T16:48:34.1492392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-11-01T16:48:34.1494274Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-11-01T16:48:34.1496165Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-11-01T16:48:34.1498501Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-11-01T16:48:34.1500684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-11-01T16:48:34.1502828Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-11-01T16:48:34.1504983Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-11-01T16:48:34.1507168Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-11-01T16:48:34.1509351Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-11-01T16:48:34.1511215Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-11-01T16:48:34.1512877Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-11-01T16:48:34.1514589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-11-01T16:48:34.1516386Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-11-01T16:48:34.1518417Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-11-01T16:48:34.1521126Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-11-01T16:48:34.1523311Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-11-01T16:48:34.1525359Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-11-01T16:48:34.1527462Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-11-01T16:48:34.1529601Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-11-01T16:48:34.1531731Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-11-01T16:48:34.1533843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-11-01T16:48:34.1535942Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-11-01T16:48:34.1538469Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-11-01T16:48:34.1643068Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-11-01T16:48:34.1645278Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-11-01T16:48:34.1647429Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-11-01T16:48:34.1649772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-11-01T16:48:34.1652037Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-11-01T16:48:34.1654191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-11-01T16:48:34.1656564Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-11-01T16:48:34.1658814Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-11-01T16:48:34.1661032Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-11-01T16:48:34.1663247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-11-01T16:48:34.1665429Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-11-01T16:48:34.1672237Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-11-01T16:48:34.1732935Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-11-01T16:48:34.1735306Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-11-01T16:48:34.1737491Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-11-01T16:48:34.1739357Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-11-01T16:48:34.1741098Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-11-01T16:48:34.1742821Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-11-01T16:48:34.1744470Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-11-01T16:48:34.1746025Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-11-01T16:48:34.1747508Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-11-01T16:48:34.1749502Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-11-01T16:48:34.1848919Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-11-01T16:48:34.1895092Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-11-01T16:48:34.1896256Z creating: build/lib/ 2024-11-01T16:48:34.1991384Z inflating: build/lib/libprotobuf-lite.a 2024-11-01T16:48:34.2483134Z inflating: build/lib/libprotobuf.a 2024-11-01T16:48:34.2493200Z inflating: build/lib/libpthreadpool.a 2024-11-01T16:48:34.3036669Z inflating: build/lib/libprotoc.a 2024-11-01T16:48:34.3045239Z inflating: build/lib/libcpuinfo.a 2024-11-01T16:48:34.3053675Z inflating: build/lib/libcpuinfo_internals.a 2024-11-01T16:48:34.3054522Z inflating: build/lib/libclog.a 2024-11-01T16:48:34.3075456Z inflating: build/lib/libpytorch_qnnpack.a 2024-11-01T16:48:34.3077778Z inflating: build/lib/libnnpack_reference_layers.a 2024-11-01T16:48:34.3097115Z inflating: build/lib/libnnpack.a 2024-11-01T16:48:34.3105291Z inflating: build/lib/libittnotify.a 2024-11-01T16:48:34.3134941Z inflating: build/lib/libtensorpipe_uv.a 2024-11-01T16:48:34.3205217Z inflating: build/lib/libgtest.a 2024-11-01T16:48:34.3206418Z inflating: build/lib/libgtest_main.a 2024-11-01T16:48:34.3286645Z inflating: build/lib/libbenchmark.a 2024-11-01T16:48:34.3287498Z inflating: build/lib/libbenchmark_main.a 2024-11-01T16:48:34.3306466Z inflating: build/lib/libgmock.a 2024-11-01T16:48:34.3374344Z inflating: build/lib/libasmjit.a 2024-11-01T16:48:34.3375140Z inflating: build/lib/libgmock_main.a 2024-11-01T16:48:34.4753305Z inflating: build/lib/libfbgemm.a 2024-11-01T16:48:34.4959913Z inflating: build/lib/libXNNPACK.a 2024-11-01T16:48:34.5584650Z inflating: build/lib/libtensorpipe.a 2024-11-01T16:48:34.5706747Z inflating: build/lib/libgloo.a 2024-11-01T16:48:34.5752673Z inflating: build/lib/libonnx_proto.a 2024-11-01T16:48:34.6183412Z inflating: build/lib/libgloo_hip.a 2024-11-01T16:48:34.6977208Z inflating: build/lib/libonnx.a 2024-11-01T16:48:35.8242300Z inflating: build/lib/libdnnl.a 2024-11-01T16:48:35.8264249Z inflating: build/lib/libfmt.a 2024-11-01T16:48:35.8613278Z inflating: build/lib/libkineto.a 2024-11-01T16:48:35.8718677Z inflating: build/lib/libc10.so 2024-11-01T16:48:35.8720243Z inflating: build/lib/libtorch_global_deps.so 2024-11-01T16:48:35.8770786Z inflating: build/lib/libc10_hip.so 2024-11-01T16:48:35.8772181Z inflating: build/lib/libcaffe2_nvrtc.so 2024-11-01T16:48:38.5444840Z inflating: build/lib/libtorch_cpu.so 2024-11-01T16:48:38.5449540Z inflating: build/lib/libunbox_lib.a 2024-11-01T16:48:38.5453898Z inflating: build/lib/libshm.so 2024-11-01T16:48:40.8216122Z inflating: build/lib/libtorch_hip.so 2024-11-01T16:48:40.8217315Z inflating: build/lib/libtorch.so 2024-11-01T16:48:40.8238935Z inflating: build/lib/libjitbackend_test.so 2024-11-01T16:48:40.8264949Z inflating: build/lib/libbackend_with_compiler.so 2024-11-01T16:48:40.8340351Z inflating: build/lib/libtorchbind_test.so 2024-11-01T16:48:40.8366244Z inflating: build/lib/libaoti_custom_ops.so 2024-11-01T16:48:41.0515807Z inflating: build/lib/libtorch_python.so 2024-11-01T16:48:41.0551974Z inflating: build/lib/libnnapi_backend.so 2024-11-01T16:48:41.0552782Z creating: build/bin/ 2024-11-01T16:48:41.0553431Z creating: build/bin/CMakeFiles/ 2024-11-01T16:48:41.0554249Z inflating: build/bin/cmake_install.cmake 2024-11-01T16:48:41.0555133Z inflating: build/bin/CTestTestfile.cmake 2024-11-01T16:48:41.1028531Z inflating: build/bin/protoc-3.13.0.0 2024-11-01T16:48:41.1501926Z inflating: build/bin/protoc 2024-11-01T16:48:41.1556606Z inflating: build/bin/c10_DeviceGuard_test 2024-11-01T16:48:41.1612676Z inflating: build/bin/c10_Device_test 2024-11-01T16:48:41.1666512Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-11-01T16:48:41.1729491Z inflating: build/bin/c10_DispatchKeySet_test 2024-11-01T16:48:41.1786153Z inflating: build/bin/c10_Scalar_test 2024-11-01T16:48:41.1838803Z inflating: build/bin/c10_StreamGuard_test 2024-11-01T16:48:41.1896067Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-11-01T16:48:41.1950227Z inflating: build/bin/c10_SymInt_test 2024-11-01T16:48:41.2010399Z inflating: build/bin/c10_SizesAndStrides_test 2024-11-01T16:48:41.2069622Z inflating: build/bin/c10_InlineStreamGuard_test 2024-11-01T16:48:41.2145183Z inflating: build/bin/c10_cow_test 2024-11-01T16:48:41.2201742Z inflating: build/bin/c10_Bitset_test 2024-11-01T16:48:41.2253446Z inflating: build/bin/c10_ConstexprCrc_test 2024-11-01T16:48:41.2306486Z inflating: build/bin/c10_DeadlockDetection_test 2024-11-01T16:48:41.2360531Z inflating: build/bin/c10_Half_test 2024-11-01T16:48:41.2420611Z inflating: build/bin/c10_LeftRight_test 2024-11-01T16:48:41.2477525Z inflating: build/bin/c10_NetworkFlow_test 2024-11-01T16:48:41.2536005Z inflating: build/bin/c10_Metaprogramming_test 2024-11-01T16:48:41.2589454Z inflating: build/bin/c10_Synchronized_test 2024-11-01T16:48:41.2648411Z inflating: build/bin/c10_ThreadLocal_test 2024-11-01T16:48:41.2703505Z inflating: build/bin/c10_TypeList_test 2024-11-01T16:48:41.2758158Z inflating: build/bin/c10_TypeIndex_test 2024-11-01T16:48:41.2810119Z inflating: build/bin/c10_TypeTraits_test 2024-11-01T16:48:41.2865316Z inflating: build/bin/c10_accumulate_test 2024-11-01T16:48:41.2924431Z inflating: build/bin/c10_bfloat16_test 2024-11-01T16:48:41.2978040Z inflating: build/bin/c10_bit_cast_test 2024-11-01T16:48:41.3039472Z inflating: build/bin/c10_complex_math_test 2024-11-01T16:48:41.3098838Z inflating: build/bin/c10_complex_test 2024-11-01T16:48:41.3154608Z inflating: build/bin/c10_exception_test 2024-11-01T16:48:41.3208268Z inflating: build/bin/c10_flags_test 2024-11-01T16:48:41.3261715Z inflating: build/bin/c10_generic_math_test 2024-11-01T16:48:41.3315864Z inflating: build/bin/c10_irange_test 2024-11-01T16:48:41.3494040Z inflating: build/bin/c10_intrusive_ptr_test 2024-11-01T16:48:41.3550640Z inflating: build/bin/c10_lazy_test 2024-11-01T16:48:41.3611884Z inflating: build/bin/c10_logging_test 2024-11-01T16:48:41.3691990Z inflating: build/bin/c10_optional_test 2024-11-01T16:48:41.3758291Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-11-01T16:48:41.3815638Z inflating: build/bin/c10_registry_test 2024-11-01T16:48:41.3870401Z inflating: build/bin/c10_ssize_test 2024-11-01T16:48:41.3925918Z inflating: build/bin/c10_string_util_test 2024-11-01T16:48:41.4089469Z inflating: build/bin/c10_small_vector_test 2024-11-01T16:48:41.4152802Z inflating: build/bin/c10_string_view_test 2024-11-01T16:48:41.4206693Z inflating: build/bin/c10_tempfile_test 2024-11-01T16:48:41.4257990Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-11-01T16:48:41.4317808Z inflating: build/bin/c10_typeid_test 2024-11-01T16:48:41.4369767Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-11-01T16:48:41.4422943Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-11-01T16:48:41.4474894Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-11-01T16:48:41.4527133Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-11-01T16:48:41.4579198Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-11-01T16:48:41.4631328Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-11-01T16:48:41.4683042Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-11-01T16:48:41.4735212Z inflating: build/bin/c10_hip_HIPTest 2024-11-01T16:48:41.5146397Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-11-01T16:48:41.5568951Z inflating: build/bin/vec_test_all_types_AVX512 2024-11-01T16:48:41.6006463Z inflating: build/bin/vec_test_all_types_AVX2 2024-11-01T16:48:41.6063366Z inflating: build/bin/test_edge_op_registration 2024-11-01T16:48:41.6119644Z inflating: build/bin/BackoffTest 2024-11-01T16:48:41.6176154Z inflating: build/bin/FileStoreTest 2024-11-01T16:48:41.6235475Z inflating: build/bin/TCPStoreTest 2024-11-01T16:48:41.6293538Z inflating: build/bin/HashStoreTest 2024-11-01T16:48:41.6362236Z inflating: build/bin/ProcessGroupGlooTest 2024-11-01T16:48:41.6365057Z inflating: build/bin/example_allreduce 2024-11-01T16:48:41.6369595Z inflating: build/bin/torch_shm_manager 2024-11-01T16:48:41.6432797Z inflating: build/bin/static_runtime_bench 2024-11-01T16:48:41.6703031Z inflating: build/bin/static_runtime_test 2024-11-01T16:48:41.6781607Z inflating: build/bin/Dict_test 2024-11-01T16:48:41.6837054Z inflating: build/bin/Dimname_test 2024-11-01T16:48:41.6906140Z inflating: build/bin/MaybeOwned_test 2024-11-01T16:48:41.6958034Z inflating: build/bin/hip_optional_test 2024-11-01T16:48:41.7020035Z inflating: build/bin/NamedTensor_test 2024-11-01T16:48:41.7083625Z inflating: build/bin/apply_utils_test 2024-11-01T16:48:41.7146322Z inflating: build/bin/atest 2024-11-01T16:48:41.7214147Z inflating: build/bin/basic 2024-11-01T16:48:41.7272289Z inflating: build/bin/broadcast_test 2024-11-01T16:48:41.7326881Z inflating: build/bin/cpu_allocator_test 2024-11-01T16:48:41.7388736Z inflating: build/bin/cpu_generator_test 2024-11-01T16:48:41.7446068Z inflating: build/bin/cpu_profiling_allocator_test 2024-11-01T16:48:41.7545507Z inflating: build/bin/cpu_rng_test 2024-11-01T16:48:41.7598553Z inflating: build/bin/dispatch_key_set_test 2024-11-01T16:48:41.7652434Z inflating: build/bin/dlconvertor_test 2024-11-01T16:48:41.7714387Z inflating: build/bin/extension_backend_test 2024-11-01T16:48:41.7773425Z inflating: build/bin/half_test 2024-11-01T16:48:41.7874970Z inflating: build/bin/ivalue_test 2024-11-01T16:48:41.7928043Z inflating: build/bin/lazy_tensor_test 2024-11-01T16:48:41.7986407Z inflating: build/bin/math_kernel_test 2024-11-01T16:48:41.8043564Z inflating: build/bin/memory_format_test 2024-11-01T16:48:41.8100291Z inflating: build/bin/memory_overlapping_test 2024-11-01T16:48:41.8156741Z inflating: build/bin/mobile_memory_cleanup 2024-11-01T16:48:41.8210584Z inflating: build/bin/operator_name_test 2024-11-01T16:48:41.8271079Z inflating: build/bin/native_test 2024-11-01T16:48:41.8326120Z inflating: build/bin/operators_test 2024-11-01T16:48:41.8381613Z inflating: build/bin/packedtensoraccessor_test 2024-11-01T16:48:41.8454193Z inflating: build/bin/pow_test 2024-11-01T16:48:41.8514789Z inflating: build/bin/quantized_test 2024-11-01T16:48:41.8568394Z inflating: build/bin/reduce_ops_test 2024-11-01T16:48:41.8622889Z inflating: build/bin/reportMemoryUsage_test 2024-11-01T16:48:41.8683413Z inflating: build/bin/scalar_tensor_test 2024-11-01T16:48:41.8738453Z inflating: build/bin/StorageUtils_test 2024-11-01T16:48:41.8800699Z inflating: build/bin/scalar_test 2024-11-01T16:48:41.8856729Z inflating: build/bin/stride_properties_test 2024-11-01T16:48:41.8942074Z inflating: build/bin/tensor_iterator_test 2024-11-01T16:48:41.9002266Z inflating: build/bin/type_ptr_test 2024-11-01T16:48:41.9062208Z inflating: build/bin/test_parallel 2024-11-01T16:48:41.9064995Z inflating: build/bin/thread_init_test 2024-11-01T16:48:41.9129739Z inflating: build/bin/type_test 2024-11-01T16:48:41.9186851Z inflating: build/bin/undefined_tensor_test 2024-11-01T16:48:41.9188357Z inflating: build/bin/verify_api_visibility 2024-11-01T16:48:41.9263792Z inflating: build/bin/legacy_vmap_test 2024-11-01T16:48:41.9319687Z inflating: build/bin/weakref_test 2024-11-01T16:48:41.9375504Z inflating: build/bin/wrapdim_test 2024-11-01T16:48:41.9431841Z inflating: build/bin/xla_tensor_test 2024-11-01T16:48:41.9496669Z inflating: build/bin/IListRef_test 2024-11-01T16:48:41.9613328Z inflating: build/bin/List_test 2024-11-01T16:48:41.9745650Z inflating: build/bin/kernel_function_legacy_test 2024-11-01T16:48:41.9849734Z inflating: build/bin/kernel_function_test 2024-11-01T16:48:41.9920764Z inflating: build/bin/KernelFunction_test 2024-11-01T16:48:42.0058862Z inflating: build/bin/kernel_lambda_legacy_test 2024-11-01T16:48:42.0170401Z inflating: build/bin/kernel_lambda_test 2024-11-01T16:48:42.0235682Z inflating: build/bin/kernel_stackbased_test 2024-11-01T16:48:42.0339081Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-11-01T16:48:42.0391127Z inflating: build/bin/op_allowlist_test 2024-11-01T16:48:42.0445718Z inflating: build/bin/CppSignature_test 2024-11-01T16:48:42.0513661Z inflating: build/bin/inline_container_test 2024-11-01T16:48:42.0841861Z inflating: build/bin/op_registration_test 2024-11-01T16:48:42.0901249Z inflating: build/bin/backend_fallback_test 2024-11-01T16:48:42.0953419Z inflating: build/bin/hip_complex_math_test 2024-11-01T16:48:42.1009757Z inflating: build/bin/hip_apply_test 2024-11-01T16:48:42.1062338Z inflating: build/bin/hip_complex_test 2024-11-01T16:48:42.1114547Z inflating: build/bin/hip_distributions_test 2024-11-01T16:48:42.1167167Z inflating: build/bin/hip_generator_test 2024-11-01T16:48:42.1221489Z inflating: build/bin/hip_dlconvertor_test 2024-11-01T16:48:42.1273578Z inflating: build/bin/hip_half_test 2024-11-01T16:48:42.1325720Z inflating: build/bin/hip_integer_divider_test 2024-11-01T16:48:42.1378771Z inflating: build/bin/hip_packedtensoraccessor_test 2024-11-01T16:48:42.1430544Z inflating: build/bin/hip_vectorized_test 2024-11-01T16:48:42.2046068Z inflating: build/bin/test_jit 2024-11-01T16:48:42.2061324Z inflating: build/bin/tutorial_tensorexpr 2024-11-01T16:48:42.2910837Z inflating: build/bin/test_tensorexpr 2024-11-01T16:48:42.2969196Z inflating: build/bin/test_dist_autograd 2024-11-01T16:48:42.3042815Z inflating: build/bin/test_cpp_rpc 2024-11-01T16:48:42.3045342Z inflating: build/bin/parallel_benchmark 2024-11-01T16:48:42.4302430Z inflating: build/bin/test_api 2024-11-01T16:48:42.4373675Z inflating: build/bin/test_mobile_nnc 2024-11-01T16:48:42.4383451Z inflating: build/bin/aot_model_compiler_test 2024-11-01T16:48:42.4757114Z inflating: build/bin/test_lazy 2024-11-01T16:48:42.4757919Z creating: .additional_ci_files/ 2024-11-01T16:48:42.4841834Z inflating: .additional_ci_files/test-times.json 2024-11-01T16:48:42.5173557Z inflating: .additional_ci_files/test-class-times.json 2024-11-01T16:48:42.5238171Z ##[group]Run rm artifacts.zip 2024-11-01T16:48:42.5238942Z rm artifacts.zip 2024-11-01T16:48:42.5281483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:42.5282373Z env: 2024-11-01T16:48:42.5282868Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:42.5283679Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:42.5285060Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:42.5286312Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:42.5286979Z AWS_REGION: us-east-1 2024-11-01T16:48:42.5287777Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:42.5288712Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:42.5304809Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:42.5305516Z ##[endgroup] 2024-11-01T16:48:42.8431667Z ##[group]Run df -H 2024-11-01T16:48:42.8432240Z df -H 2024-11-01T16:48:42.8475394Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:42.8476317Z env: 2024-11-01T16:48:42.8476822Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:42.8477614Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:42.8478991Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:42.8480695Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:42.8481382Z AWS_REGION: us-east-1 2024-11-01T16:48:42.8482159Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:42.8483081Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:42.8495644Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:42.8496591Z ##[endgroup] 2024-11-01T16:48:42.8632097Z Filesystem Size Used Avail Use% Mounted on 2024-11-01T16:48:42.8633459Z tmpfs 109G 33M 109G 1% /run 2024-11-01T16:48:42.8639882Z /dev/nvme0n1p2 944G 65G 831G 8% / 2024-11-01T16:48:42.8641263Z tmpfs 542G 33k 542G 1% /dev/shm 2024-11-01T16:48:42.8642353Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-11-01T16:48:42.8643477Z /dev/nvme0n1p1 536M 6.4M 530M 2% /boot/efi 2024-11-01T16:48:42.8644679Z /dev/nvme1n1p1 3.8T 915G 2.7T 26% /media/4TB 2024-11-01T16:48:42.8646307Z tmpfs 109G 33k 109G 1% /run/user/1307800118 2024-11-01T16:48:42.8647662Z 172.18.148.8:/export/amd2 5.5T 290G 5.3T 6% /mnt 2024-11-01T16:48:42.8650045Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch 108G 465M 107G 1% /home/amd-pytorch 2024-11-01T16:48:42.8651720Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage 55T 55T 846G 99% /groups 2024-11-01T16:48:42.8653268Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch 1.2T 336G 846G 29% /scratch 2024-11-01T16:48:42.8654841Z pure1.jax.cs.cpe.ice.amd.com:/homes/okakarpa 108G 52M 108G 1% /home/okakarpa 2024-11-01T16:48:42.8729014Z Prepare all required actions 2024-11-01T16:48:42.8729856Z Getting action download info 2024-11-01T16:48:43.0635217Z ##[group]Run ./.github/actions/download-td-artifacts 2024-11-01T16:48:43.0635648Z with: 2024-11-01T16:48:43.0635916Z env: 2024-11-01T16:48:43.0636172Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:43.0636571Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:43.0637278Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:43.0637916Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:43.0638265Z AWS_REGION: us-east-1 2024-11-01T16:48:43.0638649Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:43.0639139Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:43.0645240Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:43.0645555Z ##[endgroup] 2024-11-01T16:48:43.0692060Z ##[group]Run seemethere/download-artifact-s3@v4 2024-11-01T16:48:43.0692782Z with: 2024-11-01T16:48:43.0693216Z name: td_results 2024-11-01T16:48:43.0693801Z s3-bucket: gha-artifacts 2024-11-01T16:48:43.0694367Z region: us-east-1 2024-11-01T16:48:43.0694845Z env: 2024-11-01T16:48:43.0695273Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:43.0695960Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:43.0697375Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:43.0698498Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:43.0699099Z AWS_REGION: us-east-1 2024-11-01T16:48:43.0699739Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:43.0700572Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:43.0711651Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:43.0712194Z ##[endgroup] 2024-11-01T16:48:43.6100340Z (node:2123104) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-11-01T16:48:43.6101642Z 2024-11-01T16:48:43.6102123Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-11-01T16:48:43.6103478Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-11-01T16:48:43.6105026Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-11-01T16:48:43.7957937Z Found 1 objects with prefix pytorch/pytorch/11632515612/td_results/ 2024-11-01T16:48:43.7960556Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:48:43.9969987Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-11-01T16:48:43.9979786Z Artifact download has finished successfully 2024-11-01T16:48:44.0460315Z ##[group]Run mkdir -p .additional_ci_files 2024-11-01T16:48:44.0461204Z mkdir -p .additional_ci_files 2024-11-01T16:48:44.0462247Z mv td_results.json .additional_ci_files/td_results.json 2024-11-01T16:48:44.0500934Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:44.0501869Z env: 2024-11-01T16:48:44.0502376Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:44.0503267Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:44.0504925Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:44.0506424Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:44.0507188Z AWS_REGION: us-east-1 2024-11-01T16:48:44.0507732Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:44.0508312Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:44.0517926Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:44.0518549Z ##[endgroup] 2024-11-01T16:48:44.0733491Z ##[group]Run .github/scripts/parse_ref.py 2024-11-01T16:48:44.0734395Z .github/scripts/parse_ref.py 2024-11-01T16:48:44.0769397Z shell: /usr/bin/bash -e {0} 2024-11-01T16:48:44.0770045Z env: 2024-11-01T16:48:44.0770536Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:44.0771320Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:44.0772703Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:44.0773964Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:44.0774641Z AWS_REGION: us-east-1 2024-11-01T16:48:44.0775361Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:44.0776658Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:44.0789051Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:44.0789443Z ##[endgroup] 2024-11-01T16:48:44.1354725Z Prepare all required actions 2024-11-01T16:48:44.1429523Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-11-01T16:48:44.1430364Z with: 2024-11-01T16:48:44.1431291Z github-token: *** 2024-11-01T16:48:44.1431852Z env: 2024-11-01T16:48:44.1432347Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:44.1433124Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:44.1434517Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:44.1435785Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:44.1436455Z AWS_REGION: us-east-1 2024-11-01T16:48:44.1437143Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:44.1438048Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:44.1450574Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:44.1451185Z ##[endgroup] 2024-11-01T16:48:44.1486066Z ##[group]Run set -eux 2024-11-01T16:48:44.1486691Z set -eux 2024-11-01T16:48:44.1487821Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-11-01T16:48:44.1525667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:44.1526491Z env: 2024-11-01T16:48:44.1526935Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:44.1527627Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:44.1528861Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:44.1530007Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:44.1530610Z AWS_REGION: us-east-1 2024-11-01T16:48:44.1531266Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:44.1532086Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:44.1543277Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:44.1544071Z GITHUB_TOKEN: *** 2024-11-01T16:48:44.1544581Z ##[endgroup] 2024-11-01T16:48:44.1622553Z + python3 .github/scripts/get_workflow_job_id.py 11632515612 gpu948c.jax.cs.cpe.ice.amd.com 2024-11-01T16:48:45.1865977Z setting job-id=32397140858 2024-11-01T16:48:45.1867379Z setting job-name=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:48:45.2328973Z Prepare all required actions 2024-11-01T16:48:45.2329750Z Getting action download info 2024-11-01T16:48:45.4160854Z ##[group]Run ./.github/actions/filter-test-configs 2024-11-01T16:48:45.4161580Z with: 2024-11-01T16:48:45.4162303Z github-token: *** 2024-11-01T16:48:45.4164262Z test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]} 2024-11-01T16:48:45.4166649Z job-name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:48:45.4167620Z env: 2024-11-01T16:48:45.4168048Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:45.4168718Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:45.4169948Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:45.4171095Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:45.4171686Z AWS_REGION: us-east-1 2024-11-01T16:48:45.4172280Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:45.4173065Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:45.4184525Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:45.4185070Z ##[endgroup] 2024-11-01T16:48:45.4258528Z ##[group]Run nick-fields/retry@v3.0.0 2024-11-01T16:48:45.4259161Z with: 2024-11-01T16:48:45.4259568Z shell: bash 2024-11-01T16:48:45.4260039Z timeout_minutes: 10 2024-11-01T16:48:45.4260548Z max_attempts: 5 2024-11-01T16:48:45.4261042Z retry_wait_seconds: 30 2024-11-01T16:48:45.4262811Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:48:45.4264643Z polling_interval_seconds: 1 2024-11-01T16:48:45.4265269Z warning_on_retry: true 2024-11-01T16:48:45.4265806Z continue_on_error: false 2024-11-01T16:48:45.4266327Z env: 2024-11-01T16:48:45.4266757Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:45.4267421Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:45.4268636Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:45.4269744Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:45.4270344Z AWS_REGION: us-east-1 2024-11-01T16:48:45.4270981Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:45.4271785Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:45.4282787Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:45.4283537Z GITHUB_TOKEN: *** 2024-11-01T16:48:45.4284031Z ##[endgroup] 2024-11-01T16:48:45.5403214Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-11-01T16:48:45.8592165Z Defaulting to user installation because normal site-packages is not writeable 2024-11-01T16:48:46.0327862Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-11-01T16:48:46.0332405Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-11-01T16:48:46.0433129Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-11-01T16:48:46.0437805Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-11-01T16:48:46.0447157Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-11-01T16:48:46.0458598Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-11-01T16:48:46.5402302Z Command completed after 1 attempt(s). 2024-11-01T16:48:46.5540732Z ##[group]Run set -x 2024-11-01T16:48:46.5541913Z set -x 2024-11-01T16:48:46.5542477Z  2024-11-01T16:48:46.5544015Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:48:46.5545335Z # in runner workspace 2024-11-01T16:48:46.5546351Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-11-01T16:48:46.5591466Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:46.5592440Z env: 2024-11-01T16:48:46.5592963Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:46.5593777Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:46.5595223Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:46.5596541Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:46.5597247Z AWS_REGION: us-east-1 2024-11-01T16:48:46.5598034Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:46.5599039Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:46.5612308Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:46.5612976Z ##[endgroup] 2024-11-01T16:48:46.5701811Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-11-01T16:48:46.6166002Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:48:46.6166968Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-11-01T16:48:46.6167799Z echo "Job name: ${JOB_NAME}" 2024-11-01T16:48:46.6168505Z  2024-11-01T16:48:46.6169481Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-11-01T16:48:46.6170708Z # in runner workspace 2024-11-01T16:48:46.6171759Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-11-01T16:48:46.6172940Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-11-01T16:48:46.6173757Z  --job-name "${JOB_NAME}" \ 2024-11-01T16:48:46.6176465Z  --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" \ 2024-11-01T16:48:46.6178950Z  --selected-test-configs "" \ 2024-11-01T16:48:46.6179757Z  --pr-number "${PR_NUMBER}" \ 2024-11-01T16:48:46.6180488Z  --tag "${TAG}" \ 2024-11-01T16:48:46.6181182Z  --event-name "${EVENT_NAME}" \ 2024-11-01T16:48:46.6181969Z  --schedule "${SCHEDULE}" \ 2024-11-01T16:48:46.6182719Z  --branch "${HEAD_BRANCH}" 2024-11-01T16:48:46.6225188Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:46.6226116Z env: 2024-11-01T16:48:46.6226621Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:46.6227414Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:46.6228783Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:46.6230061Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:46.6230745Z AWS_REGION: us-east-1 2024-11-01T16:48:46.6231500Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:46.6232457Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:46.6245103Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:46.6246015Z GITHUB_TOKEN: *** 2024-11-01T16:48:46.6247022Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:48:46.6248170Z PR_NUMBER: 2024-11-01T16:48:46.6248737Z TAG: ciflow/trunk/138766 2024-11-01T16:48:46.6249367Z EVENT_NAME: push 2024-11-01T16:48:46.6249926Z SCHEDULE: 2024-11-01T16:48:46.6250442Z HEAD_BRANCH: 2024-11-01T16:48:46.6250979Z ##[endgroup] 2024-11-01T16:48:46.6333248Z Workflow: trunk 2024-11-01T16:48:46.6334749Z Job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:48:46.9297412Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-11-01T16:48:47.2396743Z ##[group]Run echo "Filtered matrix:" 2024-11-01T16:48:47.2398024Z echo "Filtered matrix:" 2024-11-01T16:48:47.2400784Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu"}]}" 2024-11-01T16:48:47.2403077Z  2024-11-01T16:48:47.2403577Z echo 2024-11-01T16:48:47.2404247Z echo "Is the current job unstable? False" 2024-11-01T16:48:47.2405043Z  2024-11-01T16:48:47.2405517Z echo 2024-11-01T16:48:47.2406150Z echo "Is keep-going label set? False" 2024-11-01T16:48:47.2406915Z  2024-11-01T16:48:47.2407377Z echo 2024-11-01T16:48:47.2407920Z echo "Renabled issues? " 2024-11-01T16:48:47.2448631Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:47.2449541Z env: 2024-11-01T16:48:47.2450034Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:47.2450832Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:47.2452225Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:47.2453485Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:47.2454163Z AWS_REGION: us-east-1 2024-11-01T16:48:47.2454968Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:47.2455890Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:47.2468802Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:47.2469433Z ##[endgroup] 2024-11-01T16:48:47.2544698Z Filtered matrix: 2024-11-01T16:48:47.2547057Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu}]} 2024-11-01T16:48:47.2549123Z 2024-11-01T16:48:47.2549417Z Is the current job unstable? False 2024-11-01T16:48:47.2549962Z 2024-11-01T16:48:47.2550562Z Is keep-going label set? False 2024-11-01T16:48:47.2551032Z 2024-11-01T16:48:47.2551264Z Renabled issues? 2024-11-01T16:48:47.2629311Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:47.2630118Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-11-01T16:48:47.2666553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T16:48:47.2667644Z env: 2024-11-01T16:48:47.2668238Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:47.2669040Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:47.2670405Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:47.2671684Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:47.2672369Z AWS_REGION: us-east-1 2024-11-01T16:48:47.2673103Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:47.2674095Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:47.2687016Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:47.2687642Z JOB_TIMEOUT: 300 2024-11-01T16:48:47.2688200Z ##[endgroup] 2024-11-01T16:48:47.2882938Z ##[group]Run set -x 2024-11-01T16:48:47.2883721Z set -x 2024-11-01T16:48:47.2884260Z  2024-11-01T16:48:47.2884917Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-11-01T16:48:47.2885909Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-11-01T16:48:47.2886933Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-11-01T16:48:47.2887866Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-11-01T16:48:47.2888623Z else 2024-11-01T16:48:47.2889237Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:48:47.2889984Z fi 2024-11-01T16:48:47.2890472Z  2024-11-01T16:48:47.2891312Z # detached container should get cleaned up by teardown_ec2_linux 2024-11-01T16:48:47.2892660Z # TODO: Stop building test binaries as part of the build phase 2024-11-01T16:48:47.2893851Z # Used for GPU_FLAG since that doesn't play nice 2024-11-01T16:48:47.2894897Z # shellcheck disable=SC2086,SC2090 2024-11-01T16:48:47.2896201Z container_name=$(docker run \ 2024-11-01T16:48:47.2897125Z  ${GPU_FLAG:-} \ 2024-11-01T16:48:47.2897801Z  -e BUILD_ENVIRONMENT \ 2024-11-01T16:48:47.2898511Z  -e PR_NUMBER \ 2024-11-01T16:48:47.2899161Z  -e GITHUB_ACTIONS \ 2024-11-01T16:48:47.2899960Z  -e GITHUB_REPOSITORY \ 2024-11-01T16:48:47.2900683Z  -e GITHUB_WORKFLOW \ 2024-11-01T16:48:47.2901368Z  -e GITHUB_JOB \ 2024-11-01T16:48:47.2902003Z  -e GITHUB_RUN_ID \ 2024-11-01T16:48:47.2902675Z  -e GITHUB_RUN_NUMBER \ 2024-11-01T16:48:47.2903388Z  -e GITHUB_RUN_ATTEMPT \ 2024-11-01T16:48:47.2904091Z  -e JOB_ID \ 2024-11-01T16:48:47.2904687Z  -e JOB_NAME \ 2024-11-01T16:48:47.2905294Z  -e BRANCH \ 2024-11-01T16:48:47.2905881Z  -e SHA1 \ 2024-11-01T16:48:47.2906487Z  -e AWS_DEFAULT_REGION \ 2024-11-01T16:48:47.2907213Z  -e IN_WHEEL_TEST \ 2024-11-01T16:48:47.2907891Z  -e SHARD_NUMBER \ 2024-11-01T16:48:47.2908543Z  -e TEST_CONFIG \ 2024-11-01T16:48:47.2909195Z  -e NUM_TEST_SHARDS \ 2024-11-01T16:48:47.2909887Z  -e REENABLED_ISSUES \ 2024-11-01T16:48:47.2910619Z  -e CONTINUE_THROUGH_ERROR \ 2024-11-01T16:48:47.2911369Z  -e VERBOSE_TEST_LOGS \ 2024-11-01T16:48:47.2912082Z  -e TEST_SHOWLOCALS \ 2024-11-01T16:48:47.2912756Z  -e NO_TEST_TIMEOUT \ 2024-11-01T16:48:47.2913428Z  -e NO_TD \ 2024-11-01T16:48:47.2914118Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-11-01T16:48:47.2914926Z  -e SCCACHE_BUCKET \ 2024-11-01T16:48:47.2915678Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-11-01T16:48:47.2916557Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-11-01T16:48:47.2917479Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-11-01T16:48:47.2918322Z  -e TESTS_TO_INCLUDE \ 2024-11-01T16:48:47.2919187Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-11-01T16:48:47.2920128Z  --ulimit stack=10485760:83886080 \ 2024-11-01T16:48:47.2920892Z  --ulimit core=0 \ 2024-11-01T16:48:47.2921634Z  --security-opt seccomp=unconfined \ 2024-11-01T16:48:47.2922457Z  --cap-add=SYS_PTRACE \ 2024-11-01T16:48:47.2923156Z  --shm-size="8g" \ 2024-11-01T16:48:47.2923781Z  --tty \ 2024-11-01T16:48:47.2924337Z  --detach \ 2024-11-01T16:48:47.2924971Z  --name="${container_name}" \ 2024-11-01T16:48:47.2925710Z  --user jenkins \ 2024-11-01T16:48:47.2926569Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-11-01T16:48:47.2927553Z  -w /var/lib/jenkins/workspace \ 2024-11-01T16:48:47.2928323Z  "${DOCKER_IMAGE}" 2024-11-01T16:48:47.2928939Z ) 2024-11-01T16:48:47.2929548Z # save container name for later step 2024-11-01T16:48:47.2931055Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-11-01T16:48:47.2932900Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-11-01T16:48:47.2935282Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-11-01T16:48:47.2976724Z shell: /usr/bin/bash -e {0} 2024-11-01T16:48:47.2977369Z env: 2024-11-01T16:48:47.2977858Z GIT_DEFAULT_BRANCH: main 2024-11-01T16:48:47.2978638Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T16:48:47.2980021Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T16:48:47.2981270Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T16:48:47.2981941Z AWS_REGION: us-east-1 2024-11-01T16:48:47.2982669Z AWS_ACCESS_KEY_ID: *** 2024-11-01T16:48:47.2983610Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T16:48:47.2996592Z AWS_SESSION_TOKEN: *** 2024-11-01T16:48:47.2997347Z BUILD_ENVIRONMENT: linux-focal-rocm6.2-py3.10 2024-11-01T16:48:47.2998156Z PR_NUMBER: 2024-11-01T16:48:47.2998752Z GITHUB_REPOSITORY: pytorch/pytorch 2024-11-01T16:48:47.2999501Z GITHUB_WORKFLOW: trunk 2024-11-01T16:48:47.3000100Z GITHUB_JOB: test 2024-11-01T16:48:47.3000657Z GITHUB_RUN_ID: 11632515612 2024-11-01T16:48:47.3001299Z GITHUB_RUN_NUMBER: 102766 2024-11-01T16:48:47.3001937Z GITHUB_RUN_ATTEMPT: 1 2024-11-01T16:48:47.3002519Z JOB_ID: 32397140858 2024-11-01T16:48:47.3003517Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:48:47.3004611Z BRANCH: 2024-11-01T16:48:47.3005244Z SHA1: d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:48:47.3006088Z CONTINUE_THROUGH_ERROR: False 2024-11-01T16:48:47.3006768Z VERBOSE_TEST_LOGS: False 2024-11-01T16:48:47.3007398Z TEST_SHOWLOCALS: False 2024-11-01T16:48:47.3007997Z NO_TEST_TIMEOUT: False 2024-11-01T16:48:47.3008593Z NO_TD: False 2024-11-01T16:48:47.3009138Z TEST_CONFIG: distributed 2024-11-01T16:48:47.3009740Z SHARD_NUMBER: 1 2024-11-01T16:48:47.3010306Z NUM_TEST_SHARDS: 1 2024-11-01T16:48:47.3010878Z REENABLED_ISSUES: 2024-11-01T16:48:47.3011592Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-11-01T16:48:47.3013441Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:47.3015438Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:48:47.3016637Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-11-01T16:48:47.3017412Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-11-01T16:48:47.3019440Z TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-11-01T16:48:47.3021428Z ##[endgroup] 2024-11-01T16:48:47.3104340Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-11-01T16:48:47.3105557Z + [[ linux-focal-rocm6.2-py3.10 == *onnx* ]] 2024-11-01T16:48:47.3106451Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-11-01T16:48:47.3126552Z +++ nproc --ignore=2 2024-11-01T16:48:47.3166146Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_11632515612 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:bdd298b12da59246147f016e0693ffd722419941 2024-11-01T16:48:47.7046477Z + container_name=7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T16:48:47.7048150Z + echo CONTAINER_NAME=7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T16:48:47.7051045Z + docker exec -t 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-11-01T16:49:02.8485302Z Processing ./dist/torch-2.6.0a0+gitd1aa4ef-cp310-cp310-linux_x86_64.whl 2024-11-01T16:49:03.5723987Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (3.16.1) 2024-11-01T16:49:03.5727734Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (4.12.2) 2024-11-01T16:49:03.5730637Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (2.8.8) 2024-11-01T16:49:03.5733330Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (3.1.4) 2024-11-01T16:49:03.5736032Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (2024.10.0) 2024-11-01T16:49:03.5739799Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+gitd1aa4ef) (1.13.1) 2024-11-01T16:49:03.5755984Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+gitd1aa4ef) (1.3.0) 2024-11-01T16:49:03.6125421Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+gitd1aa4ef) (3.0.2) 2024-11-01T16:49:03.9272641Z Installing collected packages: torch 2024-11-01T16:49:17.4836688Z Successfully installed torch-2.6.0a0+gitd1aa4ef 2024-11-01T16:49:17.5491642Z + export TERM=vt100 2024-11-01T16:49:17.5492217Z + TERM=vt100 2024-11-01T16:49:17.5500732Z ++ dirname .ci/pytorch/test.sh 2024-11-01T16:49:17.5528477Z + source .ci/pytorch/common.sh 2024-11-01T16:49:17.5538980Z +++ dirname .ci/pytorch/common.sh 2024-11-01T16:49:17.5568039Z ++ source .ci/pytorch/common_utils.sh 2024-11-01T16:49:17.5572287Z +++ declare -f -t trap_add 2024-11-01T16:49:17.5584227Z ++ set -ex 2024-11-01T16:49:17.5584950Z ++ [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-11-01T16:49:17.5585736Z ++ unset HIP_PLATFORM 2024-11-01T16:49:17.5586366Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-11-01T16:49:17.5590149Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-11-01T16:49:17.5590951Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-11-01T16:49:17.5591634Z ++ HSAKMT_DEBUG_LEVEL=4 2024-11-01T16:49:17.5592300Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-11-01T16:49:17.5593039Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-11-01T16:49:17.5593696Z ++ BUILD_TEST_LIBTORCH=0 2024-11-01T16:49:17.5594596Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-11-01T16:49:17.5598598Z + echo 'Environment variables:' 2024-11-01T16:49:17.5599581Z Environment variables: 2024-11-01T16:49:17.5600324Z + env 2024-11-01T16:49:17.5617518Z INSTALLED_DB=yes 2024-11-01T16:49:17.5618694Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T16:49:17.5619922Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-11-01T16:49:17.5620756Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:49:17.5621625Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-11-01T16:49:17.5622440Z HOSTNAME=7da8eb8cf0fc 2024-11-01T16:49:17.5624773Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.5626359Z GITHUB_ACTION=__self 2024-11-01T16:49:17.5627000Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:49:17.5627724Z GITHUB_RUN_NUMBER=102766 2024-11-01T16:49:17.5628352Z TEST_CONFIG=distributed 2024-11-01T16:49:17.5629002Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:49:17.5629937Z AWS_DEFAULT_REGION=us-east-1 2024-11-01T16:49:17.5630929Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-11-01T16:49:17.5631822Z GITHUB_REF_TYPE=tag 2024-11-01T16:49:17.5633247Z *** 2024-11-01T16:49:17.5633680Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:49:17.5634167Z GITHUB_ACTIONS=true 2024-11-01T16:49:17.5634591Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.5635164Z GITHUB_SHA=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.5636025Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/138766 2024-11-01T16:49:17.5636785Z VERBOSE_TEST_LOGS=False 2024-11-01T16:49:17.5637529Z GITHUB_REF=refs/tags/ciflow/trunk/138766 2024-11-01T16:49:17.5637993Z SHARD_NUMBER=1 2024-11-01T16:49:17.5638345Z GITHUB_REF_PROTECTED=false 2024-11-01T16:49:17.5638730Z HOME=/var/lib/jenkins 2024-11-01T16:49:17.5639144Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:49:17.5639640Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:49:17.5640098Z LANG=C.UTF-8 2024-11-01T16:49:17.5640428Z PYTORCH_TEST_WITH_ROCM=1 2024-11-01T16:49:17.5640800Z NUM_TEST_SHARDS=1 2024-11-01T16:49:17.5641798Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.5643064Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:49:17.5643749Z MAGMA_HOME=/opt/rocm/magma 2024-11-01T16:49:17.5644731Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.5645642Z HSAKMT_DEBUG_LEVEL=4 2024-11-01T16:49:17.5646418Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:49:17.5647177Z GITHUB_EVENT_NAME=push 2024-11-01T16:49:17.5647539Z GITHUB_RUN_ID=11632515612 2024-11-01T16:49:17.5648595Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.5649695Z GITHUB_ACTOR=pytorch-bot[bot] 2024-11-01T16:49:17.5650353Z PR_NUMBER= 2024-11-01T16:49:17.5650950Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:49:17.5651653Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:49:17.5652492Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:49:17.5653348Z TERM=vt100 2024-11-01T16:49:17.5653851Z INSTALLED_VISION=yes 2024-11-01T16:49:17.5654399Z BRANCH= 2024-11-01T16:49:17.5656217Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-11-01T16:49:17.5659311Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-11-01T16:49:17.5660732Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:49:17.5661496Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-11-01T16:49:17.5662164Z REENABLED_ISSUES= 2024-11-01T16:49:17.5662688Z SHLVL=1 2024-11-01T16:49:17.5663150Z MAX_JOBS=126 2024-11-01T16:49:17.5663713Z GITHUB_ACTOR_ID=54816060 2024-11-01T16:49:17.5664543Z GITHUB_WORKFLOW_SHA=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.5665493Z GITHUB_REF_NAME=ciflow/trunk/138766 2024-11-01T16:49:17.5666192Z ROCM_PATH=/opt/rocm 2024-11-01T16:49:17.5667205Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:49:17.5668226Z GITHUB_JOB=test 2024-11-01T16:49:17.5668764Z NO_TEST_TIMEOUT=False 2024-11-01T16:49:17.5669393Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:49:17.5670097Z LC_ALL=C.UTF-8 2024-11-01T16:49:17.5670648Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:49:17.5671629Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:49:17.5673950Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:49:17.5676288Z GITHUB_BASE_REF= 2024-11-01T16:49:17.5676814Z CI=true 2024-11-01T16:49:17.5677313Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-11-01T16:49:17.5677991Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:49:17.5678644Z JOB_ID=32397140858 2024-11-01T16:49:17.5679194Z INSTALLED_PROTOBUF=yes 2024-11-01T16:49:17.5679772Z GITHUB_HEAD_REF= 2024-11-01T16:49:17.5680305Z GITHUB_ACTION_REF= 2024-11-01T16:49:17.5681084Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:49:17.5681919Z TEST_SHOWLOCALS=False 2024-11-01T16:49:17.5682508Z GITHUB_WORKFLOW=trunk 2024-11-01T16:49:17.5683117Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:49:17.5684849Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.5723546Z NO_TD=False 2024-11-01T16:49:17.5724140Z OLDPWD=/var/lib/jenkins 2024-11-01T16:49:17.5724785Z _=/usr/bin/env 2024-11-01T16:49:17.5725789Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-11-01T16:49:17.6089965Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-11-01T16:49:17.6091718Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T16:49:17.6093272Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-11-01T16:49:17.6094833Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-11-01T16:49:17.6095903Z + BUILD_DIR=build 2024-11-01T16:49:17.6096649Z + BUILD_RENAMED_DIR=build_renamed 2024-11-01T16:49:17.6097351Z + BUILD_BIN_DIR=build/bin 2024-11-01T16:49:17.6097949Z + SHARD_NUMBER=1 2024-11-01T16:49:17.6098486Z + NUM_TEST_SHARDS=1 2024-11-01T16:49:17.6099045Z + export VALGRIND=ON 2024-11-01T16:49:17.6099642Z + VALGRIND=ON 2024-11-01T16:49:17.6100463Z + [[ linux-focal-rocm6.2-py3.10 == *clang9* ]] 2024-11-01T16:49:17.6101383Z + [[ 0 == \1 ]] 2024-11-01T16:49:17.6101996Z + [[ False == \1 ]] 2024-11-01T16:49:17.6102852Z + [[ linux-focal-rocm6.2-py3.10 != *bazel* ]] 2024-11-01T16:49:17.6105265Z ++ realpath build/custom_test_artifacts 2024-11-01T16:49:17.6134098Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-11-01T16:49:17.6136910Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]] 2024-11-01T16:49:17.6138924Z + echo 'Setting INCLUDE_CLAUSE' 2024-11-01T16:49:17.6139619Z Setting INCLUDE_CLAUSE 2024-11-01T16:49:17.6142226Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl' 2024-11-01T16:49:17.6144643Z + echo 'Environment variables' 2024-11-01T16:49:17.6145106Z Environment variables 2024-11-01T16:49:17.6145447Z + env 2024-11-01T16:49:17.6162355Z INSTALLED_DB=yes 2024-11-01T16:49:17.6163476Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T16:49:17.6164656Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-11-01T16:49:17.6165454Z CONTINUE_THROUGH_ERROR=False 2024-11-01T16:49:17.6166286Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-11-01T16:49:17.6167062Z HOSTNAME=7da8eb8cf0fc 2024-11-01T16:49:17.6168679Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.6170195Z GITHUB_ACTION=__self 2024-11-01T16:49:17.6170984Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-11-01T16:49:17.6171836Z GITHUB_RUN_NUMBER=102766 2024-11-01T16:49:17.6172568Z TEST_CONFIG=distributed 2024-11-01T16:49:17.6173408Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-11-01T16:49:17.6174704Z AWS_DEFAULT_REGION=us-east-1 2024-11-01T16:49:17.6175340Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-11-01T16:49:17.6175811Z GITHUB_REF_TYPE=tag 2024-11-01T16:49:17.6176231Z *** 2024-11-01T16:49:17.6176786Z GITHUB_REPOSITORY_ID=65600975 2024-11-01T16:49:17.6177211Z GITHUB_ACTIONS=true 2024-11-01T16:49:17.6177637Z SHA1=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.6178214Z GITHUB_SHA=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.6179062Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/138766 2024-11-01T16:49:17.6179819Z VERBOSE_TEST_LOGS=False 2024-11-01T16:49:17.6180228Z GITHUB_REF=refs/tags/ciflow/trunk/138766 2024-11-01T16:49:17.6180759Z SHARD_NUMBER=1 2024-11-01T16:49:17.6181403Z GITHUB_REF_PROTECTED=false 2024-11-01T16:49:17.6182173Z HOME=/var/lib/jenkins 2024-11-01T16:49:17.6182978Z GITHUB_API_URL=https://api.github.com 2024-11-01T16:49:17.6183813Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-11-01T16:49:17.6184573Z LANG=C.UTF-8 2024-11-01T16:49:17.6185534Z PYTORCH_TEST_WITH_ROCM=1 2024-11-01T16:49:17.6186149Z NUM_TEST_SHARDS=1 2024-11-01T16:49:17.6187847Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.6190095Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 1, linux.rocm.gpu) 2024-11-01T16:49:17.6191306Z MAGMA_HOME=/opt/rocm/magma 2024-11-01T16:49:17.6193001Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.6194579Z HSAKMT_DEBUG_LEVEL=4 2024-11-01T16:49:17.6195856Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-11-01T16:49:17.6197147Z GITHUB_EVENT_NAME=push 2024-11-01T16:49:17.6197748Z GITHUB_RUN_ID=11632515612 2024-11-01T16:49:17.6199560Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.6201370Z GITHUB_ACTOR=pytorch-bot[bot] 2024-11-01T16:49:17.6202041Z PR_NUMBER= 2024-11-01T16:49:17.6202553Z GITHUB_RUN_ATTEMPT=1 2024-11-01T16:49:17.6203117Z VALGRIND=ON 2024-11-01T16:49:17.6203664Z ANACONDA_PYTHON_VERSION=3.10 2024-11-01T16:49:17.6204486Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-11-01T16:49:17.6205325Z TERM=vt100 2024-11-01T16:49:17.6205826Z INSTALLED_VISION=yes 2024-11-01T16:49:17.6206381Z BRANCH= 2024-11-01T16:49:17.6208282Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl 2024-11-01T16:49:17.6211236Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-11-01T16:49:17.6212696Z GITHUB_SERVER_URL=https://github.com 2024-11-01T16:49:17.6213466Z PYTORCH_ROCM_ARCH=gfx906;gfx90a 2024-11-01T16:49:17.6214149Z REENABLED_ISSUES= 2024-11-01T16:49:17.6214676Z SHLVL=1 2024-11-01T16:49:17.6215155Z MAX_JOBS=126 2024-11-01T16:49:17.6215683Z GITHUB_ACTOR_ID=54816060 2024-11-01T16:49:17.6216667Z GITHUB_WORKFLOW_SHA=d1aa4ef5a6c0ee2d57fb0086f5557e34537f0fea 2024-11-01T16:49:17.6217654Z GITHUB_REF_NAME=ciflow/trunk/138766 2024-11-01T16:49:17.6218362Z ROCM_PATH=/opt/rocm 2024-11-01T16:49:17.6219401Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-11-01T16:49:17.6220457Z GITHUB_JOB=test 2024-11-01T16:49:17.6220995Z NO_TEST_TIMEOUT=False 2024-11-01T16:49:17.6221624Z GITHUB_REPOSITORY=pytorch/pytorch 2024-11-01T16:49:17.6222339Z LC_ALL=C.UTF-8 2024-11-01T16:49:17.6222885Z GITHUB_RETENTION_DAYS=90 2024-11-01T16:49:17.6223523Z GITHUB_ACTION_REPOSITORY= 2024-11-01T16:49:17.6225921Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:49:17.6228339Z GITHUB_BASE_REF= 2024-11-01T16:49:17.6228867Z CI=true 2024-11-01T16:49:17.6229718Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-11-01T16:49:17.6230413Z GITHUB_REPOSITORY_OWNER=pytorch 2024-11-01T16:49:17.6231081Z JOB_ID=32397140858 2024-11-01T16:49:17.6231638Z INSTALLED_PROTOBUF=yes 2024-11-01T16:49:17.6232229Z GITHUB_HEAD_REF= 2024-11-01T16:49:17.6232763Z GITHUB_ACTION_REF= 2024-11-01T16:49:17.6233555Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-11-01T16:49:17.6234412Z TEST_SHOWLOCALS=False 2024-11-01T16:49:17.6235002Z GITHUB_WORKFLOW=trunk 2024-11-01T16:49:17.6235645Z DEBIAN_FRONTEND=noninteractive 2024-11-01T16:49:17.6237435Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_8b17b619-4850-4d3a-b5b5-fac473e43c61 2024-11-01T16:49:17.6239021Z NO_TD=False 2024-11-01T16:49:17.6239534Z OLDPWD=/var/lib/jenkins 2024-11-01T16:49:17.6240121Z _=/usr/bin/env 2024-11-01T16:49:17.6240721Z + echo 'Testing pytorch' 2024-11-01T16:49:17.6241331Z Testing pytorch 2024-11-01T16:49:17.6241902Z + export LANG=C.UTF-8 2024-11-01T16:49:17.6242891Z + LANG=C.UTF-8 2024-11-01T16:49:17.6243407Z + PR_NUMBER= 2024-11-01T16:49:17.6243974Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-11-01T16:49:17.6244793Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-11-01T16:49:17.6245763Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-11-01T16:49:17.6246604Z + export HIP_VISIBLE_DEVICES=0,1 2024-11-01T16:49:17.6247308Z + HIP_VISIBLE_DEVICES=0,1 2024-11-01T16:49:17.6247946Z + [[ distributed == \s\l\o\w ]] 2024-11-01T16:49:17.6248887Z + [[ linux-focal-rocm6.2-py3.10 == *slow-gradcheck* ]] 2024-11-01T16:49:17.6249940Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-11-01T16:49:17.6250894Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-11-01T16:49:17.6251786Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:49:17.6252650Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-11-01T16:49:17.6253426Z + [[ distributed == *crossref* ]] 2024-11-01T16:49:17.6254275Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-11-01T16:49:17.6255049Z + export VALGRIND=OFF 2024-11-01T16:49:17.6255639Z + VALGRIND=OFF 2024-11-01T16:49:17.6256150Z + rocminfo 2024-11-01T16:49:17.6404115Z ROCk module version 6.8.5 is loaded 2024-11-01T16:49:17.7818306Z ===================== 2024-11-01T16:49:17.7819076Z HSA System Attributes 2024-11-01T16:49:17.7819738Z ===================== 2024-11-01T16:49:17.7820416Z Runtime Version: 1.14 2024-11-01T16:49:17.7821112Z Runtime Ext Version: 1.6 2024-11-01T16:49:17.7821831Z System Timestamp Freq.: 1000.000000MHz 2024-11-01T16:49:17.7823071Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-11-01T16:49:17.7824464Z Machine Model: LARGE 2024-11-01T16:49:17.7825575Z System Endianness: LITTLE 2024-11-01T16:49:17.7826517Z Mwaitx: DISABLED 2024-11-01T16:49:17.7827249Z DMAbuf Support: YES 2024-11-01T16:49:17.7827707Z 2024-11-01T16:49:17.7827922Z ========== 2024-11-01T16:49:17.7828578Z HSA Agents 2024-11-01T16:49:17.7829196Z ========== 2024-11-01T16:49:17.7829776Z ******* 2024-11-01T16:49:17.7830377Z Agent 1 2024-11-01T16:49:17.7830960Z ******* 2024-11-01T16:49:17.7831981Z Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.7833164Z Uuid: CPU-XX 2024-11-01T16:49:17.7834356Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.7835444Z Vendor Name: CPU 2024-11-01T16:49:17.7836797Z Feature: None specified 2024-11-01T16:49:17.7837876Z Profile: FULL_PROFILE 2024-11-01T16:49:17.7838932Z Float Round Mode: NEAR 2024-11-01T16:49:17.7840040Z Max Queue Number: 0(0x0) 2024-11-01T16:49:17.7841721Z Queue Min Size: 0(0x0) 2024-11-01T16:49:17.7842818Z Queue Max Size: 0(0x0) 2024-11-01T16:49:17.7843847Z Queue Type: MULTI 2024-11-01T16:49:17.7844823Z Node: 0 2024-11-01T16:49:17.7845807Z Device Type: CPU 2024-11-01T16:49:17.7846699Z Cache Info: 2024-11-01T16:49:17.7847506Z L1: 32768(0x8000) KB 2024-11-01T16:49:17.7848542Z Chip ID: 0(0x0) 2024-11-01T16:49:17.7849568Z ASIC Revision: 0(0x0) 2024-11-01T16:49:17.7850627Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.7851714Z Max Clock Freq. (MHz): 2000 2024-11-01T16:49:17.7852753Z BDFID: 0 2024-11-01T16:49:17.7854149Z Internal Node ID: 0 2024-11-01T16:49:17.7855184Z Compute Unit: 64 2024-11-01T16:49:17.7856189Z SIMDs per CU: 0 2024-11-01T16:49:17.7857496Z Shader Engines: 0 2024-11-01T16:49:17.7858563Z Shader Arrs. per Eng.: 0 2024-11-01T16:49:17.7859657Z WatchPts on Addr. Ranges:1 2024-11-01T16:49:17.7860608Z Memory Properties: 2024-11-01T16:49:17.7861306Z Features: None 2024-11-01T16:49:17.7862014Z Pool Info: 2024-11-01T16:49:17.7862680Z Pool 1 2024-11-01T16:49:17.7863588Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.7864886Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:49:17.7866180Z Allocatable: TRUE 2024-11-01T16:49:17.7867499Z Alloc Granule: 4KB 2024-11-01T16:49:17.7868710Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7869861Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7870977Z Accessible by all: TRUE 2024-11-01T16:49:17.7871912Z Pool 2 2024-11-01T16:49:17.7872784Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-11-01T16:49:17.7873930Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:49:17.7875165Z Allocatable: TRUE 2024-11-01T16:49:17.7876388Z Alloc Granule: 4KB 2024-11-01T16:49:17.7877506Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7878661Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7879783Z Accessible by all: TRUE 2024-11-01T16:49:17.7880691Z Pool 3 2024-11-01T16:49:17.7881534Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.7882607Z Size: 528249788(0x1f7c73bc) KB 2024-11-01T16:49:17.7883655Z Allocatable: TRUE 2024-11-01T16:49:17.7884723Z Alloc Granule: 4KB 2024-11-01T16:49:17.7885839Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7886968Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7888054Z Accessible by all: TRUE 2024-11-01T16:49:17.7888975Z ISA Info: 2024-11-01T16:49:17.7889625Z ******* 2024-11-01T16:49:17.7890597Z Agent 2 2024-11-01T16:49:17.7891224Z ******* 2024-11-01T16:49:17.7892153Z Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.7893281Z Uuid: CPU-XX 2024-11-01T16:49:17.7894465Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.7895538Z Vendor Name: CPU 2024-11-01T16:49:17.7896693Z Feature: None specified 2024-11-01T16:49:17.7897716Z Profile: FULL_PROFILE 2024-11-01T16:49:17.7898778Z Float Round Mode: NEAR 2024-11-01T16:49:17.7899874Z Max Queue Number: 0(0x0) 2024-11-01T16:49:17.7900924Z Queue Min Size: 0(0x0) 2024-11-01T16:49:17.7901959Z Queue Max Size: 0(0x0) 2024-11-01T16:49:17.7903359Z Queue Type: MULTI 2024-11-01T16:49:17.7904328Z Node: 1 2024-11-01T16:49:17.7905298Z Device Type: CPU 2024-11-01T16:49:17.7906174Z Cache Info: 2024-11-01T16:49:17.7906978Z L1: 32768(0x8000) KB 2024-11-01T16:49:17.7907983Z Chip ID: 0(0x0) 2024-11-01T16:49:17.7908999Z ASIC Revision: 0(0x0) 2024-11-01T16:49:17.7910051Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.7911127Z Max Clock Freq. (MHz): 2000 2024-11-01T16:49:17.7912126Z BDFID: 0 2024-11-01T16:49:17.7913134Z Internal Node ID: 1 2024-11-01T16:49:17.7914174Z Compute Unit: 64 2024-11-01T16:49:17.7915199Z SIMDs per CU: 0 2024-11-01T16:49:17.7916220Z Shader Engines: 0 2024-11-01T16:49:17.7917274Z Shader Arrs. per Eng.: 0 2024-11-01T16:49:17.7918356Z WatchPts on Addr. Ranges:1 2024-11-01T16:49:17.7919285Z Memory Properties: 2024-11-01T16:49:17.7919970Z Features: None 2024-11-01T16:49:17.7920661Z Pool Info: 2024-11-01T16:49:17.7921323Z Pool 1 2024-11-01T16:49:17.7922189Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.7923268Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:49:17.7924330Z Allocatable: TRUE 2024-11-01T16:49:17.7925419Z Alloc Granule: 4KB 2024-11-01T16:49:17.7926586Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7927761Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7928884Z Accessible by all: TRUE 2024-11-01T16:49:17.7929841Z Pool 2 2024-11-01T16:49:17.7930707Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-11-01T16:49:17.7931789Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:49:17.7932847Z Allocatable: TRUE 2024-11-01T16:49:17.7933924Z Alloc Granule: 4KB 2024-11-01T16:49:17.7935051Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7936178Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7937595Z Accessible by all: TRUE 2024-11-01T16:49:17.7938896Z Pool 3 2024-11-01T16:49:17.7939767Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.7940838Z Size: 528402424(0x1f7ec7f8) KB 2024-11-01T16:49:17.7941869Z Allocatable: TRUE 2024-11-01T16:49:17.7942936Z Alloc Granule: 4KB 2024-11-01T16:49:17.7944063Z Alloc Recommended Granule:4KB 2024-11-01T16:49:17.7945206Z Alloc Alignment: 4KB 2024-11-01T16:49:17.7946299Z Accessible by all: TRUE 2024-11-01T16:49:17.7947213Z ISA Info: 2024-11-01T16:49:17.7947860Z ******* 2024-11-01T16:49:17.7948497Z Agent 3 2024-11-01T16:49:17.7949123Z ******* 2024-11-01T16:49:17.7949889Z Name: gfx90a 2024-11-01T16:49:17.7951389Z Uuid: GPU-8f375f9eb6d0d924 2024-11-01T16:49:17.7952466Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.7953533Z Vendor Name: AMD 2024-11-01T16:49:17.7954550Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.7955556Z Profile: BASE_PROFILE 2024-11-01T16:49:17.7956589Z Float Round Mode: NEAR 2024-11-01T16:49:17.7957637Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.7958685Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.7959714Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.7960720Z Queue Type: MULTI 2024-11-01T16:49:17.7961682Z Node: 2 2024-11-01T16:49:17.7962664Z Device Type: GPU 2024-11-01T16:49:17.7963523Z Cache Info: 2024-11-01T16:49:17.7964305Z L1: 16(0x10) KB 2024-11-01T16:49:17.7965262Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.7966259Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.7967265Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.7968335Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.7969396Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.7970385Z BDFID: 12800 2024-11-01T16:49:17.7971388Z Internal Node ID: 2 2024-11-01T16:49:17.7972417Z Compute Unit: 104 2024-11-01T16:49:17.7973417Z SIMDs per CU: 4 2024-11-01T16:49:17.7974447Z Shader Engines: 8 2024-11-01T16:49:17.7975502Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.7976705Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.7977791Z Coherent Host Access: FALSE 2024-11-01T16:49:17.7978715Z Memory Properties: 2024-11-01T16:49:17.7979454Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.7980433Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.7981488Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.7982537Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.7983465Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.7984314Z x 1024(0x400) 2024-11-01T16:49:17.7985578Z y 1024(0x400) 2024-11-01T16:49:17.7986520Z z 1024(0x400) 2024-11-01T16:49:17.7987528Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.7988734Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.7989806Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.7990720Z Grid Max Size per Dimension: 2024-11-01T16:49:17.7991497Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.7992426Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.7993329Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.7994335Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.7995471Z Packet Processor uCode:: 83 2024-11-01T16:49:17.7996555Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8004340Z IOMMU Support:: None 2024-11-01T16:49:17.8005234Z Pool Info: 2024-11-01T16:49:17.8005889Z Pool 1 2024-11-01T16:49:17.8006753Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8007833Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8008899Z Allocatable: TRUE 2024-11-01T16:49:17.8009984Z Alloc Granule: 4KB 2024-11-01T16:49:17.8011120Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8012261Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8013371Z Accessible by all: FALSE 2024-11-01T16:49:17.8014302Z Pool 2 2024-11-01T16:49:17.8015177Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8016609Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8017679Z Allocatable: TRUE 2024-11-01T16:49:17.8018751Z Alloc Granule: 4KB 2024-11-01T16:49:17.8019872Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8021002Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8022092Z Accessible by all: FALSE 2024-11-01T16:49:17.8023008Z Pool 3 2024-11-01T16:49:17.8023837Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8024772Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8025615Z Allocatable: TRUE 2024-11-01T16:49:17.8026490Z Alloc Granule: 4KB 2024-11-01T16:49:17.8027431Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8028366Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8029261Z Accessible by all: FALSE 2024-11-01T16:49:17.8030012Z Pool 4 2024-11-01T16:49:17.8030675Z Segment: GROUP 2024-11-01T16:49:17.8031509Z Size: 64(0x40) KB 2024-11-01T16:49:17.8032352Z Allocatable: FALSE 2024-11-01T16:49:17.8033226Z Alloc Granule: 0KB 2024-11-01T16:49:17.8034157Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8035105Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8036315Z Accessible by all: FALSE 2024-11-01T16:49:17.8037119Z ISA Info: 2024-11-01T16:49:17.8037668Z ISA 1 2024-11-01T16:49:17.8038517Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8039497Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8040422Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8041341Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8042290Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8043198Z Fast f16: TRUE 2024-11-01T16:49:17.8044084Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8044898Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8045646Z x 1024(0x400) 2024-11-01T16:49:17.8046467Z y 1024(0x400) 2024-11-01T16:49:17.8047557Z z 1024(0x400) 2024-11-01T16:49:17.8048424Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8049231Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8049914Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8050726Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8051530Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8052410Z FBarrier Max Size: 32 2024-11-01T16:49:17.8053179Z ******* 2024-11-01T16:49:17.8053697Z Agent 4 2024-11-01T16:49:17.8054217Z ******* 2024-11-01T16:49:17.8054831Z Name: gfx90a 2024-11-01T16:49:17.8055783Z Uuid: GPU-47e65c59488469bd 2024-11-01T16:49:17.8056787Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8057677Z Vendor Name: AMD 2024-11-01T16:49:17.8058549Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8059413Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8060288Z Float Round Mode: NEAR 2024-11-01T16:49:17.8061187Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8062064Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8062930Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8063778Z Queue Type: MULTI 2024-11-01T16:49:17.8064585Z Node: 3 2024-11-01T16:49:17.8065407Z Device Type: GPU 2024-11-01T16:49:17.8066153Z Cache Info: 2024-11-01T16:49:17.8066809Z L1: 16(0x10) KB 2024-11-01T16:49:17.8067609Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8068437Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8069308Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8070195Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8071077Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8071925Z BDFID: 13568 2024-11-01T16:49:17.8072763Z Internal Node ID: 3 2024-11-01T16:49:17.8073621Z Compute Unit: 104 2024-11-01T16:49:17.8074466Z SIMDs per CU: 4 2024-11-01T16:49:17.8075624Z Shader Engines: 8 2024-11-01T16:49:17.8076511Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8077424Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8078328Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8079109Z Memory Properties: 2024-11-01T16:49:17.8079724Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8080544Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8081425Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8082296Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8083075Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8083790Z x 1024(0x400) 2024-11-01T16:49:17.8084565Z y 1024(0x400) 2024-11-01T16:49:17.8085627Z z 1024(0x400) 2024-11-01T16:49:17.8086483Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8087505Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8088423Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8089187Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8089834Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8090642Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8091423Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8092270Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8093204Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8094109Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8095002Z IOMMU Support:: None 2024-11-01T16:49:17.8095748Z Pool Info: 2024-11-01T16:49:17.8096405Z Pool 1 2024-11-01T16:49:17.8097131Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8098044Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8098930Z Allocatable: TRUE 2024-11-01T16:49:17.8099834Z Alloc Granule: 4KB 2024-11-01T16:49:17.8100792Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8101739Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8102651Z Accessible by all: FALSE 2024-11-01T16:49:17.8103426Z Pool 2 2024-11-01T16:49:17.8104152Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8105055Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8105937Z Allocatable: TRUE 2024-11-01T16:49:17.8106813Z Alloc Granule: 4KB 2024-11-01T16:49:17.8107742Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8108695Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8109617Z Accessible by all: FALSE 2024-11-01T16:49:17.8110373Z Pool 3 2024-11-01T16:49:17.8111069Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8111947Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8112817Z Allocatable: TRUE 2024-11-01T16:49:17.8113693Z Alloc Granule: 4KB 2024-11-01T16:49:17.8115112Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8116257Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8117339Z Accessible by all: FALSE 2024-11-01T16:49:17.8118234Z Pool 4 2024-11-01T16:49:17.8118928Z Segment: GROUP 2024-11-01T16:49:17.8119772Z Size: 64(0x40) KB 2024-11-01T16:49:17.8120624Z Allocatable: FALSE 2024-11-01T16:49:17.8121499Z Alloc Granule: 0KB 2024-11-01T16:49:17.8122434Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8123360Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8124327Z Accessible by all: FALSE 2024-11-01T16:49:17.8125223Z ISA Info: 2024-11-01T16:49:17.8126224Z ISA 1 2024-11-01T16:49:17.8127146Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8128110Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8129018Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8129929Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8130883Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8131776Z Fast f16: TRUE 2024-11-01T16:49:17.8132658Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8133461Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8134199Z x 1024(0x400) 2024-11-01T16:49:17.8134998Z y 1024(0x400) 2024-11-01T16:49:17.8135804Z z 1024(0x400) 2024-11-01T16:49:17.8136947Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8137752Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8138438Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8139230Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8140028Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8140890Z FBarrier Max Size: 32 2024-11-01T16:49:17.8141653Z ******* 2024-11-01T16:49:17.8142173Z Agent 5 2024-11-01T16:49:17.8142682Z ******* 2024-11-01T16:49:17.8143305Z Name: gfx90a 2024-11-01T16:49:17.8144248Z Uuid: GPU-7a90e0d9d56cf49c 2024-11-01T16:49:17.8145172Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8146080Z Vendor Name: AMD 2024-11-01T16:49:17.8146935Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8147784Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8148652Z Float Round Mode: NEAR 2024-11-01T16:49:17.8149533Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8150402Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8151258Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8152105Z Queue Type: MULTI 2024-11-01T16:49:17.8152897Z Node: 4 2024-11-01T16:49:17.8153708Z Device Type: GPU 2024-11-01T16:49:17.8154427Z Cache Info: 2024-11-01T16:49:17.8155448Z L1: 16(0x10) KB 2024-11-01T16:49:17.8156277Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8157095Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8157955Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8158845Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8159732Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8160579Z BDFID: 4352 2024-11-01T16:49:17.8161421Z Internal Node ID: 4 2024-11-01T16:49:17.8162279Z Compute Unit: 104 2024-11-01T16:49:17.8163131Z SIMDs per CU: 4 2024-11-01T16:49:17.8163986Z Shader Engines: 8 2024-11-01T16:49:17.8165162Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8166082Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8167001Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8167775Z Memory Properties: 2024-11-01T16:49:17.8168391Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8169203Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8170087Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8170956Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8171729Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8172435Z x 1024(0x400) 2024-11-01T16:49:17.8173201Z y 1024(0x400) 2024-11-01T16:49:17.8173965Z z 1024(0x400) 2024-11-01T16:49:17.8174824Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8175846Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8177230Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8178011Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8178670Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8179442Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8180215Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8181091Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8182029Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8182930Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8183806Z IOMMU Support:: None 2024-11-01T16:49:17.8184569Z Pool Info: 2024-11-01T16:49:17.8185132Z Pool 1 2024-11-01T16:49:17.8185856Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8186749Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8187630Z Allocatable: TRUE 2024-11-01T16:49:17.8188528Z Alloc Granule: 4KB 2024-11-01T16:49:17.8189475Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8190422Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8191333Z Accessible by all: FALSE 2024-11-01T16:49:17.8192101Z Pool 2 2024-11-01T16:49:17.8192816Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8193690Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8194997Z Allocatable: TRUE 2024-11-01T16:49:17.8196130Z Alloc Granule: 4KB 2024-11-01T16:49:17.8197256Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8198370Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8199327Z Accessible by all: FALSE 2024-11-01T16:49:17.8200079Z Pool 3 2024-11-01T16:49:17.8200778Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8201647Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8202499Z Allocatable: TRUE 2024-11-01T16:49:17.8203374Z Alloc Granule: 4KB 2024-11-01T16:49:17.8204356Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8205824Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8206871Z Accessible by all: FALSE 2024-11-01T16:49:17.8207632Z Pool 4 2024-11-01T16:49:17.8208300Z Segment: GROUP 2024-11-01T16:49:17.8209132Z Size: 64(0x40) KB 2024-11-01T16:49:17.8209981Z Allocatable: FALSE 2024-11-01T16:49:17.8210851Z Alloc Granule: 0KB 2024-11-01T16:49:17.8211775Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8212716Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8213610Z Accessible by all: FALSE 2024-11-01T16:49:17.8214359Z ISA Info: 2024-11-01T16:49:17.8214909Z ISA 1 2024-11-01T16:49:17.8215770Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8216836Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8217762Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8218689Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8219635Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8220516Z Fast f16: TRUE 2024-11-01T16:49:17.8221399Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8222206Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8222947Z x 1024(0x400) 2024-11-01T16:49:17.8223756Z y 1024(0x400) 2024-11-01T16:49:17.8224528Z z 1024(0x400) 2024-11-01T16:49:17.8225415Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8226198Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8226893Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8227677Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8228470Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8229337Z FBarrier Max Size: 32 2024-11-01T16:49:17.8230094Z ******* 2024-11-01T16:49:17.8230612Z Agent 6 2024-11-01T16:49:17.8231106Z ******* 2024-11-01T16:49:17.8231725Z Name: gfx90a 2024-11-01T16:49:17.8232671Z Uuid: GPU-8ad9228e7f5394ba 2024-11-01T16:49:17.8233576Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8234770Z Vendor Name: AMD 2024-11-01T16:49:17.8235660Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8236539Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8237434Z Float Round Mode: NEAR 2024-11-01T16:49:17.8238333Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8239204Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8240060Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8240905Z Queue Type: MULTI 2024-11-01T16:49:17.8241703Z Node: 5 2024-11-01T16:49:17.8242517Z Device Type: GPU 2024-11-01T16:49:17.8243241Z Cache Info: 2024-11-01T16:49:17.8243897Z L1: 16(0x10) KB 2024-11-01T16:49:17.8244994Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8245830Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8246682Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8247559Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8248424Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8249267Z BDFID: 5120 2024-11-01T16:49:17.8250096Z Internal Node ID: 5 2024-11-01T16:49:17.8250959Z Compute Unit: 104 2024-11-01T16:49:17.8251794Z SIMDs per CU: 4 2024-11-01T16:49:17.8252647Z Shader Engines: 8 2024-11-01T16:49:17.8253530Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8254475Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8255397Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8256170Z Memory Properties: 2024-11-01T16:49:17.8257045Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8257867Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8258749Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8259626Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8260407Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8261113Z x 1024(0x400) 2024-11-01T16:49:17.8261874Z y 1024(0x400) 2024-11-01T16:49:17.8262621Z z 1024(0x400) 2024-11-01T16:49:17.8263453Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8264484Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8265388Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8266162Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8266799Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8267566Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8268323Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8269165Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8270087Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8270983Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8271891Z IOMMU Support:: None 2024-11-01T16:49:17.8272633Z Pool Info: 2024-11-01T16:49:17.8273184Z Pool 1 2024-11-01T16:49:17.8274223Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8275148Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8276021Z Allocatable: TRUE 2024-11-01T16:49:17.8276927Z Alloc Granule: 4KB 2024-11-01T16:49:17.8277875Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8278826Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8279745Z Accessible by all: FALSE 2024-11-01T16:49:17.8280517Z Pool 2 2024-11-01T16:49:17.8281240Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8282121Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8282978Z Allocatable: TRUE 2024-11-01T16:49:17.8284157Z Alloc Granule: 4KB 2024-11-01T16:49:17.8285108Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8286050Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8286965Z Accessible by all: FALSE 2024-11-01T16:49:17.8287740Z Pool 3 2024-11-01T16:49:17.8288433Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8289319Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8290183Z Allocatable: TRUE 2024-11-01T16:49:17.8291053Z Alloc Granule: 4KB 2024-11-01T16:49:17.8291992Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8292938Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8293872Z Accessible by all: FALSE 2024-11-01T16:49:17.8294635Z Pool 4 2024-11-01T16:49:17.8295301Z Segment: GROUP 2024-11-01T16:49:17.8296148Z Size: 64(0x40) KB 2024-11-01T16:49:17.8297108Z Allocatable: FALSE 2024-11-01T16:49:17.8297991Z Alloc Granule: 0KB 2024-11-01T16:49:17.8298910Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8299837Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8300732Z Accessible by all: FALSE 2024-11-01T16:49:17.8301488Z ISA Info: 2024-11-01T16:49:17.8302041Z ISA 1 2024-11-01T16:49:17.8302899Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8303871Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8304788Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8305716Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8306662Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8307550Z Fast f16: TRUE 2024-11-01T16:49:17.8308447Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8309262Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8310008Z x 1024(0x400) 2024-11-01T16:49:17.8310802Z y 1024(0x400) 2024-11-01T16:49:17.8311578Z z 1024(0x400) 2024-11-01T16:49:17.8312777Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8313600Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8314292Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8315086Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8315884Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8316761Z FBarrier Max Size: 32 2024-11-01T16:49:17.8317520Z ******* 2024-11-01T16:49:17.8318034Z Agent 7 2024-11-01T16:49:17.8318544Z ******* 2024-11-01T16:49:17.8319164Z Name: gfx90a 2024-11-01T16:49:17.8320120Z Uuid: GPU-bdbf1ce15a4e2cd8 2024-11-01T16:49:17.8321027Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8321911Z Vendor Name: AMD 2024-11-01T16:49:17.8323059Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8323909Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8324768Z Float Round Mode: NEAR 2024-11-01T16:49:17.8325650Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8326544Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8327433Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8328285Z Queue Type: MULTI 2024-11-01T16:49:17.8329090Z Node: 6 2024-11-01T16:49:17.8329902Z Device Type: GPU 2024-11-01T16:49:17.8330624Z Cache Info: 2024-11-01T16:49:17.8331285Z L1: 16(0x10) KB 2024-11-01T16:49:17.8332093Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8332933Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8333792Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8334671Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8335552Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8336481Z BDFID: 44544 2024-11-01T16:49:17.8337315Z Internal Node ID: 6 2024-11-01T16:49:17.8338173Z Compute Unit: 104 2024-11-01T16:49:17.8339003Z SIMDs per CU: 4 2024-11-01T16:49:17.8339848Z Shader Engines: 8 2024-11-01T16:49:17.8340724Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8341645Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8342568Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8343337Z Memory Properties: 2024-11-01T16:49:17.8343957Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8344792Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8345679Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8346556Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8347340Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8348056Z x 1024(0x400) 2024-11-01T16:49:17.8348827Z y 1024(0x400) 2024-11-01T16:49:17.8349583Z z 1024(0x400) 2024-11-01T16:49:17.8350418Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8351725Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8352676Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8353455Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8354101Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8354878Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8355662Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8356505Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8357444Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8358339Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8359216Z IOMMU Support:: None 2024-11-01T16:49:17.8359951Z Pool Info: 2024-11-01T16:49:17.8360502Z Pool 1 2024-11-01T16:49:17.8361228Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8362422Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8363315Z Allocatable: TRUE 2024-11-01T16:49:17.8364218Z Alloc Granule: 4KB 2024-11-01T16:49:17.8365168Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8366124Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8367048Z Accessible by all: FALSE 2024-11-01T16:49:17.8367819Z Pool 2 2024-11-01T16:49:17.8368539Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8369420Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8370289Z Allocatable: TRUE 2024-11-01T16:49:17.8371171Z Alloc Granule: 4KB 2024-11-01T16:49:17.8372113Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8373032Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8373932Z Accessible by all: FALSE 2024-11-01T16:49:17.8374687Z Pool 3 2024-11-01T16:49:17.8375376Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8376253Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8377356Z Allocatable: TRUE 2024-11-01T16:49:17.8378232Z Alloc Granule: 4KB 2024-11-01T16:49:17.8379168Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8380089Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8381013Z Accessible by all: FALSE 2024-11-01T16:49:17.8381784Z Pool 4 2024-11-01T16:49:17.8382444Z Segment: GROUP 2024-11-01T16:49:17.8383281Z Size: 64(0x40) KB 2024-11-01T16:49:17.8384124Z Allocatable: FALSE 2024-11-01T16:49:17.8385121Z Alloc Granule: 0KB 2024-11-01T16:49:17.8386209Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8387313Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8388364Z Accessible by all: FALSE 2024-11-01T16:49:17.8388870Z ISA Info: 2024-11-01T16:49:17.8389184Z ISA 1 2024-11-01T16:49:17.8389656Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8390352Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8390878Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8391398Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8391913Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8392406Z Fast f16: TRUE 2024-11-01T16:49:17.8392890Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8393343Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8393759Z x 1024(0x400) 2024-11-01T16:49:17.8394200Z y 1024(0x400) 2024-11-01T16:49:17.8394643Z z 1024(0x400) 2024-11-01T16:49:17.8395124Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8395556Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8396092Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8396525Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8396964Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8397438Z FBarrier Max Size: 32 2024-11-01T16:49:17.8397853Z ******* 2024-11-01T16:49:17.8398134Z Agent 8 2024-11-01T16:49:17.8398417Z ******* 2024-11-01T16:49:17.8398760Z Name: gfx90a 2024-11-01T16:49:17.8399284Z Uuid: GPU-fbca43c355c8b7ae 2024-11-01T16:49:17.8399783Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8400267Z Vendor Name: AMD 2024-11-01T16:49:17.8400742Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8401214Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8401694Z Float Round Mode: NEAR 2024-11-01T16:49:17.8402179Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8402657Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8403125Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8403587Z Queue Type: MULTI 2024-11-01T16:49:17.8404029Z Node: 7 2024-11-01T16:49:17.8404501Z Device Type: GPU 2024-11-01T16:49:17.8404911Z Cache Info: 2024-11-01T16:49:17.8405269Z L1: 16(0x10) KB 2024-11-01T16:49:17.8405707Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8406166Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8406634Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8407117Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8407598Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8408055Z BDFID: 45824 2024-11-01T16:49:17.8408516Z Internal Node ID: 7 2024-11-01T16:49:17.8408988Z Compute Unit: 104 2024-11-01T16:49:17.8409450Z SIMDs per CU: 4 2024-11-01T16:49:17.8409921Z Shader Engines: 8 2024-11-01T16:49:17.8410401Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8410893Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8411526Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8411958Z Memory Properties: 2024-11-01T16:49:17.8412302Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8412749Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8413240Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8413719Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8414151Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8414546Z x 1024(0x400) 2024-11-01T16:49:17.8414973Z y 1024(0x400) 2024-11-01T16:49:17.8415385Z z 1024(0x400) 2024-11-01T16:49:17.8415843Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8416474Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8416986Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8417618Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8417990Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8418427Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8418853Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8419320Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8419836Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8420331Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8420812Z IOMMU Support:: None 2024-11-01T16:49:17.8421216Z Pool Info: 2024-11-01T16:49:17.8421515Z Pool 1 2024-11-01T16:49:17.8421914Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8422413Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8422907Z Allocatable: TRUE 2024-11-01T16:49:17.8423409Z Alloc Granule: 4KB 2024-11-01T16:49:17.8423934Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8424471Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8424985Z Accessible by all: FALSE 2024-11-01T16:49:17.8425413Z Pool 2 2024-11-01T16:49:17.8425816Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8426310Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8426787Z Allocatable: TRUE 2024-11-01T16:49:17.8427277Z Alloc Granule: 4KB 2024-11-01T16:49:17.8427794Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8428314Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8428812Z Accessible by all: FALSE 2024-11-01T16:49:17.8429230Z Pool 3 2024-11-01T16:49:17.8429610Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8430094Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8430560Z Allocatable: TRUE 2024-11-01T16:49:17.8431045Z Alloc Granule: 4KB 2024-11-01T16:49:17.8431563Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8432082Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8432581Z Accessible by all: FALSE 2024-11-01T16:49:17.8433003Z Pool 4 2024-11-01T16:49:17.8433530Z Segment: GROUP 2024-11-01T16:49:17.8434011Z Size: 64(0x40) KB 2024-11-01T16:49:17.8434502Z Allocatable: FALSE 2024-11-01T16:49:17.8434990Z Alloc Granule: 0KB 2024-11-01T16:49:17.8435513Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8436038Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8436546Z Accessible by all: FALSE 2024-11-01T16:49:17.8436972Z ISA Info: 2024-11-01T16:49:17.8437278Z ISA 1 2024-11-01T16:49:17.8437742Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8438278Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8438793Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8439440Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8439967Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8440455Z Fast f16: TRUE 2024-11-01T16:49:17.8440938Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8441388Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8441797Z x 1024(0x400) 2024-11-01T16:49:17.8442238Z y 1024(0x400) 2024-11-01T16:49:17.8442671Z z 1024(0x400) 2024-11-01T16:49:17.8443152Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8443594Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8443978Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8444428Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8444881Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8445356Z FBarrier Max Size: 32 2024-11-01T16:49:17.8445772Z ******* 2024-11-01T16:49:17.8446062Z Agent 9 2024-11-01T16:49:17.8446337Z ******* 2024-11-01T16:49:17.8446680Z Name: gfx90a 2024-11-01T16:49:17.8447201Z Uuid: GPU-224bf7aab4354d06 2024-11-01T16:49:17.8447700Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8448187Z Vendor Name: AMD 2024-11-01T16:49:17.8448658Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8449128Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8449616Z Float Round Mode: NEAR 2024-11-01T16:49:17.8450104Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8450587Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8451057Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8451529Z Queue Type: MULTI 2024-11-01T16:49:17.8451971Z Node: 8 2024-11-01T16:49:17.8452422Z Device Type: GPU 2024-11-01T16:49:17.8452820Z Cache Info: 2024-11-01T16:49:17.8453184Z L1: 16(0x10) KB 2024-11-01T16:49:17.8453625Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8454088Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8454716Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8455225Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8455711Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8456176Z BDFID: 36352 2024-11-01T16:49:17.8456709Z Internal Node ID: 8 2024-11-01T16:49:17.8457199Z Compute Unit: 104 2024-11-01T16:49:17.8457665Z SIMDs per CU: 4 2024-11-01T16:49:17.8458139Z Shader Engines: 8 2024-11-01T16:49:17.8458632Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8459134Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8459644Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8460240Z Memory Properties: 2024-11-01T16:49:17.8460589Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8461044Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8461534Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8462018Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8462457Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8462862Z x 1024(0x400) 2024-11-01T16:49:17.8463292Z y 1024(0x400) 2024-11-01T16:49:17.8463713Z z 1024(0x400) 2024-11-01T16:49:17.8464173Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8464739Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8465253Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8465382Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8465540Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8465684Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8465834Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8466023Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8466227Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8466407Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8466599Z IOMMU Support:: None 2024-11-01T16:49:17.8466707Z Pool Info: 2024-11-01T16:49:17.8466819Z Pool 1 2024-11-01T16:49:17.8467020Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8467190Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8467396Z Allocatable: TRUE 2024-11-01T16:49:17.8467577Z Alloc Granule: 4KB 2024-11-01T16:49:17.8467799Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8467981Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8468182Z Accessible by all: FALSE 2024-11-01T16:49:17.8468290Z Pool 2 2024-11-01T16:49:17.8468494Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8468662Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8468846Z Allocatable: TRUE 2024-11-01T16:49:17.8469025Z Alloc Granule: 4KB 2024-11-01T16:49:17.8469225Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8469569Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8469775Z Accessible by all: FALSE 2024-11-01T16:49:17.8469891Z Pool 3 2024-11-01T16:49:17.8470076Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8470239Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8470425Z Allocatable: TRUE 2024-11-01T16:49:17.8470611Z Alloc Granule: 4KB 2024-11-01T16:49:17.8470814Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8471000Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8471186Z Accessible by all: FALSE 2024-11-01T16:49:17.8471309Z Pool 4 2024-11-01T16:49:17.8471489Z Segment: GROUP 2024-11-01T16:49:17.8471778Z Size: 64(0x40) KB 2024-11-01T16:49:17.8471963Z Allocatable: FALSE 2024-11-01T16:49:17.8472138Z Alloc Granule: 0KB 2024-11-01T16:49:17.8472345Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8472521Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8472717Z Accessible by all: FALSE 2024-11-01T16:49:17.8472822Z ISA Info: 2024-11-01T16:49:17.8472932Z ISA 1 2024-11-01T16:49:17.8473194Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8473399Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8473589Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8473791Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8473992Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8474163Z Fast f16: TRUE 2024-11-01T16:49:17.8474360Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8474487Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8474647Z x 1024(0x400) 2024-11-01T16:49:17.8474802Z y 1024(0x400) 2024-11-01T16:49:17.8474962Z z 1024(0x400) 2024-11-01T16:49:17.8475154Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8475278Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8475435Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8475597Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8475770Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8475958Z FBarrier Max Size: 32 2024-11-01T16:49:17.8476068Z ******* 2024-11-01T16:49:17.8476166Z Agent 10 2024-11-01T16:49:17.8476271Z ******* 2024-11-01T16:49:17.8476437Z Name: gfx90a 2024-11-01T16:49:17.8476666Z Uuid: GPU-8473d8ace24d57eb 2024-11-01T16:49:17.8476859Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.8477037Z Vendor Name: AMD 2024-11-01T16:49:17.8477203Z Feature: KERNEL_DISPATCH 2024-11-01T16:49:17.8477381Z Profile: BASE_PROFILE 2024-11-01T16:49:17.8477698Z Float Round Mode: NEAR 2024-11-01T16:49:17.8477899Z Max Queue Number: 128(0x80) 2024-11-01T16:49:17.8478082Z Queue Min Size: 64(0x40) 2024-11-01T16:49:17.8478252Z Queue Max Size: 131072(0x20000) 2024-11-01T16:49:17.8478426Z Queue Type: MULTI 2024-11-01T16:49:17.8478580Z Node: 9 2024-11-01T16:49:17.8478754Z Device Type: GPU 2024-11-01T16:49:17.8478858Z Cache Info: 2024-11-01T16:49:17.8479022Z L1: 16(0x10) KB 2024-11-01T16:49:17.8479173Z L2: 8192(0x2000) KB 2024-11-01T16:49:17.8479357Z Chip ID: 29708(0x740c) 2024-11-01T16:49:17.8479533Z ASIC Revision: 1(0x1) 2024-11-01T16:49:17.8479875Z Cacheline Size: 64(0x40) 2024-11-01T16:49:17.8480067Z Max Clock Freq. (MHz): 1700 2024-11-01T16:49:17.8480223Z BDFID: 37632 2024-11-01T16:49:17.8480406Z Internal Node ID: 9 2024-11-01T16:49:17.8480576Z Compute Unit: 104 2024-11-01T16:49:17.8480747Z SIMDs per CU: 4 2024-11-01T16:49:17.8480918Z Shader Engines: 8 2024-11-01T16:49:17.8481107Z Shader Arrs. per Eng.: 1 2024-11-01T16:49:17.8481296Z WatchPts on Addr. Ranges:4 2024-11-01T16:49:17.8481493Z Coherent Host Access: FALSE 2024-11-01T16:49:17.8481605Z Memory Properties: 2024-11-01T16:49:17.8481753Z Features: KERNEL_DISPATCH 2024-11-01T16:49:17.8481947Z Fast F16 Operation: TRUE 2024-11-01T16:49:17.8482129Z Wavefront Size: 64(0x40) 2024-11-01T16:49:17.8482315Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8482443Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8482601Z x 1024(0x400) 2024-11-01T16:49:17.8482745Z y 1024(0x400) 2024-11-01T16:49:17.8482889Z z 1024(0x400) 2024-11-01T16:49:17.8483075Z Max Waves Per CU: 32(0x20) 2024-11-01T16:49:17.8483327Z Max Work-item Per CU: 2048(0x800) 2024-11-01T16:49:17.8483506Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8483630Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8483778Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8483935Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8484089Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8484283Z Max fbarriers/Workgrp: 32 2024-11-01T16:49:17.8484487Z Packet Processor uCode:: 83 2024-11-01T16:49:17.8484668Z SDMA engine uCode:: 8 2024-11-01T16:49:17.8484852Z IOMMU Support:: None 2024-11-01T16:49:17.8484959Z Pool Info: 2024-11-01T16:49:17.8485073Z Pool 1 2024-11-01T16:49:17.8485269Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-11-01T16:49:17.8485446Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8485624Z Allocatable: TRUE 2024-11-01T16:49:17.8485961Z Alloc Granule: 4KB 2024-11-01T16:49:17.8486175Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8486369Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8486569Z Accessible by all: FALSE 2024-11-01T16:49:17.8486672Z Pool 2 2024-11-01T16:49:17.8486873Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-11-01T16:49:17.8487030Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8487214Z Allocatable: TRUE 2024-11-01T16:49:17.8487389Z Alloc Granule: 4KB 2024-11-01T16:49:17.8487593Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8487773Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8488098Z Accessible by all: FALSE 2024-11-01T16:49:17.8488199Z Pool 3 2024-11-01T16:49:17.8488384Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-11-01T16:49:17.8488554Z Size: 67092480(0x3ffc000) KB 2024-11-01T16:49:17.8488732Z Allocatable: TRUE 2024-11-01T16:49:17.8488925Z Alloc Granule: 4KB 2024-11-01T16:49:17.8489130Z Alloc Recommended Granule:2048KB 2024-11-01T16:49:17.8489316Z Alloc Alignment: 4KB 2024-11-01T16:49:17.8489503Z Accessible by all: FALSE 2024-11-01T16:49:17.8489612Z Pool 4 2024-11-01T16:49:17.8489786Z Segment: GROUP 2024-11-01T16:49:17.8489952Z Size: 64(0x40) KB 2024-11-01T16:49:17.8490133Z Allocatable: FALSE 2024-11-01T16:49:17.8490312Z Alloc Granule: 0KB 2024-11-01T16:49:17.8490517Z Alloc Recommended Granule:0KB 2024-11-01T16:49:17.8490712Z Alloc Alignment: 0KB 2024-11-01T16:49:17.8490905Z Accessible by all: FALSE 2024-11-01T16:49:17.8491009Z ISA Info: 2024-11-01T16:49:17.8491113Z ISA 1 2024-11-01T16:49:17.8491374Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-11-01T16:49:17.8491579Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-11-01T16:49:17.8491755Z Profiles: HSA_PROFILE_BASE 2024-11-01T16:49:17.8491958Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8492151Z Default Rounding Mode: NEAR 2024-11-01T16:49:17.8492325Z Fast f16: TRUE 2024-11-01T16:49:17.8492512Z Workgroup Max Size: 1024(0x400) 2024-11-01T16:49:17.8492659Z Workgroup Max Size per Dimension: 2024-11-01T16:49:17.8492823Z x 1024(0x400) 2024-11-01T16:49:17.8492977Z y 1024(0x400) 2024-11-01T16:49:17.8493139Z z 1024(0x400) 2024-11-01T16:49:17.8493321Z Grid Max Size: 4294967295(0xffffffff) 2024-11-01T16:49:17.8493451Z Grid Max Size per Dimension: 2024-11-01T16:49:17.8493605Z x 4294967295(0xffffffff) 2024-11-01T16:49:17.8493765Z y 4294967295(0xffffffff) 2024-11-01T16:49:17.8494057Z z 4294967295(0xffffffff) 2024-11-01T16:49:17.8494259Z FBarrier Max Size: 32 2024-11-01T16:49:17.8494361Z *** Done *** 2024-11-01T16:49:17.8494463Z + rocminfo 2024-11-01T16:49:17.8494619Z + grep -E 'Name:.*\sgfx|Marketing' 2024-11-01T16:49:17.9528074Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.9528685Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-11-01T16:49:17.9529080Z Name: gfx90a 2024-11-01T16:49:17.9529484Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9529791Z Name: gfx90a 2024-11-01T16:49:17.9530187Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9530486Z Name: gfx90a 2024-11-01T16:49:17.9530887Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9531696Z Name: gfx90a 2024-11-01T16:49:17.9532086Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9532417Z Name: gfx90a 2024-11-01T16:49:17.9532786Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9533094Z Name: gfx90a 2024-11-01T16:49:17.9533478Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9533791Z Name: gfx90a 2024-11-01T16:49:17.9534156Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9534457Z Name: gfx90a 2024-11-01T16:49:17.9534817Z Marketing Name: AMD Instinct MI250X/MI250 2024-11-01T16:49:17.9716884Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-11-01T16:49:17.9718083Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-11-01T16:49:17.9719065Z + pip_install --user ninja==1.10.2 2024-11-01T16:49:17.9720095Z + pip install --progress-bar off --user ninja==1.10.2 2024-11-01T16:49:18.5656768Z Collecting ninja==1.10.2 2024-11-01T16:49:18.6426861Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-11-01T16:49:18.6738604Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-11-01T16:49:19.0498919Z Installing collected packages: ninja 2024-11-01T16:49:19.0582112Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-11-01T16:49:19.0584492Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-11-01T16:49:19.0602469Z Successfully installed ninja-1.10.2 2024-11-01T16:49:19.1303815Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:49:19.1308773Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:49:19.1311678Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-11-01T16:49:19.1312508Z + install_tlparse 2024-11-01T16:49:19.1313197Z + pip_install --user tlparse==0.3.25 2024-11-01T16:49:19.1314211Z + pip install --progress-bar off --user tlparse==0.3.25 2024-11-01T16:49:19.7240580Z Collecting tlparse==0.3.25 2024-11-01T16:49:19.8044468Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-11-01T16:49:19.8398336Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-11-01T16:49:20.3169227Z Installing collected packages: tlparse 2024-11-01T16:49:20.3665093Z Successfully installed tlparse-0.3.25 2024-11-01T16:49:20.4422442Z ++ python -m site --user-base 2024-11-01T16:49:20.4929027Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-11-01T16:49:20.4931265Z + [[ linux-focal-rocm6.2-py3.10 == *asan* ]] 2024-11-01T16:49:20.4932000Z + [[ linux-focal-rocm6.2-py3.10 == *-debug* ]] 2024-11-01T16:49:20.4932613Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-11-01T16:49:20.4933517Z + echo 'We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass' 2024-11-01T16:49:20.4934640Z We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass 2024-11-01T16:49:20.4940697Z + cd test 2024-11-01T16:49:20.4943361Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-11-01T16:49:24.1481950Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-11-01T16:49:24.1483157Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-11-01T16:49:24.1494069Z + DYNAMO_BENCHMARK_FLAGS=() 2024-11-01T16:49:24.1495114Z + [[ distributed == *pr_time_benchmarks* ]] 2024-11-01T16:49:24.1496059Z + [[ distributed == *dynamo_eager* ]] 2024-11-01T16:49:24.1497323Z + [[ distributed == *aot_eager* ]] 2024-11-01T16:49:24.1498139Z + [[ distributed == *aot_inductor* ]] 2024-11-01T16:49:24.1498930Z + [[ distributed == *inductor* ]] 2024-11-01T16:49:24.1499684Z + [[ distributed == *dynamic* ]] 2024-11-01T16:49:24.1500416Z + [[ distributed == *cpu* ]] 2024-11-01T16:49:24.1501450Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-11-01T16:49:24.1559368Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-11-01T16:49:24.1560430Z + [[ linux-focal-rocm6.2-py3.10 == *-bazel-* ]] 2024-11-01T16:49:24.1569806Z + cd test 2024-11-01T16:49:24.1571118Z + python -c 'import torch; print(torch.__config__.show())' 2024-11-01T16:49:27.1703127Z PyTorch built with: 2024-11-01T16:49:27.1704062Z - GCC 9.4 2024-11-01T16:49:27.1704677Z - C++ Version: 201703 2024-11-01T16:49:27.1706309Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:49:27.1708345Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:49:27.1709581Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:49:27.1710515Z - LAPACK is enabled (usually provided by MKL) 2024-11-01T16:49:27.1711390Z - NNPACK is enabled 2024-11-01T16:49:27.1712084Z - CPU capability usage: AVX2 2024-11-01T16:49:27.1712831Z - HIP Runtime 6.2.41133 2024-11-01T16:49:27.1713493Z - MIOpen 3.2.0 2024-11-01T16:49:27.1714082Z - Magma 2.7.2 2024-11-01T16:49:27.1727288Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-11-01T16:49:27.1738576Z 2024-11-01T16:49:27.8965491Z + cd test 2024-11-01T16:49:27.8966619Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-11-01T16:49:29.4593879Z ATen/Parallel: 2024-11-01T16:49:29.4594675Z at::get_num_threads() : 128 2024-11-01T16:49:29.4595414Z at::get_num_interop_threads() : 128 2024-11-01T16:49:29.4596215Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-11-01T16:49:29.4596943Z omp_get_max_threads() : 128 2024-11-01T16:49:29.4598821Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-11-01T16:49:29.4600314Z mkl_get_max_threads() : 128 2024-11-01T16:49:29.4601495Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-11-01T16:49:29.4602638Z std::thread::hardware_concurrency() : 128 2024-11-01T16:49:29.4603407Z Environment variables: 2024-11-01T16:49:29.4604030Z OMP_NUM_THREADS : [not set] 2024-11-01T16:49:29.4639844Z MKL_NUM_THREADS : [not set] 2024-11-01T16:49:29.4640530Z ATen parallel backend: OpenMP 2024-11-01T16:49:29.4640976Z 2024-11-01T16:49:31.5738866Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-11-01T16:49:31.5739812Z + [[ distributed == *backward* ]] 2024-11-01T16:49:31.5740570Z + [[ distributed == *xla* ]] 2024-11-01T16:49:31.5741261Z + [[ distributed == *executorch* ]] 2024-11-01T16:49:31.5742072Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-11-01T16:49:31.5743052Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-11-01T16:49:31.5743914Z + [[ distributed == distributed ]] 2024-11-01T16:49:31.5744609Z + test_distributed 2024-11-01T16:49:31.5745337Z + echo 'Testing distributed python tests' 2024-11-01T16:49:31.5746155Z Testing distributed python tests 2024-11-01T16:49:31.5749035Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose 2024-11-01T16:49:31.7023921Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T16:49:31.7026061Z import pkg_resources 2024-11-01T16:49:38.0819620Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-11-01T16:49:38.2795505Z Ignoring disabled issues: [''] 2024-11-01T16:49:38.3052719Z Found test times from artifacts 2024-11-01T16:49:38.3666976Z Found test times from artifacts 2024-11-01T16:49:38.3682830Z Running all tests 2024-11-01T16:49:38.3687001Z Running parallel tests on 8 processes 2024-11-01T16:49:38.3687835Z Name: tests to run (est. time: 27.52min) 2024-11-01T16:49:38.3688619Z Serial tests (4): 2024-11-01T16:49:38.3689259Z distributed/test_c10d_common 1/1 2024-11-01T16:49:38.3690023Z distributed/test_c10d_nccl 1/3 2024-11-01T16:49:38.3690804Z distributed/test_c10d_nccl 2/3 2024-11-01T16:49:38.3691508Z distributed/test_c10d_nccl 3/3 2024-11-01T16:49:38.3692201Z Parallel tests (0): 2024-11-01T16:49:38.3692826Z Name: excluded (est. time: 0.0min) 2024-11-01T16:49:38.3693497Z Serial tests (0): 2024-11-01T16:49:38.3694064Z Parallel tests (0): 2024-11-01T16:49:38.3834845Z Running distributed/test_c10d_common 1/1 ... [2024-11-01 16:49:38.383147] 2024-11-01T16:49:38.3835971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T16:49:38.3839196Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:49:38.383497] 2024-11-01T16:54:16.8426064Z 2024-11-01T16:54:16.8428455Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_4129bc862a2e16a9_.log 2024-11-01T16:54:16.8452062Z Running 23 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-11-01T16:54:16.8474367Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier 2024-11-01T16:54:16.8476583Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype 2024-11-01T16:54:16.8478961Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype 2024-11-01T16:54:16.8481412Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype 2024-11-01T16:54:16.8483826Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype 2024-11-01T16:54:16.8485955Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level 2024-11-01T16:54:16.8488063Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr 2024-11-01T16:54:16.8490420Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config 2024-11-01T16:54:16.8492685Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives 2024-11-01T16:54:16.8495015Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name 2024-11-01T16:54:16.8498300Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends 2024-11-01T16:54:16.8500894Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available 2024-11-01T16:54:16.8503617Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv 2024-11-01T16:54:16.8506509Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-11-01T16:54:16.8509853Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-11-01T16:54:16.8512404Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop 2024-11-01T16:54:16.8514476Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable 2024-11-01T16:54:16.8516723Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal 2024-11-01T16:54:16.8518761Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle 2024-11-01T16:54:16.8520676Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank 2024-11-01T16:54:16.8523064Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback 2024-11-01T16:54:16.8525094Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv 2024-11-01T16:54:16.8527047Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback 2024-11-01T16:54:16.8528211Z 2024-11-01T16:54:16.8529155Z Running distributed/test_c10d_nccl 1/3 ... [2024-11-01 16:54:16.843849] 2024-11-01T16:54:16.8530204Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T16:54:16.8533348Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 16:54:16.844421] 2024-11-01T17:14:05.7234669Z 2024-11-01T17:14:05.7237029Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_8c29d79fc9f58b3a_.log 2024-11-01T17:14:05.7305264Z Running 75 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-11-01T17:14:05.7371032Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors 2024-11-01T17:14:05.7373180Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus 2024-11-01T17:14:05.7375433Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs 2024-11-01T17:14:05.7377850Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg 2024-11-01T17:14:05.7380040Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_multi_pg_unordered 2024-11-01T17:14:05.7382403Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False 2024-11-01T17:14:05.7384734Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True 2024-11-01T17:14:05.7386977Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group 2024-11-01T17:14:05.7389325Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl 2024-11-01T17:14:05.7391655Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check 2024-11-01T17:14:05.7393956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout 2024-11-01T17:14:05.7396291Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16 2024-11-01T17:14:05.7398559Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2 2024-11-01T17:14:05.7400868Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True 2024-11-01T17:14:05.7403122Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init 2024-11-01T17:14:05.7405447Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module 2024-11-01T17:14:05.7407876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view 2024-11-01T17:14:05.7410222Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output 2024-11-01T17:14:05.7413011Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-11-01T17:14:05.7415703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-11-01T17:14:05.7418585Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False 2024-11-01T17:14:05.7421352Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl 2024-11-01T17:14:05.7423811Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params 2024-11-01T17:14:05.7426116Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters 2024-11-01T17:14:05.7428585Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view 2024-11-01T17:14:05.7431516Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info 2024-11-01T17:14:05.7434175Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off 2024-11-01T17:14:05.7436922Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail 2024-11-01T17:14:05.7439764Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off 2024-11-01T17:14:05.7442150Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16 2024-11-01T17:14:05.7444358Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl 2024-11-01T17:14:05.7446686Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view 2024-11-01T17:14:05.7449175Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess 2024-11-01T17:14:05.7451703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module 2024-11-01T17:14:05.7454050Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module 2024-11-01T17:14:05.7457021Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None 2024-11-01T17:14:05.7459816Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids 2024-11-01T17:14:05.7462455Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason 2024-11-01T17:14:05.7464688Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad 2024-11-01T17:14:05.7466905Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error 2024-11-01T17:14:05.7469155Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl 2024-11-01T17:14:05.7471223Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object 2024-11-01T17:14:05.7473142Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast 2024-11-01T17:14:05.7475265Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq 2024-11-01T17:14:05.7477104Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp 2024-11-01T17:14:05.7478954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_get_future_result 2024-11-01T17:14:05.7480901Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env 2024-11-01T17:14:05.7482982Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_nonzero_exit 2024-11-01T17:14:05.7485058Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigterm 2024-11-01T17:14:05.7487130Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier 2024-11-01T17:14:05.7489177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl_float8_errors 2024-11-01T17:14:05.7491361Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config 2024-11-01T17:14:05.7493150Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced 2024-11-01T17:14:05.7495033Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default 2024-11-01T17:14:05.7497071Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup 2024-11-01T17:14:05.7498942Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group 2024-11-01T17:14:05.7500596Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor 2024-11-01T17:14:05.7502561Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base 2024-11-01T17:14:05.7504997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2 2024-11-01T17:14:05.7507618Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend 2024-11-01T17:14:05.7509790Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup 2024-11-01T17:14:05.7511553Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup 2024-11-01T17:14:05.7513284Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup 2024-11-01T17:14:05.7514987Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup 2024-11-01T17:14:05.7516758Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata 2024-11-01T17:14:05.7518490Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe 2024-11-01T17:14:05.7520408Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True 2024-11-01T17:14:05.7522267Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long 2024-11-01T17:14:05.7524211Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False 2024-11-01T17:14:05.7526552Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False 2024-11-01T17:14:05.7528897Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True 2024-11-01T17:14:05.7542772Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False 2024-11-01T17:14:05.7545197Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True 2024-11-01T17:14:05.7547282Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired 2024-11-01T17:14:05.7549344Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale 2024-11-01T17:14:05.7550590Z 2024-11-01T17:14:05.7551218Z Running distributed/test_c10d_nccl 2/3 ... [2024-11-01 17:14:05.725902] 2024-11-01T17:14:05.7552169Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:14:05.7554979Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:14:05.726461] 2024-11-01T17:30:48.8286165Z 2024-11-01T17:30:48.8288768Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_f3a658e11989f39f_.log 2024-11-01T17:30:48.8367223Z Running 67 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-11-01T17:30:48.8427310Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl 2024-11-01T17:30:48.8429940Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg 2024-11-01T17:30:48.8432253Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup 2024-11-01T17:30:48.8434556Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg 2024-11-01T17:30:48.8436973Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0 2024-11-01T17:30:48.8439291Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16 2024-11-01T17:30:48.8441500Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64 2024-11-01T17:30:48.8443774Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn 2024-11-01T17:30:48.8446272Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check 2024-11-01T17:30:48.8448353Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter 2024-11-01T17:30:48.8450599Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False 2024-11-01T17:30:48.8452834Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p 2024-11-01T17:30:48.8455099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0 2024-11-01T17:30:48.8457563Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc 2024-11-01T17:30:48.8460071Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view 2024-11-01T17:30:48.8462882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value 2024-11-01T17:30:48.8465765Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl 2024-11-01T17:30:48.8468342Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view 2024-11-01T17:30:48.8470891Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-11-01T17:30:48.8473822Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-11-01T17:30:48.8476681Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False 2024-11-01T17:30:48.8479488Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view 2024-11-01T17:30:48.8482210Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph 2024-11-01T17:30:48.8484856Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl 2024-11-01T17:30:48.8487370Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config 2024-11-01T17:30:48.8489722Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing 2024-11-01T17:30:48.8492035Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl 2024-11-01T17:30:48.8494891Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail 2024-11-01T17:30:48.8497973Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule 2024-11-01T17:30:48.8500330Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state 2024-11-01T17:30:48.8502733Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward 2024-11-01T17:30:48.8505072Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list 2024-11-01T17:30:48.8507543Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list 2024-11-01T17:30:48.8510313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None 2024-11-01T17:30:48.8512486Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg 2024-11-01T17:30:48.8514620Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view 2024-11-01T17:30:48.8516693Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops 2024-11-01T17:30:48.8518663Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_clean_exit 2024-11-01T17:30:48.8520703Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_sigkill 2024-11-01T17:30:48.8522683Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking 2024-11-01T17:30:48.8524590Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl 2024-11-01T17:30:48.8526391Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl 2024-11-01T17:30:48.8528131Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl 2024-11-01T17:30:48.8529765Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier 2024-11-01T17:30:48.8531384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids 2024-11-01T17:30:48.8533154Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off 2024-11-01T17:30:48.8534912Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership 2024-11-01T17:30:48.8536840Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream 2024-11-01T17:30:48.8538655Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k 2024-11-01T17:30:48.8540444Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k_float8_errors 2024-11-01T17:30:48.8542109Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited 2024-11-01T17:30:48.8543997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives 2024-11-01T17:30:48.8546177Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0 2024-11-01T17:30:48.8548237Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1 2024-11-01T17:30:48.8550454Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync 2024-11-01T17:30:48.8552306Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_False 2024-11-01T17:30:48.8554246Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_async_op_True 2024-11-01T17:30:48.8556403Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False 2024-11-01T17:30:48.8558814Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False 2024-11-01T17:30:48.8561172Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True 2024-11-01T17:30:48.8563476Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False 2024-11-01T17:30:48.8566027Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True 2024-11-01T17:30:48.8568285Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True 2024-11-01T17:30:48.8570569Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False 2024-11-01T17:30:48.8572877Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True 2024-11-01T17:30:48.8575197Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False 2024-11-01T17:30:48.8577545Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group 2024-11-01T17:30:48.8578761Z 2024-11-01T17:30:48.8579634Z Running distributed/test_c10d_nccl 3/3 ... [2024-11-01 17:30:48.831057] 2024-11-01T17:30:48.8580568Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:30:48.8583351Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-11-01 17:30:48.831638] 2024-11-01T17:46:04.7435663Z 2024-11-01T17:46:04.7438164Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_2cf17448ffe8b12a_.log 2024-11-01T17:46:04.7515964Z Running 58 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-11-01T17:46:04.7569304Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-11-01T17:46:04.7571721Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup 2024-11-01T17:46:04.7573997Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context 2024-11-01T17:46:04.7576761Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-11-01T17:46:04.7578867Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-11-01T17:46:04.7581078Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-11-01T17:46:04.7583482Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-11-01T17:46:04.7585866Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-11-01T17:46:04.7588393Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-11-01T17:46:04.7591060Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-11-01T17:46:04.7593476Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-11-01T17:46:04.7595834Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-11-01T17:46:04.7598362Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-11-01T17:46:04.7601137Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-11-01T17:46:04.7604009Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-11-01T17:46:04.7606771Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-11-01T17:46:04.7609381Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-11-01T17:46:04.7612053Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-11-01T17:46:04.7614818Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-11-01T17:46:04.7617559Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-11-01T17:46:04.7619924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-11-01T17:46:04.7622482Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-11-01T17:46:04.7625006Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-11-01T17:46:04.7627610Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-11-01T17:46:04.7630173Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-11-01T17:46:04.7632823Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-11-01T17:46:04.7635315Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-11-01T17:46:04.7637754Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-11-01T17:46:04.7640473Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-11-01T17:46:04.7642741Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-11-01T17:46:04.7644837Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-11-01T17:46:04.7646812Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-11-01T17:46:04.7648916Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-11-01T17:46:04.7651085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-11-01T17:46:04.7653186Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-11-01T17:46:04.7655384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-11-01T17:46:04.7657686Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-11-01T17:46:04.7659660Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-11-01T17:46:04.7661536Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-11-01T17:46:04.7663802Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-11-01T17:46:04.7666560Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-11-01T17:46:04.7669333Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-11-01T17:46:04.7672151Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-11-01T17:46:04.7674733Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-11-01T17:46:04.7676938Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-11-01T17:46:04.7679072Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup 2024-11-01T17:46:04.7681063Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup 2024-11-01T17:46:04.7683510Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0 2024-11-01T17:46:04.7685876Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1 2024-11-01T17:46:04.7688347Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-11-01T17:46:04.7690915Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-11-01T17:46:04.7693399Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-11-01T17:46:04.7695888Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-11-01T17:46:04.7698777Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-11-01T17:46:04.7701758Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-11-01T17:46:04.7704326Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-11-01T17:46:04.7706468Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-11-01T17:46:04.7708469Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-11-01T17:46:04.7709595Z 2024-11-01T17:46:06.0854200Z Running test batch 'tests to run' cost 3387.72 seconds 2024-11-01T17:46:07.2349350Z 2024-11-01T17:46:07.2349770Z real 56m35.661s 2024-11-01T17:46:07.2350433Z user 77m33.658s 2024-11-01T17:46:07.2351009Z sys 100m12.767s 2024-11-01T17:46:07.2351567Z + assert_git_not_dirty 2024-11-01T17:46:07.2352642Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-11-01T17:46:07.2353662Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-11-01T17:46:07.2354623Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-11-01T17:46:07.2355385Z + [[ 1 == 1 ]] 2024-11-01T17:46:07.2356044Z + echo 'Testing distributed C++ tests' 2024-11-01T17:46:07.2356805Z Testing distributed C++ tests 2024-11-01T17:46:07.2364272Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:07.2396936Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:07.2424941Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:07.2426721Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:07.2428208Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-11-01T17:46:07.3708738Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:46:07.3710713Z import pkg_resources 2024-11-01T17:46:13.6940904Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-11-01T17:46:13.7063475Z Found test times from artifacts 2024-11-01T17:46:13.7576975Z Found test times from artifacts 2024-11-01T17:46:13.7588424Z Running all tests 2024-11-01T17:46:13.7592660Z Running parallel tests on 8 processes 2024-11-01T17:46:13.7593511Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:46:13.7594262Z Serial tests (1): 2024-11-01T17:46:13.7594871Z cpp/FileStoreTest 1/1 2024-11-01T17:46:13.7595502Z Parallel tests (0): 2024-11-01T17:46:13.7596127Z Name: excluded (est. time: 0.0min) 2024-11-01T17:46:13.7596814Z Serial tests (0): 2024-11-01T17:46:13.7597379Z Parallel tests (0): 2024-11-01T17:46:13.7733449Z Running cpp/FileStoreTest 1/1 ... [2024-11-01 17:46:13.773022] 2024-11-01T17:46:13.7734510Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:46:13.7742860Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-3fdcf0b8f1ca5632.xml', '-x', '--reruns=2'] ... [2024-11-01 17:46:13.773812] 2024-11-01T17:46:16.8952572Z 2024-11-01T17:46:16.8954259Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_1678cda2f81e4c81_.log 2024-11-01T17:46:16.8955288Z 2024-11-01T17:46:20.8943726Z Running test batch 'tests to run' cost 7.13 seconds 2024-11-01T17:46:21.8964840Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-11-01T17:46:22.0229125Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:46:22.0231366Z import pkg_resources 2024-11-01T17:46:28.2779450Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-11-01T17:46:28.2896871Z Found test times from artifacts 2024-11-01T17:46:28.3400335Z Found test times from artifacts 2024-11-01T17:46:28.3414602Z Running all tests 2024-11-01T17:46:28.3418946Z Running parallel tests on 8 processes 2024-11-01T17:46:28.3419784Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:46:28.3420523Z Serial tests (1): 2024-11-01T17:46:28.3421138Z cpp/HashStoreTest 1/1 2024-11-01T17:46:28.3421786Z Parallel tests (0): 2024-11-01T17:46:28.3422407Z Name: excluded (est. time: 0.0min) 2024-11-01T17:46:28.3423096Z Serial tests (0): 2024-11-01T17:46:28.3423659Z Parallel tests (0): 2024-11-01T17:46:28.3554813Z Running cpp/HashStoreTest 1/1 ... [2024-11-01 17:46:28.355122] 2024-11-01T17:46:28.3555828Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:46:28.3561821Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-47b81cce572569d1.xml', '-x', '--reruns=2'] ... [2024-11-01 17:46:28.355677] 2024-11-01T17:46:31.1760190Z 2024-11-01T17:46:31.1762617Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_dc8821ebefccd44b_.log 2024-11-01T17:46:31.1764275Z 2024-11-01T17:46:35.0320297Z Running test batch 'tests to run' cost 6.69 seconds 2024-11-01T17:46:35.9921188Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-11-01T17:46:36.1186892Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:46:36.1188938Z import pkg_resources 2024-11-01T17:46:42.4481608Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-11-01T17:46:42.4610187Z Found test times from artifacts 2024-11-01T17:46:42.5178304Z Found test times from artifacts 2024-11-01T17:46:42.5193683Z Running all tests 2024-11-01T17:46:42.5197946Z Running parallel tests on 8 processes 2024-11-01T17:46:42.5198760Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:46:42.5200143Z Serial tests (1): 2024-11-01T17:46:42.5200758Z cpp/TCPStoreTest 1/1 2024-11-01T17:46:42.5201384Z Parallel tests (0): 2024-11-01T17:46:42.5201997Z Name: excluded (est. time: 0.0min) 2024-11-01T17:46:42.5202666Z Serial tests (0): 2024-11-01T17:46:42.5203223Z Parallel tests (0): 2024-11-01T17:46:42.5334587Z Running cpp/TCPStoreTest 1/1 ... [2024-11-01 17:46:42.533174] 2024-11-01T17:46:42.5335608Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:46:42.5342172Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5b7244a6ab98ee2f.xml', '-x', '--reruns=2'] ... [2024-11-01 17:46:42.533787] 2024-11-01T17:46:55.3196336Z 2024-11-01T17:46:55.3198400Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_4307294b6a1f2b1a_.log 2024-11-01T17:46:55.3200262Z 2024-11-01T17:46:56.5120860Z Running test batch 'tests to run' cost 13.99 seconds 2024-11-01T17:46:57.4739873Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-11-01T17:46:57.4740739Z + [[ 1 == 1 ]] 2024-11-01T17:46:57.4741270Z + test_rpc 2024-11-01T17:46:57.4741877Z + echo 'Testing RPC C++ tests' 2024-11-01T17:46:57.4742550Z Testing RPC C++ tests 2024-11-01T17:46:57.4748804Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:57.4791858Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:57.4827358Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-11-01T17:46:57.4828819Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-11-01T17:46:57.6102998Z /var/lib/jenkins/pytorch/test/run_test.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-11-01T17:46:57.6105046Z import pkg_resources 2024-11-01T17:47:03.9608682Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-11-01T17:47:03.9726259Z Found test times from artifacts 2024-11-01T17:47:04.0242044Z Found test times from artifacts 2024-11-01T17:47:04.0258391Z Running all tests 2024-11-01T17:47:04.0264232Z Running parallel tests on 8 processes 2024-11-01T17:47:04.0265134Z Name: tests to run (est. time: 0.0min) 2024-11-01T17:47:04.0265872Z Serial tests (1): 2024-11-01T17:47:04.0266467Z cpp/test_cpp_rpc 1/1 2024-11-01T17:47:04.0267085Z Parallel tests (0): 2024-11-01T17:47:04.0267700Z Name: excluded (est. time: 0.0min) 2024-11-01T17:47:04.0271369Z Serial tests (0): 2024-11-01T17:47:04.0272134Z Parallel tests (0): 2024-11-01T17:47:04.0402879Z Running cpp/test_cpp_rpc 1/1 ... [2024-11-01 17:47:04.039972] 2024-11-01T17:47:04.0403919Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-11-01T17:47:04.0409978Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-c9f316e4f72c47f4.xml', '-x', '--reruns=2'] ... [2024-11-01 17:47:04.040573] 2024-11-01T17:47:12.8700526Z 2024-11-01T17:47:12.8703796Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_cd5a02d8fb40506b_.log 2024-11-01T17:47:12.8705518Z 2024-11-01T17:47:14.2575234Z Running test batch 'tests to run' cost 10.23 seconds 2024-11-01T17:47:15.2422088Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-11-01T17:47:15.2424213Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-11-01T17:47:15.2426663Z docker exec -t "7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-11-01T17:47:15.2468410Z shell: /usr/bin/bash -e {0} 2024-11-01T17:47:15.2469059Z env: 2024-11-01T17:47:15.2469562Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:15.2470340Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:15.2471725Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:15.2472990Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:15.2474172Z AWS_REGION: us-east-1 2024-11-01T17:47:15.2474908Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:15.2475890Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:15.2488445Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:15.2489473Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:15.2490578Z ##[endgroup] 2024-11-01T17:47:15.3592091Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-11-01T17:47:15.4061677Z ##[group]Run cat test/**/*_toprint.log || true 2024-11-01T17:47:15.4062673Z cat test/**/*_toprint.log || true 2024-11-01T17:47:15.4106816Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:47:15.4107739Z env: 2024-11-01T17:47:15.4108235Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:15.4109025Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:15.4110397Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:15.4111705Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:15.4112379Z AWS_REGION: us-east-1 2024-11-01T17:47:15.4113140Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:15.4114079Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:15.4126701Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:15.4127737Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:15.4128853Z ##[endgroup] 2024-11-01T17:47:15.4372034Z cat: 'test/**/*_toprint.log': No such file or directory 2024-11-01T17:47:15.4440317Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:47:15.4441211Z kill "$MONITOR_SCRIPT_PID" 2024-11-01T17:47:15.4482012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:47:15.4482918Z env: 2024-11-01T17:47:15.4483421Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:15.4484194Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:15.4485595Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:15.4486891Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:15.4487565Z AWS_REGION: us-east-1 2024-11-01T17:47:15.4488315Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:15.4489265Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:15.4502185Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:15.4503242Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:15.4504392Z MONITOR_SCRIPT_PID: 2123009 2024-11-01T17:47:15.4505036Z ##[endgroup] 2024-11-01T17:47:15.4881722Z Prepare all required actions 2024-11-01T17:47:15.4882506Z Getting action download info 2024-11-01T17:47:15.9694839Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-11-01T17:47:16.5300207Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-11-01T17:47:16.5300654Z with: 2024-11-01T17:47:16.5300904Z use-gha: true 2024-11-01T17:47:16.5301311Z file-suffix: test-distributed-1-1-linux.rocm.gpu_32397140858 2024-11-01T17:47:16.5301824Z s3-bucket: gha-artifacts 2024-11-01T17:47:16.5302131Z env: 2024-11-01T17:47:16.5302393Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:16.5302784Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:16.5303463Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:16.5304088Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:16.5304425Z AWS_REGION: us-east-1 2024-11-01T17:47:16.5304854Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:16.5305331Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:16.5311392Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:16.5311911Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:16.5312466Z ##[endgroup] 2024-11-01T17:47:16.5395620Z ##[group]Run actions/upload-artifact@v3 2024-11-01T17:47:16.5396007Z with: 2024-11-01T17:47:16.5396513Z name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip 2024-11-01T17:47:16.5397119Z retention-days: 14 2024-11-01T17:47:16.5397692Z if-no-files-found: warn 2024-11-01T17:47:16.5398010Z path: test/**/*.json 2024-11-01T17:47:16.5398323Z include-hidden-files: false 2024-11-01T17:47:16.5398644Z env: 2024-11-01T17:47:16.5398893Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:16.5399278Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:16.5399960Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:16.5400606Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:16.5400960Z AWS_REGION: us-east-1 2024-11-01T17:47:16.5401332Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:16.5401803Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:16.5407911Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:16.5408428Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:16.5408979Z ##[endgroup] 2024-11-01T17:47:17.1059941Z With the provided path, there will be 10 files uploaded 2024-11-01T17:47:17.1063325Z Starting artifact upload 2024-11-01T17:47:17.1066348Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-11-01T17:47:17.1068880Z Artifact name is valid! 2024-11-01T17:47:17.2947436Z Container for artifact "test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip" successfully created. Starting upload of file(s) 2024-11-01T17:47:17.8289531Z Total size of all the files uploaded is 40956 bytes 2024-11-01T17:47:17.8290770Z File upload process has finished. Finalizing the artifact upload 2024-11-01T17:47:17.8689084Z Artifact has been finalized. All files have been successfully uploaded! 2024-11-01T17:47:17.8690056Z 2024-11-01T17:47:17.8690727Z The raw size of all the files that were specified for upload is 359866 bytes 2024-11-01T17:47:17.8693018Z The size of all the files that were uploaded is 40956 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-11-01T17:47:17.8694672Z 2024-11-01T17:47:17.8696751Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-11-01T17:47:17.8698642Z 2024-11-01T17:47:17.8699963Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip has been successfully uploaded! 2024-11-01T17:47:17.8956273Z ##[group]Run actions/upload-artifact@v3 2024-11-01T17:47:17.8957054Z with: 2024-11-01T17:47:17.8958077Z name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip 2024-11-01T17:47:17.8959311Z retention-days: 14 2024-11-01T17:47:17.8959913Z if-no-files-found: ignore 2024-11-01T17:47:17.8960584Z path: test/**/*.xml test/**/*.csv 2024-11-01T17:47:17.8961321Z include-hidden-files: false 2024-11-01T17:47:17.8961937Z env: 2024-11-01T17:47:17.8962410Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:17.8963192Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:17.8964573Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:17.8965835Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:17.8966504Z AWS_REGION: us-east-1 2024-11-01T17:47:17.8967225Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:17.8968170Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:17.8981173Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:17.8982240Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:17.8983365Z ##[endgroup] 2024-11-01T17:47:18.4682388Z With the provided path, there will be 229 files uploaded 2024-11-01T17:47:18.4683839Z Starting artifact upload 2024-11-01T17:47:18.4686438Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-11-01T17:47:18.4689439Z Artifact name is valid! 2024-11-01T17:47:18.6622936Z Container for artifact "test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip" successfully created. Starting upload of file(s) 2024-11-01T17:47:26.8508492Z Total size of all the files uploaded is 56530 bytes 2024-11-01T17:47:26.8509743Z File upload process has finished. Finalizing the artifact upload 2024-11-01T17:47:26.8925621Z Artifact has been finalized. All files have been successfully uploaded! 2024-11-01T17:47:26.8926465Z 2024-11-01T17:47:26.8927059Z The raw size of all the files that were specified for upload is 86981 bytes 2024-11-01T17:47:26.8929216Z The size of all the files that were uploaded is 56530 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-11-01T17:47:26.8930752Z 2024-11-01T17:47:26.8932564Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-11-01T17:47:26.8934395Z 2024-11-01T17:47:26.8935675Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip has been successfully uploaded! 2024-11-01T17:47:26.9237456Z ##[group]Run actions/upload-artifact@v3 2024-11-01T17:47:26.9238344Z with: 2024-11-01T17:47:26.9239324Z name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip 2024-11-01T17:47:26.9240534Z retention-days: 14 2024-11-01T17:47:26.9241242Z if-no-files-found: ignore 2024-11-01T17:47:26.9241984Z path: usage_log.txt test/**/*.log 2024-11-01T17:47:26.9242837Z include-hidden-files: false 2024-11-01T17:47:26.9243558Z env: 2024-11-01T17:47:26.9244126Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:26.9244957Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:26.9246392Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:26.9247730Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:26.9248508Z AWS_REGION: us-east-1 2024-11-01T17:47:26.9249331Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:26.9250286Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:26.9263280Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:26.9264412Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:26.9265635Z ##[endgroup] 2024-11-01T17:47:27.5008350Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-11-01T17:47:27.5012693Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-11-01T17:47:27.5014723Z With the provided path, there will be 9 files uploaded 2024-11-01T17:47:27.5016206Z Starting artifact upload 2024-11-01T17:47:27.5019497Z For more detailed logs during the artifact upload process, enable step-debugging: https://docs.github.com/actions/monitoring-and-troubleshooting-workflows/enabling-debug-logging#enabling-step-debug-logging 2024-11-01T17:47:27.5022059Z Artifact name is valid! 2024-11-01T17:47:27.6869114Z Container for artifact "logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip" successfully created. Starting upload of file(s) 2024-11-01T17:47:28.2679950Z Total size of all the files uploaded is 281728 bytes 2024-11-01T17:47:28.2681188Z File upload process has finished. Finalizing the artifact upload 2024-11-01T17:47:28.3201110Z Artifact has been finalized. All files have been successfully uploaded! 2024-11-01T17:47:28.3201984Z 2024-11-01T17:47:28.3202592Z The raw size of all the files that were specified for upload is 9715543 bytes 2024-11-01T17:47:28.3204777Z The size of all the files that were uploaded is 281728 bytes. This takes into account any gzip compression used to reduce the upload size, time and storage 2024-11-01T17:47:28.3206321Z 2024-11-01T17:47:28.3208144Z Note: The size of downloaded zips can differ significantly from the reported size. For more information see: https://github.com/actions/upload-artifact#zipped-artifact-downloads 2024-11-01T17:47:28.3210651Z 2024-11-01T17:47:28.3211855Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu_32397140858.zip has been successfully uploaded! 2024-11-01T17:47:28.3472431Z ##[group]Run # shellcheck disable=SC2156 2024-11-01T17:47:28.3473354Z # shellcheck disable=SC2156 2024-11-01T17:47:28.3474800Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-11-01T17:47:28.3516689Z shell: /usr/bin/bash -e {0} 2024-11-01T17:47:28.3517359Z env: 2024-11-01T17:47:28.3517866Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:28.3518661Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:28.3520062Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:28.3521333Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:28.3522014Z AWS_REGION: us-east-1 2024-11-01T17:47:28.3522776Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:28.3523720Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:28.3536559Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:28.3537669Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:28.3538810Z ##[endgroup] 2024-11-01T17:47:28.7205166Z Prepare all required actions 2024-11-01T17:47:28.7206003Z Getting action download info 2024-11-01T17:47:28.7274594Z ##[group]Run ./.github/actions/teardown-rocm 2024-11-01T17:47:28.7275368Z env: 2024-11-01T17:47:28.7275864Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:28.7276651Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:28.7278034Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:28.7279298Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:28.7279988Z AWS_REGION: us-east-1 2024-11-01T17:47:28.7280745Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:28.7281809Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:28.7294660Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:28.7295729Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:28.7296999Z ##[endgroup] 2024-11-01T17:47:28.7332682Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:47:28.7334038Z # ignore expansion of "docker ps -q" since it could be empty 2024-11-01T17:47:28.7335091Z # shellcheck disable=SC2046 2024-11-01T17:47:28.7335934Z docker stop $(docker ps -q) || true 2024-11-01T17:47:28.7337181Z # Prune all stopped containers. 2024-11-01T17:47:28.7337990Z docker container prune -f 2024-11-01T17:47:28.7377034Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:47:28.7377817Z env: 2024-11-01T17:47:28.7378262Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:28.7378940Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:28.7380141Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:28.7381271Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:28.7381972Z AWS_REGION: us-east-1 2024-11-01T17:47:28.7382726Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:28.7383692Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:28.7395511Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:28.7396422Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:28.7397391Z ##[endgroup] 2024-11-01T17:47:29.5395463Z 7da8eb8cf0fc 2024-11-01T17:47:36.4756191Z Deleted Containers: 2024-11-01T17:47:36.4757270Z 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:36.4758084Z 2024-11-01T17:47:36.4758374Z Total reclaimed space: 8.226GB 2024-11-01T17:47:36.4885060Z Prepare all required actions 2024-11-01T17:47:36.4960768Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-11-01T17:47:36.4961642Z with: 2024-11-01T17:47:36.4962181Z diskspace-cutoff: 70 2024-11-01T17:47:36.4962779Z env: 2024-11-01T17:47:36.4963800Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:36.4964636Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:36.4966127Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:36.4967507Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:36.4968224Z AWS_REGION: us-east-1 2024-11-01T17:47:36.4968991Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:36.4969995Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:36.4983422Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:36.4984544Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:36.4985720Z ##[endgroup] 2024-11-01T17:47:36.5023316Z ##[group]Run set -ex 2024-11-01T17:47:36.5023878Z set -ex 2024-11-01T17:47:36.5024391Z diskspace_cutoff=70 2024-11-01T17:47:36.5025197Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-11-01T17:47:36.5026513Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-11-01T17:47:36.5028580Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-11-01T17:47:36.5031114Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-11-01T17:47:36.5032149Z  docker system prune -af 2024-11-01T17:47:36.5033370Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-11-01T17:47:36.5034690Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-11-01T17:47:36.5036072Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-11-01T17:47:36.5037254Z  echo "$msg" 2024-11-01T17:47:36.5037822Z  exit 1 2024-11-01T17:47:36.5038346Z  else 2024-11-01T17:47:36.5038982Z  difference=$((diskspace - diskspace_new)) 2024-11-01T17:47:36.5039927Z  echo "Diskspace saved: $difference percent" 2024-11-01T17:47:36.5040687Z  fi 2024-11-01T17:47:36.5041142Z fi 2024-11-01T17:47:36.5079123Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-11-01T17:47:36.5080085Z env: 2024-11-01T17:47:36.5080613Z GIT_DEFAULT_BRANCH: main 2024-11-01T17:47:36.5081369Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-11-01T17:47:36.5082621Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-11-01T17:47:36.5083749Z AWS_DEFAULT_REGION: us-east-1 2024-11-01T17:47:36.5084352Z AWS_REGION: us-east-1 2024-11-01T17:47:36.5084999Z AWS_ACCESS_KEY_ID: *** 2024-11-01T17:47:36.5085818Z AWS_SECRET_ACCESS_KEY: *** 2024-11-01T17:47:36.5097012Z AWS_SESSION_TOKEN: *** 2024-11-01T17:47:36.5097935Z CONTAINER_NAME: 7da8eb8cf0fc619d7d27c497f6ea835305e46895b1c64b173c2f854c585eb0f5 2024-11-01T17:47:36.5098928Z ##[endgroup] 2024-11-01T17:47:36.5178443Z + diskspace_cutoff=70 2024-11-01T17:47:36.5187136Z ++ docker info -f '{{.DockerRootDir}}' 2024-11-01T17:47:36.6137563Z + docker_root_dir=/media/4TB/docker-rootless 2024-11-01T17:47:36.6152430Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-11-01T17:47:36.6156020Z ++ sed -n 2p 2024-11-01T17:47:36.6160808Z ++ sed s/%// 2024-11-01T17:47:36.6163338Z ++ sed 's/ //' 2024-11-01T17:47:36.6206527Z + diskspace=26 2024-11-01T17:47:36.6208304Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-11-01T17:47:36.6210032Z + [[ 26 -ge 70 ]] 2024-11-01T17:47:36.6317795Z Post job cleanup. 2024-11-01T17:47:36.6396114Z Post job cleanup. 2024-11-01T17:47:36.7128905Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-11-01T17:47:36.7724105Z Post job cleanup. 2024-11-01T17:47:36.9470328Z Post job cleanup. 2024-11-01T17:47:36.9577651Z Post job cleanup. 2024-11-01T17:47:36.9667954Z Post job cleanup. 2024-11-01T17:47:37.0753972Z [command]/usr/bin/git version 2024-11-01T17:47:37.0805962Z git version 2.34.1 2024-11-01T17:47:37.0849827Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/57f4c295-a331-4865-95ac-193a6a9609fc' before making global git config changes 2024-11-01T17:47:37.0852137Z Adding repository directory to the temporary git global config as a safe directory 2024-11-01T17:47:37.0854217Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-11-01T17:47:37.0912403Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-11-01T17:47:37.0976168Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-11-01T17:47:37.1638683Z Entering 'android/libs/fbjni' 2024-11-01T17:47:37.1761723Z Entering 'third_party/FP16' 2024-11-01T17:47:37.1881620Z Entering 'third_party/FXdiv' 2024-11-01T17:47:37.2000584Z Entering 'third_party/NNPACK' 2024-11-01T17:47:37.2120773Z Entering 'third_party/NVTX' 2024-11-01T17:47:37.2241298Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:47:37.2360015Z Entering 'third_party/XNNPACK' 2024-11-01T17:47:37.2515062Z Entering 'third_party/benchmark' 2024-11-01T17:47:37.2633945Z Entering 'third_party/composable_kernel' 2024-11-01T17:47:37.2762971Z Entering 'third_party/cpp-httplib' 2024-11-01T17:47:37.2879982Z Entering 'third_party/cpuinfo' 2024-11-01T17:47:37.2999124Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:47:37.3116625Z Entering 'third_party/cutlass' 2024-11-01T17:47:37.3252594Z Entering 'third_party/eigen' 2024-11-01T17:47:37.3372989Z Entering 'third_party/fbgemm' 2024-11-01T17:47:37.3486910Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:47:37.3597952Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:47:37.3710639Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:47:37.3839339Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:47:37.3952496Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:47:37.4067860Z Entering 'third_party/flatbuffers' 2024-11-01T17:47:37.4190323Z Entering 'third_party/fmt' 2024-11-01T17:47:37.4307372Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:47:37.4422598Z Entering 'third_party/gloo' 2024-11-01T17:47:37.4540444Z Entering 'third_party/googletest' 2024-11-01T17:47:37.4660419Z Entering 'third_party/ideep' 2024-11-01T17:47:37.4770824Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:47:37.4900868Z Entering 'third_party/ittapi' 2024-11-01T17:47:37.5016488Z Entering 'third_party/kineto' 2024-11-01T17:47:37.5132283Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:47:37.5241628Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:47:37.5356601Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:47:37.5467372Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:47:37.5581988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:47:37.5689617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:47:37.5807559Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:47:37.5920775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:47:37.6032857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:47:37.6147068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:47:37.6264312Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:47:37.6375874Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:47:37.6493968Z Entering 'third_party/mimalloc' 2024-11-01T17:47:37.6610751Z Entering 'third_party/nccl/nccl' 2024-11-01T17:47:37.6726016Z Entering 'third_party/nlohmann' 2024-11-01T17:47:37.6846634Z Entering 'third_party/onnx' 2024-11-01T17:47:37.6994259Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:47:37.7119327Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:47:37.7236282Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:47:37.7347565Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:47:37.7459542Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:47:37.7568415Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:47:37.7682440Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:47:37.7793428Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:47:37.7902418Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:47:37.8010382Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:47:37.8126602Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:47:37.8243785Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:47:37.8402496Z Entering 'third_party/pocketfft' 2024-11-01T17:47:37.8518344Z Entering 'third_party/protobuf' 2024-11-01T17:47:37.8639950Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:47:37.8753292Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:47:37.8874117Z Entering 'third_party/psimd' 2024-11-01T17:47:37.8989959Z Entering 'third_party/pthreadpool' 2024-11-01T17:47:37.9104309Z Entering 'third_party/pybind11' 2024-11-01T17:47:37.9220314Z Entering 'third_party/python-peachpy' 2024-11-01T17:47:37.9337564Z Entering 'third_party/sleef' 2024-11-01T17:47:37.9452548Z Entering 'third_party/tensorpipe' 2024-11-01T17:47:37.9567262Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:47:37.9678832Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:47:37.9788472Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:47:37.9899478Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:47:38.0004677Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:47:38.0168137Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-11-01T17:47:38.0216982Z http.https://github.com/.extraheader 2024-11-01T17:47:38.0238616Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-11-01T17:47:38.0308970Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-11-01T17:47:38.0932987Z Entering 'android/libs/fbjni' 2024-11-01T17:47:38.0997931Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1089740Z Entering 'third_party/FP16' 2024-11-01T17:47:38.1154539Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1247336Z Entering 'third_party/FXdiv' 2024-11-01T17:47:38.1310955Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1400011Z Entering 'third_party/NNPACK' 2024-11-01T17:47:38.1462040Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1552768Z Entering 'third_party/NVTX' 2024-11-01T17:47:38.1615631Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1707886Z Entering 'third_party/VulkanMemoryAllocator' 2024-11-01T17:47:38.1769796Z http.https://github.com/.extraheader 2024-11-01T17:47:38.1863562Z Entering 'third_party/XNNPACK' 2024-11-01T17:47:38.1925802Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2055487Z Entering 'third_party/benchmark' 2024-11-01T17:47:38.2117916Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2209433Z Entering 'third_party/composable_kernel' 2024-11-01T17:47:38.2272402Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2375293Z Entering 'third_party/cpp-httplib' 2024-11-01T17:47:38.2438400Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2530292Z Entering 'third_party/cpuinfo' 2024-11-01T17:47:38.2593686Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2684951Z Entering 'third_party/cudnn_frontend' 2024-11-01T17:47:38.2746465Z http.https://github.com/.extraheader 2024-11-01T17:47:38.2837356Z Entering 'third_party/cutlass' 2024-11-01T17:47:38.2902525Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3009385Z Entering 'third_party/eigen' 2024-11-01T17:47:38.3072143Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3167937Z Entering 'third_party/fbgemm' 2024-11-01T17:47:38.3231233Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3320127Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-11-01T17:47:38.3378264Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3468419Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-11-01T17:47:38.3528921Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3619958Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-11-01T17:47:38.3680391Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3784455Z Entering 'third_party/fbgemm/third_party/googletest' 2024-11-01T17:47:38.3843888Z http.https://github.com/.extraheader 2024-11-01T17:47:38.3931414Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-11-01T17:47:38.3991437Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4086652Z Entering 'third_party/flatbuffers' 2024-11-01T17:47:38.4150908Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4248844Z Entering 'third_party/fmt' 2024-11-01T17:47:38.4311637Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4400782Z Entering 'third_party/gemmlowp/gemmlowp' 2024-11-01T17:47:38.4463233Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4554825Z Entering 'third_party/gloo' 2024-11-01T17:47:38.4617060Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4708950Z Entering 'third_party/googletest' 2024-11-01T17:47:38.4770987Z http.https://github.com/.extraheader 2024-11-01T17:47:38.4864141Z Entering 'third_party/ideep' 2024-11-01T17:47:38.4927446Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5013207Z Entering 'third_party/ideep/mkl-dnn' 2024-11-01T17:47:38.5072178Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5179765Z Entering 'third_party/ittapi' 2024-11-01T17:47:38.5241640Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5330626Z Entering 'third_party/kineto' 2024-11-01T17:47:38.5393899Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5481667Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-11-01T17:47:38.5540133Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5626606Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-11-01T17:47:38.5687978Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5780507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-11-01T17:47:38.5839018Z http.https://github.com/.extraheader 2024-11-01T17:47:38.5927965Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-11-01T17:47:38.5986227Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6075472Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-11-01T17:47:38.6134387Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6219325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-11-01T17:47:38.6278472Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6375082Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-11-01T17:47:38.6435161Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6525408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-11-01T17:47:38.6583499Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6673355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-11-01T17:47:38.6731225Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6823227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-11-01T17:47:38.6883520Z http.https://github.com/.extraheader 2024-11-01T17:47:38.6978209Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-11-01T17:47:38.7036570Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7127199Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-11-01T17:47:38.7186470Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7280736Z Entering 'third_party/mimalloc' 2024-11-01T17:47:38.7343925Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7436333Z Entering 'third_party/nccl/nccl' 2024-11-01T17:47:38.7499180Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7589751Z Entering 'third_party/nlohmann' 2024-11-01T17:47:38.7652074Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7746886Z Entering 'third_party/onnx' 2024-11-01T17:47:38.7808437Z http.https://github.com/.extraheader 2024-11-01T17:47:38.7928764Z Entering 'third_party/onnx/third_party/pybind11' 2024-11-01T17:47:38.7990788Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8091367Z Entering 'third_party/opentelemetry-cpp' 2024-11-01T17:47:38.8153186Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8243824Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-11-01T17:47:38.8303389Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8391913Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-11-01T17:47:38.8450875Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8538358Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-11-01T17:47:38.8597900Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8684257Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-11-01T17:47:38.8743792Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8835214Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-11-01T17:47:38.8893111Z http.https://github.com/.extraheader 2024-11-01T17:47:38.8983713Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-11-01T17:47:38.9042373Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9128039Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-11-01T17:47:38.9188770Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9271307Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-11-01T17:47:38.9331094Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9436808Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-11-01T17:47:38.9495509Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9591737Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-11-01T17:47:38.9650801Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9785275Z Entering 'third_party/pocketfft' 2024-11-01T17:47:38.9847283Z http.https://github.com/.extraheader 2024-11-01T17:47:38.9936488Z Entering 'third_party/protobuf' 2024-11-01T17:47:38.9999193Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0091512Z Entering 'third_party/protobuf/third_party/benchmark' 2024-11-01T17:47:39.0150483Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0239594Z Entering 'third_party/protobuf/third_party/googletest' 2024-11-01T17:47:39.0298533Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0391172Z Entering 'third_party/psimd' 2024-11-01T17:47:39.0453804Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0543709Z Entering 'third_party/pthreadpool' 2024-11-01T17:47:39.0605962Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0695664Z Entering 'third_party/pybind11' 2024-11-01T17:47:39.0756368Z http.https://github.com/.extraheader 2024-11-01T17:47:39.0847125Z Entering 'third_party/python-peachpy' 2024-11-01T17:47:39.0909694Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1000343Z Entering 'third_party/sleef' 2024-11-01T17:47:39.1062472Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1151534Z Entering 'third_party/tensorpipe' 2024-11-01T17:47:39.1214568Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1300931Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-11-01T17:47:39.1359883Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1447819Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-11-01T17:47:39.1506486Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1593049Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-11-01T17:47:39.1650881Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1737570Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-11-01T17:47:39.1796102Z http.https://github.com/.extraheader 2024-11-01T17:47:39.1879495Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-11-01T17:47:39.1938570Z http.https://github.com/.extraheader 2024-11-01T17:47:39.2555874Z Cleaning up orphan processes