2024-12-18T03:33:26.7682512Z Current runner version: '2.321.0' 2024-12-18T03:33:26.7688748Z Runner name: 'gpu9489.jax.cs.cpe.ice.amd.com' 2024-12-18T03:33:26.7689654Z Runner group name: 'linux.rocm.gpu.group' 2024-12-18T03:33:26.7690675Z Machine name: 'gpu9489' 2024-12-18T03:33:26.7694081Z ##[group]GITHUB_TOKEN Permissions 2024-12-18T03:33:26.7695991Z Contents: read 2024-12-18T03:33:26.7696576Z Metadata: read 2024-12-18T03:33:26.7697142Z ##[endgroup] 2024-12-18T03:33:26.7699611Z Secret source: Actions 2024-12-18T03:33:26.7700233Z Prepare workflow directory 2024-12-18T03:33:27.3517477Z Prepare all required actions 2024-12-18T03:33:27.3553317Z Getting action download info 2024-12-18T03:33:27.7101176Z Download action repository 'pytorch/pytorch@release/2.6' (SHA:0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T03:33:33.2590592Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502) 2024-12-18T03:33:33.7901876Z Download action repository 'aws-actions/amazon-ecr-login@v2' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2024-12-18T03:33:34.2088805Z Download action repository 'pytorch/test-infra@release/2.6' (SHA:eb0adf5a84668865394af69e26428b32c8105c1c) 2024-12-18T03:33:35.1514695Z Download action repository 'actions/upload-artifact@v4' (SHA:6f51ac03b9356f520e9adb1b1b7802705f340c2b) 2024-12-18T03:33:35.7957107Z Getting action download info 2024-12-18T03:33:35.9403017Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-12-18T03:33:36.4175925Z Getting action download info 2024-12-18T03:33:36.5665126Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2024-12-18T03:33:37.0148513Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/release/2.6 (0cdf8b1d09254cfda66191d1bd01e3041c3c76f7) 2024-12-18T03:33:37.0150180Z ##[group] Inputs 2024-12-18T03:33:37.0150457Z build-environment: linux-focal-rocm6.2-py3.10 2024-12-18T03:33:37.0151644Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]} 2024-12-18T03:33:37.0153086Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:33:37.0153580Z sync-tag: 2024-12-18T03:33:37.0154209Z timeout-minutes: 300 2024-12-18T03:33:37.0154415Z tests-to-include: 2024-12-18T03:33:37.0154870Z disable-monitor: true 2024-12-18T03:33:37.0155077Z ##[endgroup] 2024-12-18T03:33:37.0155469Z Complete job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:33:37.1792647Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@release/2.6 2024-12-18T03:33:37.1793394Z with: 2024-12-18T03:33:37.1793575Z no-sudo: true 2024-12-18T03:33:37.1793766Z submodules: recursive 2024-12-18T03:33:37.1793953Z fetch-depth: 0 2024-12-18T03:33:37.1794265Z env: 2024-12-18T03:33:37.1794430Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:33:37.1794635Z ##[endgroup] 2024-12-18T03:33:37.1866520Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T03:33:37.1867296Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2024-12-18T03:33:37.1902666Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:33:37.1902967Z env: 2024-12-18T03:33:37.1903126Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:33:37.1903322Z ##[endgroup] 2024-12-18T03:33:37.2165039Z ##[group]Run retry () { 2024-12-18T03:33:37.2165299Z retry () { 2024-12-18T03:33:37.2165884Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-12-18T03:33:37.2166228Z } 2024-12-18T03:33:37.2166437Z echo "${GITHUB_WORKSPACE}" 2024-12-18T03:33:37.2166714Z if [ -z "${NO_SUDO}" ]; then 2024-12-18T03:33:37.2167018Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T03:33:37.2167307Z else 2024-12-18T03:33:37.2167528Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-12-18T03:33:37.2167805Z fi 2024-12-18T03:33:37.2168012Z mkdir "${GITHUB_WORKSPACE}" 2024-12-18T03:33:37.2209093Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:33:37.2209441Z env: 2024-12-18T03:33:37.2209637Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:33:37.2209864Z NO_SUDO: true 2024-12-18T03:33:37.2210070Z ##[endgroup] 2024-12-18T03:33:37.2304812Z /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:33:41.2703653Z ##[group]Run malfet/checkout@silent-checkout 2024-12-18T03:33:41.2704301Z with: 2024-12-18T03:33:41.2704743Z ref: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:33:41.2705314Z fetch-depth: 0 2024-12-18T03:33:41.2705727Z submodules: recursive 2024-12-18T03:33:41.2706178Z quiet-checkout: true 2024-12-18T03:33:41.2706634Z repository: pytorch/pytorch 2024-12-18T03:33:41.2707366Z token: *** 2024-12-18T03:33:41.2707767Z ssh-strict: true 2024-12-18T03:33:41.2708220Z persist-credentials: true 2024-12-18T03:33:41.2708691Z clean: true 2024-12-18T03:33:41.2709124Z sparse-checkout-cone-mode: true 2024-12-18T03:33:41.2709642Z lfs: false 2024-12-18T03:33:41.2710029Z set-safe-directory: true 2024-12-18T03:33:41.2710496Z env: 2024-12-18T03:33:41.2710862Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:33:41.2711312Z ##[endgroup] 2024-12-18T03:33:41.3920206Z Syncing repository: pytorch/pytorch 2024-12-18T03:33:41.3922685Z ##[group]Getting Git version info 2024-12-18T03:33:41.3923680Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-12-18T03:33:41.3924909Z [command]/usr/bin/git version 2024-12-18T03:33:41.3925397Z git version 2.34.1 2024-12-18T03:33:41.3927007Z ##[endgroup] 2024-12-18T03:33:41.3933898Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/ccde02ec-1a57-4cb0-8c02-57818414459c' before making global git config changes 2024-12-18T03:33:41.3935576Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T03:33:41.3936940Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:33:41.3966148Z Deleting the contents of '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2024-12-18T03:33:41.3970707Z ##[group]Initializing the repository 2024-12-18T03:33:41.3973468Z [command]/usr/bin/git init /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:33:41.4036653Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-12-18T03:33:41.4038032Z hint: is subject to change. To configure the initial branch name to use in all 2024-12-18T03:33:41.4039016Z hint: of your new repositories, which will suppress this warning, call: 2024-12-18T03:33:41.4039744Z hint: 2024-12-18T03:33:41.4040399Z hint: git config --global init.defaultBranch 2024-12-18T03:33:41.4041113Z hint: 2024-12-18T03:33:41.4041808Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-12-18T03:33:41.4042957Z hint: 'development'. The just-created branch can be renamed via this command: 2024-12-18T03:33:41.4043854Z hint: 2024-12-18T03:33:41.4044308Z hint: git branch -m 2024-12-18T03:33:41.4051124Z Initialized empty Git repository in /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/ 2024-12-18T03:33:41.4061177Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-12-18T03:33:41.4121968Z ##[endgroup] 2024-12-18T03:33:41.4122902Z ##[group]Disabling automatic garbage collection 2024-12-18T03:33:41.4125673Z [command]/usr/bin/git config --local gc.auto 0 2024-12-18T03:33:41.4180498Z ##[endgroup] 2024-12-18T03:33:41.4181364Z ##[group]Setting up auth 2024-12-18T03:33:41.4187517Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T03:33:41.4243038Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T03:33:41.4858087Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T03:33:41.4919149Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T03:33:41.5550141Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T03:33:41.5631252Z ##[endgroup] 2024-12-18T03:33:41.5632121Z ##[group]Fetching the repository 2024-12-18T03:33:41.5643373Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-12-18T03:33:47.8467897Z remote: Enumerating objects: 1055612 2024-12-18T03:33:47.8468669Z remote: Enumerating objects: 1056590, done. 2024-12-18T03:33:47.8469334Z remote: Counting objects: 0% (1/978) 2024-12-18T03:33:47.8469954Z remote: Counting objects: 1% (10/978) 2024-12-18T03:33:47.8470578Z remote: Counting objects: 2% (20/978) 2024-12-18T03:33:47.8471279Z remote: Counting objects: 3% (30/978) 2024-12-18T03:33:47.8471971Z remote: Counting objects: 4% (40/978) 2024-12-18T03:33:47.8472918Z remote: Counting objects: 5% (49/978) 2024-12-18T03:33:47.8473668Z remote: Counting objects: 6% (59/978) 2024-12-18T03:33:47.8474370Z remote: Counting objects: 7% (69/978) 2024-12-18T03:33:47.8475076Z remote: Counting objects: 8% (79/978) 2024-12-18T03:33:47.8475773Z remote: Counting objects: 9% (89/978) 2024-12-18T03:33:47.8476476Z remote: Counting objects: 10% (98/978) 2024-12-18T03:33:47.8477172Z remote: Counting objects: 11% (108/978) 2024-12-18T03:33:47.8477802Z remote: Counting objects: 12% (118/978) 2024-12-18T03:33:47.8478397Z remote: Counting objects: 13% (128/978) 2024-12-18T03:33:47.8478987Z remote: Counting objects: 14% (137/978) 2024-12-18T03:33:47.8479762Z remote: Counting objects: 15% (147/978) 2024-12-18T03:33:47.8480346Z remote: Counting objects: 16% (157/978) 2024-12-18T03:33:47.8480918Z remote: Counting objects: 17% (167/978) 2024-12-18T03:33:47.8481502Z remote: Counting objects: 18% (177/978) 2024-12-18T03:33:47.8482092Z remote: Counting objects: 19% (186/978) 2024-12-18T03:33:47.8483280Z remote: Counting objects: 20% (196/978) 2024-12-18T03:33:47.8484046Z remote: Counting objects: 21% (206/978) 2024-12-18T03:33:47.8484775Z remote: Counting objects: 22% (216/978) 2024-12-18T03:33:47.8485485Z remote: Counting objects: 23% (225/978) 2024-12-18T03:33:47.8486147Z remote: Counting objects: 24% (235/978) 2024-12-18T03:33:47.8486734Z remote: Counting objects: 25% (245/978) 2024-12-18T03:33:47.8487375Z remote: Counting objects: 26% (255/978) 2024-12-18T03:33:47.8488028Z remote: Counting objects: 27% (265/978) 2024-12-18T03:33:47.8488619Z remote: Counting objects: 28% (274/978) 2024-12-18T03:33:47.8489193Z remote: Counting objects: 29% (284/978) 2024-12-18T03:33:47.8489784Z remote: Counting objects: 30% (294/978) 2024-12-18T03:33:47.8490376Z remote: Counting objects: 31% (304/978) 2024-12-18T03:33:47.8490956Z remote: Counting objects: 32% (313/978) 2024-12-18T03:33:47.8491549Z remote: Counting objects: 33% (323/978) 2024-12-18T03:33:47.8492543Z remote: Counting objects: 34% (333/978) 2024-12-18T03:33:47.8493152Z remote: Counting objects: 35% (343/978) 2024-12-18T03:33:47.8493743Z remote: Counting objects: 36% (353/978) 2024-12-18T03:33:47.8494330Z remote: Counting objects: 37% (362/978) 2024-12-18T03:33:47.8494928Z remote: Counting objects: 38% (372/978) 2024-12-18T03:33:47.8495512Z remote: Counting objects: 39% (382/978) 2024-12-18T03:33:47.8496098Z remote: Counting objects: 40% (392/978) 2024-12-18T03:33:47.8496687Z remote: Counting objects: 41% (401/978) 2024-12-18T03:33:47.8497266Z remote: Counting objects: 42% (411/978) 2024-12-18T03:33:47.8497853Z remote: Counting objects: 43% (421/978) 2024-12-18T03:33:47.8498438Z remote: Counting objects: 44% (431/978) 2024-12-18T03:33:47.8499010Z remote: Counting objects: 45% (441/978) 2024-12-18T03:33:47.8499614Z remote: Counting objects: 46% (450/978) 2024-12-18T03:33:47.8500215Z remote: Counting objects: 47% (460/978) 2024-12-18T03:33:47.8500887Z remote: Counting objects: 48% (470/978) 2024-12-18T03:33:47.8501592Z remote: Counting objects: 49% (480/978) 2024-12-18T03:33:47.8502326Z remote: Counting objects: 50% (489/978) 2024-12-18T03:33:47.8503021Z remote: Counting objects: 51% (499/978) 2024-12-18T03:33:47.8503620Z remote: Counting objects: 52% (509/978) 2024-12-18T03:33:47.8504212Z remote: Counting objects: 53% (519/978) 2024-12-18T03:33:47.8504804Z remote: Counting objects: 54% (529/978) 2024-12-18T03:33:47.8505387Z remote: Counting objects: 55% (538/978) 2024-12-18T03:33:47.8505999Z remote: Counting objects: 56% (548/978) 2024-12-18T03:33:47.8506589Z remote: Counting objects: 57% (558/978) 2024-12-18T03:33:47.8507169Z remote: Counting objects: 58% (568/978) 2024-12-18T03:33:47.8507772Z remote: Counting objects: 59% (578/978) 2024-12-18T03:33:47.8508375Z remote: Counting objects: 60% (587/978) 2024-12-18T03:33:47.8508987Z remote: Counting objects: 61% (597/978) 2024-12-18T03:33:47.8509590Z remote: Counting objects: 62% (607/978) 2024-12-18T03:33:47.8510234Z remote: Counting objects: 63% (617/978) 2024-12-18T03:33:47.8510869Z remote: Counting objects: 64% (626/978) 2024-12-18T03:33:47.8511553Z remote: Counting objects: 65% (636/978) 2024-12-18T03:33:47.8512375Z remote: Counting objects: 66% (646/978) 2024-12-18T03:33:47.8513015Z remote: Counting objects: 67% (656/978) 2024-12-18T03:33:47.8513595Z remote: Counting objects: 68% (666/978) 2024-12-18T03:33:47.8514185Z remote: Counting objects: 69% (675/978) 2024-12-18T03:33:47.8514867Z remote: Counting objects: 70% (685/978) 2024-12-18T03:33:47.8515544Z remote: Counting objects: 71% (695/978) 2024-12-18T03:33:47.8516626Z remote: Counting objects: 72% (705/978) 2024-12-18T03:33:47.8517246Z remote: Counting objects: 73% (714/978) 2024-12-18T03:33:47.8517837Z remote: Counting objects: 74% (724/978) 2024-12-18T03:33:47.8518422Z remote: Counting objects: 75% (734/978) 2024-12-18T03:33:47.8519007Z remote: Counting objects: 76% (744/978) 2024-12-18T03:33:47.8519582Z remote: Counting objects: 77% (754/978) 2024-12-18T03:33:47.8520168Z remote: Counting objects: 78% (763/978) 2024-12-18T03:33:47.8520762Z remote: Counting objects: 79% (773/978) 2024-12-18T03:33:47.8521337Z remote: Counting objects: 80% (783/978) 2024-12-18T03:33:47.8522031Z remote: Counting objects: 81% (793/978) 2024-12-18T03:33:47.8522723Z remote: Counting objects: 82% (802/978) 2024-12-18T03:33:47.8523403Z remote: Counting objects: 83% (812/978) 2024-12-18T03:33:47.8524097Z remote: Counting objects: 84% (822/978) 2024-12-18T03:33:47.8524828Z remote: Counting objects: 85% (832/978) 2024-12-18T03:33:47.8525520Z remote: Counting objects: 86% (842/978) 2024-12-18T03:33:47.8526554Z remote: Counting objects: 87% (851/978) 2024-12-18T03:33:47.8527156Z remote: Counting objects: 88% (861/978) 2024-12-18T03:33:47.8527739Z remote: Counting objects: 89% (871/978) 2024-12-18T03:33:47.8528329Z remote: Counting objects: 90% (881/978) 2024-12-18T03:33:47.8528917Z remote: Counting objects: 91% (890/978) 2024-12-18T03:33:47.8529487Z remote: Counting objects: 92% (900/978) 2024-12-18T03:33:47.8530074Z remote: Counting objects: 93% (910/978) 2024-12-18T03:33:47.8530666Z remote: Counting objects: 94% (920/978) 2024-12-18T03:33:47.8531246Z remote: Counting objects: 95% (930/978) 2024-12-18T03:33:47.8531890Z remote: Counting objects: 96% (939/978) 2024-12-18T03:33:47.8532602Z remote: Counting objects: 97% (949/978) 2024-12-18T03:33:47.8533296Z remote: Counting objects: 98% (959/978) 2024-12-18T03:33:47.8533993Z remote: Counting objects: 99% (969/978) 2024-12-18T03:33:47.8534654Z remote: Counting objects: 100% (978/978) 2024-12-18T03:33:47.8535289Z remote: Counting objects: 100% (978/978), done. 2024-12-18T03:33:47.8578663Z remote: Compressing objects: 0% (1/395) 2024-12-18T03:33:47.8771167Z remote: Compressing objects: 1% (4/395) 2024-12-18T03:33:47.8977074Z remote: Compressing objects: 2% (8/395) 2024-12-18T03:33:47.9034002Z remote: Compressing objects: 3% (12/395) 2024-12-18T03:33:47.9141800Z remote: Compressing objects: 4% (16/395) 2024-12-18T03:33:47.9316559Z remote: Compressing objects: 5% (20/395) 2024-12-18T03:33:47.9781820Z remote: Compressing objects: 6% (24/395) 2024-12-18T03:33:48.0573756Z remote: Compressing objects: 7% (28/395) 2024-12-18T03:33:48.1352222Z remote: Compressing objects: 8% (32/395) 2024-12-18T03:33:48.1851062Z remote: Compressing objects: 9% (36/395) 2024-12-18T03:33:48.2352623Z remote: Compressing objects: 10% (40/395) 2024-12-18T03:33:48.2562298Z remote: Compressing objects: 11% (44/395) 2024-12-18T03:33:48.2643120Z remote: Compressing objects: 12% (48/395) 2024-12-18T03:33:48.2650663Z remote: Compressing objects: 13% (52/395) 2024-12-18T03:33:48.2660549Z remote: Compressing objects: 14% (56/395) 2024-12-18T03:33:48.2666659Z remote: Compressing objects: 15% (60/395) 2024-12-18T03:33:48.2670983Z remote: Compressing objects: 16% (64/395) 2024-12-18T03:33:48.2678569Z remote: Compressing objects: 17% (68/395) 2024-12-18T03:33:48.2679348Z remote: Compressing objects: 18% (72/395) 2024-12-18T03:33:48.2684793Z remote: Compressing objects: 19% (76/395) 2024-12-18T03:33:48.2685625Z remote: Compressing objects: 20% (79/395) 2024-12-18T03:33:48.2688823Z remote: Compressing objects: 21% (83/395) 2024-12-18T03:33:48.2696217Z remote: Compressing objects: 22% (87/395) 2024-12-18T03:33:48.2702260Z remote: Compressing objects: 23% (91/395) 2024-12-18T03:33:48.2708114Z remote: Compressing objects: 24% (95/395) 2024-12-18T03:33:48.2713205Z remote: Compressing objects: 25% (99/395) 2024-12-18T03:33:48.2720809Z remote: Compressing objects: 26% (103/395) 2024-12-18T03:33:48.2726195Z remote: Compressing objects: 27% (107/395) 2024-12-18T03:33:48.2730779Z remote: Compressing objects: 28% (111/395) 2024-12-18T03:33:48.2734993Z remote: Compressing objects: 29% (115/395) 2024-12-18T03:33:48.2744445Z remote: Compressing objects: 30% (119/395) 2024-12-18T03:33:48.2748305Z remote: Compressing objects: 31% (123/395) 2024-12-18T03:33:48.2751998Z remote: Compressing objects: 32% (127/395) 2024-12-18T03:33:48.2758363Z remote: Compressing objects: 33% (131/395) 2024-12-18T03:33:48.2770762Z remote: Compressing objects: 34% (135/395) 2024-12-18T03:33:48.2780391Z remote: Compressing objects: 35% (139/395) 2024-12-18T03:33:48.2781463Z remote: Compressing objects: 36% (143/395) 2024-12-18T03:33:48.2784010Z remote: Compressing objects: 37% (147/395) 2024-12-18T03:33:48.2792121Z remote: Compressing objects: 38% (151/395) 2024-12-18T03:33:48.2799223Z remote: Compressing objects: 39% (155/395) 2024-12-18T03:33:48.2800259Z remote: Compressing objects: 40% (158/395) 2024-12-18T03:33:48.2805177Z remote: Compressing objects: 41% (162/395) 2024-12-18T03:33:48.2809382Z remote: Compressing objects: 42% (166/395) 2024-12-18T03:33:48.2814267Z remote: Compressing objects: 43% (170/395) 2024-12-18T03:33:48.2818485Z remote: Compressing objects: 44% (174/395) 2024-12-18T03:33:48.2827236Z remote: Compressing objects: 45% (178/395) 2024-12-18T03:33:48.2831600Z remote: Compressing objects: 46% (182/395) 2024-12-18T03:33:48.2832629Z remote: Compressing objects: 47% (186/395) 2024-12-18T03:33:48.2838537Z remote: Compressing objects: 48% (190/395) 2024-12-18T03:33:48.2840318Z remote: Compressing objects: 49% (194/395) 2024-12-18T03:33:48.2844745Z remote: Compressing objects: 50% (198/395) 2024-12-18T03:33:48.2851516Z remote: Compressing objects: 51% (202/395) 2024-12-18T03:33:48.2861834Z remote: Compressing objects: 52% (206/395) 2024-12-18T03:33:48.2867293Z remote: Compressing objects: 53% (210/395) 2024-12-18T03:33:48.2871330Z remote: Compressing objects: 54% (214/395) 2024-12-18T03:33:48.2881045Z remote: Compressing objects: 55% (218/395) 2024-12-18T03:33:48.2885666Z remote: Compressing objects: 56% (222/395) 2024-12-18T03:33:48.2887026Z remote: Compressing objects: 57% (226/395) 2024-12-18T03:33:48.2892169Z remote: Compressing objects: 58% (230/395) 2024-12-18T03:33:48.2895545Z remote: Compressing objects: 59% (234/395) 2024-12-18T03:33:48.2897985Z remote: Compressing objects: 60% (237/395) 2024-12-18T03:33:48.2902434Z remote: Compressing objects: 61% (241/395) 2024-12-18T03:33:48.2904725Z remote: Compressing objects: 62% (245/395) 2024-12-18T03:33:48.2907909Z remote: Compressing objects: 63% (249/395) 2024-12-18T03:33:48.2910640Z remote: Compressing objects: 64% (253/395) 2024-12-18T03:33:48.2912649Z remote: Compressing objects: 65% (257/395) 2024-12-18T03:33:48.2915753Z remote: Compressing objects: 66% (261/395) 2024-12-18T03:33:48.2921166Z remote: Compressing objects: 67% (265/395) 2024-12-18T03:33:48.2921793Z remote: Compressing objects: 68% (269/395) 2024-12-18T03:33:48.2923206Z remote: Compressing objects: 69% (273/395) 2024-12-18T03:33:48.2929250Z remote: Compressing objects: 70% (277/395) 2024-12-18T03:33:48.2934427Z remote: Compressing objects: 71% (281/395) 2024-12-18T03:33:48.2939704Z remote: Compressing objects: 72% (285/395) 2024-12-18T03:33:48.2944943Z remote: Compressing objects: 73% (289/395) 2024-12-18T03:33:48.2945660Z remote: Compressing objects: 74% (293/395) 2024-12-18T03:33:48.2949392Z remote: Compressing objects: 75% (297/395) 2024-12-18T03:33:48.2952444Z remote: Compressing objects: 76% (301/395) 2024-12-18T03:33:48.2954970Z remote: Compressing objects: 77% (305/395) 2024-12-18T03:33:48.2957846Z remote: Compressing objects: 78% (309/395) 2024-12-18T03:33:48.2959997Z remote: Compressing objects: 79% (313/395) 2024-12-18T03:33:48.2960901Z remote: Compressing objects: 80% (316/395) 2024-12-18T03:33:48.2962296Z remote: Compressing objects: 81% (320/395) 2024-12-18T03:33:48.2963797Z remote: Compressing objects: 82% (324/395) 2024-12-18T03:33:48.2967385Z remote: Compressing objects: 83% (328/395) 2024-12-18T03:33:48.2970044Z remote: Compressing objects: 84% (332/395) 2024-12-18T03:33:48.2973249Z remote: Compressing objects: 85% (336/395) 2024-12-18T03:33:48.2974986Z remote: Compressing objects: 86% (340/395) 2024-12-18T03:33:48.2976746Z remote: Compressing objects: 87% (344/395) 2024-12-18T03:33:48.2979180Z remote: Compressing objects: 88% (348/395) 2024-12-18T03:33:48.2979880Z remote: Compressing objects: 89% (352/395) 2024-12-18T03:33:48.2985288Z remote: Compressing objects: 90% (356/395) 2024-12-18T03:33:48.2985939Z remote: Compressing objects: 91% (360/395) 2024-12-18T03:33:48.2986548Z remote: Compressing objects: 92% (364/395) 2024-12-18T03:33:48.2987174Z remote: Compressing objects: 93% (368/395) 2024-12-18T03:33:48.2987898Z remote: Compressing objects: 94% (372/395) 2024-12-18T03:33:48.2989725Z remote: Compressing objects: 95% (376/395) 2024-12-18T03:33:48.2990691Z remote: Compressing objects: 96% (380/395) 2024-12-18T03:33:48.2992049Z remote: Compressing objects: 97% (384/395) 2024-12-18T03:33:48.2994684Z remote: Compressing objects: 98% (388/395) 2024-12-18T03:33:48.2995878Z remote: Compressing objects: 99% (392/395) 2024-12-18T03:33:48.2997095Z remote: Compressing objects: 100% (395/395) 2024-12-18T03:33:48.2997781Z remote: Compressing objects: 100% (395/395), done. 2024-12-18T03:34:04.3810619Z remote: Total 1056590 (delta 804), reused 593 (delta 583), pack-reused 1055612 (from 5) 2024-12-18T03:34:15.2248613Z [command]/usr/bin/git rev-parse --verify --quiet 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7^{object} 2024-12-18T03:34:15.2301966Z 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:34:15.2314759Z ##[endgroup] 2024-12-18T03:34:15.2315654Z ##[group]Determining the checkout info 2024-12-18T03:34:15.2316483Z ##[endgroup] 2024-12-18T03:34:15.2317134Z ##[group]Checking out the ref 2024-12-18T03:34:15.2320188Z [command]/usr/bin/git checkout --quiet --force 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:34:16.7487338Z ##[endgroup] 2024-12-18T03:34:16.7488162Z ##[group]Setting up auth for fetching submodules 2024-12-18T03:34:16.7489428Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-12-18T03:34:16.7561529Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-12-18T03:34:16.7617041Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-12-18T03:34:16.7683607Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-12-18T03:34:16.7738476Z ##[endgroup] 2024-12-18T03:34:16.7739322Z ##[group]Fetching submodules 2024-12-18T03:34:16.7742155Z [command]/usr/bin/git submodule sync --recursive 2024-12-18T03:34:16.8428236Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-12-18T03:34:16.9113347Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-12-18T03:34:16.9122313Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-12-18T03:34:16.9130359Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-12-18T03:34:16.9139324Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-12-18T03:34:16.9148186Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2024-12-18T03:34:16.9158300Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-12-18T03:34:16.9166855Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-12-18T03:34:16.9176298Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-12-18T03:34:16.9185168Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2024-12-18T03:34:16.9195197Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-12-18T03:34:16.9204097Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-12-18T03:34:16.9213818Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-12-18T03:34:16.9223694Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-12-18T03:34:16.9233752Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-12-18T03:34:16.9244350Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-12-18T03:34:16.9259449Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-12-18T03:34:16.9270492Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-12-18T03:34:16.9282587Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:34:16.9291588Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-12-18T03:34:16.9301038Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-12-18T03:34:16.9310215Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-12-18T03:34:16.9320537Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-12-18T03:34:16.9330101Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-12-18T03:34:16.9340136Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-12-18T03:34:16.9349834Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-12-18T03:34:16.9360523Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-12-18T03:34:16.9370724Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-12-18T03:34:16.9381327Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-12-18T03:34:16.9391698Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-12-18T03:34:16.9402806Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-12-18T03:34:16.9413260Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-12-18T03:34:16.9424322Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-12-18T03:34:16.9435007Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-12-18T03:34:16.9446973Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-12-18T03:34:16.9457846Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-12-18T03:34:16.9468979Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-12-18T03:34:16.9651399Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-12-18T03:34:19.6426304Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-12-18T03:34:23.1893093Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-12-18T03:34:26.2663098Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-12-18T03:34:29.3170293Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2024-12-18T03:34:32.0667489Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-12-18T03:34:35.2423427Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-12-18T03:34:44.1529620Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-12-18T03:34:47.3345807Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2024-12-18T03:34:51.1839752Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-12-18T03:34:54.3632437Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-12-18T03:34:57.5423274Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-12-18T03:35:01.2058282Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-12-18T03:35:05.2024018Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-12-18T03:35:09.0114803Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-12-18T03:35:19.7833783Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-12-18T03:35:23.5482746Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-12-18T03:35:27.6816199Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-12-18T03:35:31.3021023Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-12-18T03:35:33.9266806Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-12-18T03:35:37.3589922Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-12-18T03:35:40.5539541Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-12-18T03:35:43.5203901Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-12-18T03:35:47.6458268Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-12-18T03:35:51.1188050Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-12-18T03:35:54.2243185Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-12-18T03:36:01.1130457Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-12-18T03:36:05.3419685Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-12-18T03:36:11.3551850Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-12-18T03:36:14.5943315Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-12-18T03:36:23.3219734Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-12-18T03:36:26.6765881Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-12-18T03:36:29.6944749Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-12-18T03:36:33.7583750Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-12-18T03:36:37.3670148Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-12-18T03:36:41.6396807Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-12-18T03:36:45.7007759Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-12-18T03:36:45.7667356Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-12-18T03:36:45.8263469Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-12-18T03:36:45.9225656Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-12-18T03:36:46.0325201Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1' 2024-12-18T03:36:46.1436765Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-12-18T03:36:47.0240589Z Submodule path 'third_party/XNNPACK': checked out '4ea82e595b36106653175dcb04b2aa532660d0d8' 2024-12-18T03:36:47.1225377Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-12-18T03:36:47.4456985Z Submodule path 'third_party/composable_kernel': checked out '50ee4267e27b875d149e642f4cebd47be1dc3b57' 2024-12-18T03:36:47.5736310Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-12-18T03:36:47.7514018Z Submodule path 'third_party/cpuinfo': checked out '1e83a2fdd3102f65c6f1fb602c1b320486218a99' 2024-12-18T03:36:47.8649114Z Submodule path 'third_party/cudnn_frontend': checked out '936021bfed8c91dc416af1588b2c4eca631a9e45' 2024-12-18T03:36:48.4929203Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-12-18T03:36:48.8216648Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-12-18T03:36:49.0213346Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-12-18T03:36:49.0342041Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:36:49.0351151Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:36:49.0360113Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:36:49.0369915Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:36:49.0380280Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:36:49.0507520Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-12-18T03:36:53.9018988Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-12-18T03:36:58.7614828Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-12-18T03:37:04.3685226Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-12-18T03:37:09.9861680Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-12-18T03:37:15.2721126Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-12-18T03:37:15.4484713Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-12-18T03:37:15.9545813Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-12-18T03:37:16.1001490Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-12-18T03:37:16.1629830Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-12-18T03:37:16.3809492Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-12-18T03:37:16.5030282Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-12-18T03:37:16.6265753Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-12-18T03:37:16.7236980Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-12-18T03:37:16.8425976Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59' 2024-12-18T03:37:16.9122690Z Submodule path 'third_party/ideep': checked out 'c7ccd5bdbe5434ba156f4e856dcef0601637334b' 2024-12-18T03:37:16.9246188Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-12-18T03:37:16.9360564Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-12-18T03:37:31.5758809Z Submodule path 'third_party/ideep/mkl-dnn': checked out '66f0cb9eb66affd2da3bf5f8d897376f04aae6af' 2024-12-18T03:37:31.6633112Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-12-18T03:37:31.8297550Z Submodule path 'third_party/kineto': checked out '338140f58a28d599da3434ced4fd2d75dd1a213d' 2024-12-18T03:37:31.8429971Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:37:31.8438369Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:37:31.8447866Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:37:31.8569705Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-12-18T03:37:37.6488436Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-12-18T03:37:44.0047962Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-12-18T03:37:50.7335588Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-12-18T03:37:50.7451828Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:37:50.7460615Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:37:50.7467202Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:37:50.7476098Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:37:50.7485654Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:37:50.7495162Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:37:50.7504014Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:37:50.7513436Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:37:50.7637379Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-12-18T03:37:57.2487027Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-12-18T03:38:02.9117892Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-12-18T03:38:09.9482400Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-12-18T03:38:16.1797423Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-12-18T03:38:22.9025234Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-12-18T03:38:29.5110587Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-12-18T03:38:39.9897962Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-12-18T03:38:46.6253346Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-12-18T03:38:46.7030937Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-12-18T03:38:46.8114171Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-12-18T03:38:46.8662715Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-12-18T03:38:46.8788377Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:38:46.8905535Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-12-18T03:38:53.6420129Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-12-18T03:38:53.7216271Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-12-18T03:38:53.8335475Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-12-18T03:38:54.0096782Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-12-18T03:38:54.0828351Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-12-18T03:38:54.1938887Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-12-18T03:38:54.3298868Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-12-18T03:38:54.4330811Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-12-18T03:38:54.5332119Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-12-18T03:38:54.7368430Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-12-18T03:38:55.2351469Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c' 2024-12-18T03:38:55.2513335Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-12-18T03:38:55.2662862Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-12-18T03:39:02.4656957Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-12-18T03:39:02.6200965Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-12-18T03:39:02.6333187Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:02.6341781Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:02.6349880Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:02.6359159Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:02.6369041Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:02.6378031Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:02.6386503Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:02.6395318Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:02.6514979Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-12-18T03:39:07.5222857Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-12-18T03:39:11.7574328Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-12-18T03:39:14.4120397Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-12-18T03:39:20.4282172Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-12-18T03:39:20.9994576Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-12-18T03:39:21.5169327Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-12-18T03:39:22.1094077Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-12-18T03:39:31.3548804Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-12-18T03:39:31.4692718Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-12-18T03:39:31.5371262Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-12-18T03:39:31.7288010Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-12-18T03:39:31.7950451Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-12-18T03:39:31.8643142Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-12-18T03:39:31.9355568Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-12-18T03:39:31.9487873Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:31.9496081Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:31.9611766Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-12-18T03:39:33.5235788Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-12-18T03:39:35.0043721Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-12-18T03:39:35.1244182Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-12-18T03:39:35.7357927Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-12-18T03:39:35.8083551Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-12-18T03:39:36.1853148Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-12-18T03:39:36.1991275Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:36.1999704Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:36.2131170Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-12-18T03:39:36.9605243Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-12-18T03:39:38.1227385Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-12-18T03:39:38.2813964Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-12-18T03:39:38.3361202Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-12-18T03:39:38.4048660Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-12-18T03:39:38.5259699Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2024-12-18T03:39:38.6318298Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-12-18T03:39:38.7532178Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-12-18T03:39:38.8525323Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-12-18T03:39:38.8647203Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:38.8654984Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:38.8663964Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:38.8672410Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:38.8792917Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-12-18T03:39:39.9117090Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-12-18T03:39:40.4840411Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-12-18T03:39:42.8191000Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-12-18T03:39:44.6431970Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-12-18T03:39:44.7136941Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-12-18T03:39:44.8653591Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-12-18T03:39:44.9630826Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-12-18T03:39:44.9747080Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:44.9874259Z Cloning into '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-12-18T03:39:45.5194103Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-12-18T03:39:45.5441013Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-12-18T03:39:45.6108227Z Entering 'android/libs/fbjni' 2024-12-18T03:39:45.6220470Z Entering 'third_party/FP16' 2024-12-18T03:39:45.6332988Z Entering 'third_party/FXdiv' 2024-12-18T03:39:45.6449755Z Entering 'third_party/NNPACK' 2024-12-18T03:39:45.6565157Z Entering 'third_party/NVTX' 2024-12-18T03:39:45.6672038Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:39:45.6781046Z Entering 'third_party/XNNPACK' 2024-12-18T03:39:45.6923346Z Entering 'third_party/benchmark' 2024-12-18T03:39:45.7031384Z Entering 'third_party/composable_kernel' 2024-12-18T03:39:45.7147383Z Entering 'third_party/cpp-httplib' 2024-12-18T03:39:45.7252204Z Entering 'third_party/cpuinfo' 2024-12-18T03:39:45.7359151Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:39:45.7465461Z Entering 'third_party/cutlass' 2024-12-18T03:39:45.7588317Z Entering 'third_party/eigen' 2024-12-18T03:39:45.7698277Z Entering 'third_party/fbgemm' 2024-12-18T03:39:45.7805974Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:39:45.7905895Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:39:45.8009197Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:39:45.8130792Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:39:45.8229789Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:39:45.8326402Z Entering 'third_party/flatbuffers' 2024-12-18T03:39:45.8440052Z Entering 'third_party/fmt' 2024-12-18T03:39:45.8543543Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:39:45.8652781Z Entering 'third_party/gloo' 2024-12-18T03:39:45.8760673Z Entering 'third_party/googletest' 2024-12-18T03:39:45.8866813Z Entering 'third_party/ideep' 2024-12-18T03:39:45.8970712Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:39:45.9088258Z Entering 'third_party/ittapi' 2024-12-18T03:39:45.9196767Z Entering 'third_party/kineto' 2024-12-18T03:39:45.9299502Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:39:45.9396149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:39:45.9498348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:39:45.9600816Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:39:45.9699072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:39:45.9794963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:39:45.9900522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:39:46.0000343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:39:46.0098902Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:39:46.0204545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:39:46.0309333Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:39:46.0409915Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:39:46.0518416Z Entering 'third_party/mimalloc' 2024-12-18T03:39:46.0628771Z Entering 'third_party/nccl/nccl' 2024-12-18T03:39:46.0737285Z Entering 'third_party/nlohmann' 2024-12-18T03:39:46.0843645Z Entering 'third_party/onnx' 2024-12-18T03:39:46.0982182Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:39:46.1093548Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:39:46.1200480Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:46.1300133Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:46.1396867Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:46.1494232Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:46.1595107Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:46.1694960Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:46.1791612Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:46.1886982Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:46.1995371Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:46.2101214Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:46.2251570Z Entering 'third_party/pocketfft' 2024-12-18T03:39:46.2357047Z Entering 'third_party/protobuf' 2024-12-18T03:39:46.2468310Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:46.2573613Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:46.2681548Z Entering 'third_party/psimd' 2024-12-18T03:39:46.2789624Z Entering 'third_party/pthreadpool' 2024-12-18T03:39:46.2896424Z Entering 'third_party/pybind11' 2024-12-18T03:39:46.3001131Z Entering 'third_party/python-peachpy' 2024-12-18T03:39:46.3109758Z Entering 'third_party/sleef' 2024-12-18T03:39:46.3212250Z Entering 'third_party/tensorpipe' 2024-12-18T03:39:46.3314251Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:46.3414331Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:46.3514814Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:46.3614642Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:46.3709666Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:46.3856009Z ##[endgroup] 2024-12-18T03:39:46.3857131Z ##[group]Persisting credentials for submodules 2024-12-18T03:39:46.3862329Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-12-18T03:39:46.4538241Z Entering 'android/libs/fbjni' 2024-12-18T03:39:46.4663924Z Entering 'third_party/FP16' 2024-12-18T03:39:46.4786935Z Entering 'third_party/FXdiv' 2024-12-18T03:39:46.4910194Z Entering 'third_party/NNPACK' 2024-12-18T03:39:46.5035196Z Entering 'third_party/NVTX' 2024-12-18T03:39:46.5159251Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:39:46.5280309Z Entering 'third_party/XNNPACK' 2024-12-18T03:39:46.5434727Z Entering 'third_party/benchmark' 2024-12-18T03:39:46.5554857Z Entering 'third_party/composable_kernel' 2024-12-18T03:39:46.5693728Z Entering 'third_party/cpp-httplib' 2024-12-18T03:39:46.5814878Z Entering 'third_party/cpuinfo' 2024-12-18T03:39:46.5938908Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:39:46.6059486Z Entering 'third_party/cutlass' 2024-12-18T03:39:46.6201640Z Entering 'third_party/eigen' 2024-12-18T03:39:46.6328331Z Entering 'third_party/fbgemm' 2024-12-18T03:39:46.6446878Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:39:46.6563155Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:39:46.6678030Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:39:46.6807707Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:39:46.6920548Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:39:46.7040738Z Entering 'third_party/flatbuffers' 2024-12-18T03:39:46.7167269Z Entering 'third_party/fmt' 2024-12-18T03:39:46.7281355Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:39:46.7403570Z Entering 'third_party/gloo' 2024-12-18T03:39:46.7525551Z Entering 'third_party/googletest' 2024-12-18T03:39:46.7644963Z Entering 'third_party/ideep' 2024-12-18T03:39:46.7760638Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:39:46.7894353Z Entering 'third_party/ittapi' 2024-12-18T03:39:46.8012633Z Entering 'third_party/kineto' 2024-12-18T03:39:46.8130878Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:39:46.8240782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:39:46.8358808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:39:46.8475314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:39:46.8590946Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:39:46.8700423Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:39:46.8822958Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:39:46.8939024Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:39:46.9052872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:39:46.9169006Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:39:46.9286389Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:39:46.9404689Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:39:46.9527239Z Entering 'third_party/mimalloc' 2024-12-18T03:39:46.9647918Z Entering 'third_party/nccl/nccl' 2024-12-18T03:39:46.9767604Z Entering 'third_party/nlohmann' 2024-12-18T03:39:46.9894196Z Entering 'third_party/onnx' 2024-12-18T03:39:47.0041549Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:39:47.0169981Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:39:47.0292722Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:47.0407225Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:47.0520739Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:47.0633827Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:47.0752935Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:47.0869050Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:47.0983771Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:47.1092416Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:47.1212959Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:47.1334601Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:47.1501180Z Entering 'third_party/pocketfft' 2024-12-18T03:39:47.1621345Z Entering 'third_party/protobuf' 2024-12-18T03:39:47.1743162Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:47.1857944Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:47.1981422Z Entering 'third_party/psimd' 2024-12-18T03:39:47.2102981Z Entering 'third_party/pthreadpool' 2024-12-18T03:39:47.2222737Z Entering 'third_party/pybind11' 2024-12-18T03:39:47.2341230Z Entering 'third_party/python-peachpy' 2024-12-18T03:39:47.2459564Z Entering 'third_party/sleef' 2024-12-18T03:39:47.2581074Z Entering 'third_party/tensorpipe' 2024-12-18T03:39:47.2693769Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:47.2810690Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:47.2920268Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:47.3035394Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:47.3141038Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:47.3302628Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-12-18T03:39:47.3950990Z Entering 'android/libs/fbjni' 2024-12-18T03:39:47.4052384Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-12-18T03:39:47.4108823Z Entering 'third_party/FP16' 2024-12-18T03:39:47.4209946Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-12-18T03:39:47.4266716Z Entering 'third_party/FXdiv' 2024-12-18T03:39:47.4367864Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-12-18T03:39:47.4425094Z Entering 'third_party/NNPACK' 2024-12-18T03:39:47.4534275Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-12-18T03:39:47.4592435Z Entering 'third_party/NVTX' 2024-12-18T03:39:47.4698746Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2024-12-18T03:39:47.4759945Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:39:47.4862026Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-12-18T03:39:47.4920799Z Entering 'third_party/XNNPACK' 2024-12-18T03:39:47.5022637Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-12-18T03:39:47.5117909Z Entering 'third_party/benchmark' 2024-12-18T03:39:47.5217592Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:39:47.5277058Z Entering 'third_party/composable_kernel' 2024-12-18T03:39:47.5378508Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2024-12-18T03:39:47.5453482Z Entering 'third_party/cpp-httplib' 2024-12-18T03:39:47.5560748Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-12-18T03:39:47.5613750Z Entering 'third_party/cpuinfo' 2024-12-18T03:39:47.5721867Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T03:39:47.5774185Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:39:47.5878345Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-12-18T03:39:47.5936225Z Entering 'third_party/cutlass' 2024-12-18T03:39:47.6034513Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-12-18T03:39:47.6111514Z Entering 'third_party/eigen' 2024-12-18T03:39:47.6213349Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-12-18T03:39:47.6280542Z Entering 'third_party/fbgemm' 2024-12-18T03:39:47.6378422Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-12-18T03:39:47.6433081Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:39:47.6536996Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-12-18T03:39:47.6591241Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:39:47.6690050Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-12-18T03:39:47.6745154Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:39:47.6842616Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-12-18T03:39:47.6913862Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:39:47.7011647Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:47.7065662Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:39:47.7165074Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-12-18T03:39:47.7225218Z Entering 'third_party/flatbuffers' 2024-12-18T03:39:47.7326815Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-12-18T03:39:47.7390737Z Entering 'third_party/fmt' 2024-12-18T03:39:47.7494402Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-12-18T03:39:47.7553736Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:39:47.7655827Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-12-18T03:39:47.7715578Z Entering 'third_party/gloo' 2024-12-18T03:39:47.7814503Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-12-18T03:39:47.7871691Z Entering 'third_party/googletest' 2024-12-18T03:39:47.7974095Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:47.8030773Z Entering 'third_party/ideep' 2024-12-18T03:39:47.8136418Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-12-18T03:39:47.8191470Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:39:47.8290764Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-12-18T03:39:47.8364488Z Entering 'third_party/ittapi' 2024-12-18T03:39:47.8466780Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-12-18T03:39:47.8525042Z Entering 'third_party/kineto' 2024-12-18T03:39:47.8626411Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-12-18T03:39:47.8681256Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:39:47.8778910Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-12-18T03:39:47.8829253Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:39:47.8930377Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-12-18T03:39:47.8989154Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:39:47.9086758Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-12-18T03:39:47.9140610Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:39:47.9242901Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-12-18T03:39:47.9298133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:39:47.9396530Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-12-18T03:39:47.9446795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:39:47.9548198Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-12-18T03:39:47.9609755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:39:47.9708913Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-12-18T03:39:47.9762718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:39:47.9861642Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:47.9923547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:39:48.0030393Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-12-18T03:39:48.0091290Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:39:48.0193527Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-12-18T03:39:48.0257947Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:39:48.0360210Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-12-18T03:39:48.0416251Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:39:48.0514190Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-12-18T03:39:48.0577941Z Entering 'third_party/mimalloc' 2024-12-18T03:39:48.0679624Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-12-18T03:39:48.0736785Z Entering 'third_party/nccl/nccl' 2024-12-18T03:39:48.0839069Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-12-18T03:39:48.0896060Z Entering 'third_party/nlohmann' 2024-12-18T03:39:48.0997669Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-12-18T03:39:48.1058993Z Entering 'third_party/onnx' 2024-12-18T03:39:48.1160002Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-12-18T03:39:48.1249768Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:39:48.1352844Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:39:48.1420829Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:39:48.1524250Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-12-18T03:39:48.1582315Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:48.1683496Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:39:48.1738606Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:48.1839699Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:48.1896065Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:48.1996176Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-12-18T03:39:48.2053366Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:48.2151925Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-12-18T03:39:48.2210490Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:48.2310014Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-12-18T03:39:48.2364596Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:48.2465093Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-12-18T03:39:48.2519746Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:48.2619140Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-12-18T03:39:48.2667498Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:48.2766563Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-12-18T03:39:48.2826196Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:48.2925766Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-12-18T03:39:48.2987189Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:48.3084518Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-12-18T03:39:48.3184564Z Entering 'third_party/pocketfft' 2024-12-18T03:39:48.3286452Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-12-18T03:39:48.3341637Z Entering 'third_party/protobuf' 2024-12-18T03:39:48.3444321Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-12-18T03:39:48.3505119Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:48.3604427Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-12-18T03:39:48.3657622Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:48.3758067Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:48.3820830Z Entering 'third_party/psimd' 2024-12-18T03:39:48.3923193Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-12-18T03:39:48.3980075Z Entering 'third_party/pthreadpool' 2024-12-18T03:39:48.4082829Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-12-18T03:39:48.4139788Z Entering 'third_party/pybind11' 2024-12-18T03:39:48.4240789Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:39:48.4298848Z Entering 'third_party/python-peachpy' 2024-12-18T03:39:48.4400217Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-12-18T03:39:48.4456741Z Entering 'third_party/sleef' 2024-12-18T03:39:48.4556159Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-12-18T03:39:48.4612156Z Entering 'third_party/tensorpipe' 2024-12-18T03:39:48.4712499Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-12-18T03:39:48.4769275Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:48.4866823Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-12-18T03:39:48.4922192Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:48.5018776Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-12-18T03:39:48.5072044Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:48.5167338Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-12-18T03:39:48.5223230Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:48.5319799Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-12-18T03:39:48.5369740Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:48.5466307Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-12-18T03:39:48.6030380Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-12-18T03:39:48.6682073Z Entering 'android/libs/fbjni' 2024-12-18T03:39:48.6792859Z Entering 'third_party/FP16' 2024-12-18T03:39:48.6905262Z Entering 'third_party/FXdiv' 2024-12-18T03:39:48.7013824Z Entering 'third_party/NNPACK' 2024-12-18T03:39:48.7120628Z Entering 'third_party/NVTX' 2024-12-18T03:39:48.7229583Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:39:48.7337701Z Entering 'third_party/XNNPACK' 2024-12-18T03:39:48.7477900Z Entering 'third_party/benchmark' 2024-12-18T03:39:48.7583293Z Entering 'third_party/composable_kernel' 2024-12-18T03:39:48.7705667Z Entering 'third_party/cpp-httplib' 2024-12-18T03:39:48.7812392Z Entering 'third_party/cpuinfo' 2024-12-18T03:39:48.7918944Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:39:48.8028437Z Entering 'third_party/cutlass' 2024-12-18T03:39:48.8153363Z Entering 'third_party/eigen' 2024-12-18T03:39:48.8266566Z Entering 'third_party/fbgemm' 2024-12-18T03:39:48.8371673Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:39:48.8472153Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:39:48.8581631Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:39:48.8699587Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:39:48.8800095Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:39:48.8906264Z Entering 'third_party/flatbuffers' 2024-12-18T03:39:48.9021078Z Entering 'third_party/fmt' 2024-12-18T03:39:48.9130774Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:39:48.9243101Z Entering 'third_party/gloo' 2024-12-18T03:39:48.9348054Z Entering 'third_party/googletest' 2024-12-18T03:39:48.9455926Z Entering 'third_party/ideep' 2024-12-18T03:39:48.9557579Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:39:48.9681000Z Entering 'third_party/ittapi' 2024-12-18T03:39:48.9785235Z Entering 'third_party/kineto' 2024-12-18T03:39:48.9887757Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:39:48.9983796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:39:49.0088485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:39:49.0192605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:39:49.0293747Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:39:49.0393558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:39:49.0503412Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:39:49.0605621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:39:49.0709178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:39:49.0814689Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:39:49.0923127Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:39:49.1027757Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:39:49.1133193Z Entering 'third_party/mimalloc' 2024-12-18T03:39:49.1239200Z Entering 'third_party/nccl/nccl' 2024-12-18T03:39:49.1346124Z Entering 'third_party/nlohmann' 2024-12-18T03:39:49.1456396Z Entering 'third_party/onnx' 2024-12-18T03:39:49.1589748Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:39:49.1703108Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:39:49.1810569Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:49.1907959Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:49.2005846Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:49.2101799Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:49.2201051Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:49.2296200Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:49.2391235Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:49.2484487Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:49.2584906Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:49.2690365Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:49.2835019Z Entering 'third_party/pocketfft' 2024-12-18T03:39:49.2939839Z Entering 'third_party/protobuf' 2024-12-18T03:39:49.3050671Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:49.3149884Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:49.3256687Z Entering 'third_party/psimd' 2024-12-18T03:39:49.3364232Z Entering 'third_party/pthreadpool' 2024-12-18T03:39:49.3470038Z Entering 'third_party/pybind11' 2024-12-18T03:39:49.3579458Z Entering 'third_party/python-peachpy' 2024-12-18T03:39:49.3685063Z Entering 'third_party/sleef' 2024-12-18T03:39:49.3787058Z Entering 'third_party/tensorpipe' 2024-12-18T03:39:49.3888435Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:49.3985201Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:49.4083788Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:49.4180858Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:49.4279055Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:49.4423924Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-12-18T03:39:49.5065619Z Entering 'android/libs/fbjni' 2024-12-18T03:39:49.5174469Z Entering 'third_party/FP16' 2024-12-18T03:39:49.5281666Z Entering 'third_party/FXdiv' 2024-12-18T03:39:49.5386903Z Entering 'third_party/NNPACK' 2024-12-18T03:39:49.5492685Z Entering 'third_party/NVTX' 2024-12-18T03:39:49.5599464Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T03:39:49.5706523Z Entering 'third_party/XNNPACK' 2024-12-18T03:39:49.5845341Z Entering 'third_party/benchmark' 2024-12-18T03:39:49.5947290Z Entering 'third_party/composable_kernel' 2024-12-18T03:39:49.6069542Z Entering 'third_party/cpp-httplib' 2024-12-18T03:39:49.6172448Z Entering 'third_party/cpuinfo' 2024-12-18T03:39:49.6275099Z Entering 'third_party/cudnn_frontend' 2024-12-18T03:39:49.6381439Z Entering 'third_party/cutlass' 2024-12-18T03:39:49.6508769Z Entering 'third_party/eigen' 2024-12-18T03:39:49.6617664Z Entering 'third_party/fbgemm' 2024-12-18T03:39:49.6721871Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T03:39:49.6822930Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T03:39:49.6922942Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T03:39:49.7034439Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T03:39:49.7132707Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T03:39:49.7241164Z Entering 'third_party/flatbuffers' 2024-12-18T03:39:49.7352063Z Entering 'third_party/fmt' 2024-12-18T03:39:49.7457980Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T03:39:49.7562390Z Entering 'third_party/gloo' 2024-12-18T03:39:49.7668675Z Entering 'third_party/googletest' 2024-12-18T03:39:49.7777164Z Entering 'third_party/ideep' 2024-12-18T03:39:49.7879460Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T03:39:49.8000808Z Entering 'third_party/ittapi' 2024-12-18T03:39:49.8103857Z Entering 'third_party/kineto' 2024-12-18T03:39:49.8208435Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T03:39:49.8310389Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T03:39:49.8417323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T03:39:49.8512911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T03:39:49.8612456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T03:39:49.8711004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T03:39:49.8820058Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T03:39:49.8921681Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T03:39:49.9023480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T03:39:49.9127077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T03:39:49.9233880Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T03:39:49.9336978Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T03:39:49.9445058Z Entering 'third_party/mimalloc' 2024-12-18T03:39:49.9553511Z Entering 'third_party/nccl/nccl' 2024-12-18T03:39:49.9660254Z Entering 'third_party/nlohmann' 2024-12-18T03:39:49.9771891Z Entering 'third_party/onnx' 2024-12-18T03:39:49.9906498Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T03:39:50.0025579Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T03:39:50.0144416Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T03:39:50.0250810Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T03:39:50.0361026Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T03:39:50.0460683Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T03:39:50.0565330Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T03:39:50.0665480Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T03:39:50.0766108Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T03:39:50.0865550Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T03:39:50.0974973Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T03:39:50.1085740Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T03:39:50.1240649Z Entering 'third_party/pocketfft' 2024-12-18T03:39:50.1344503Z Entering 'third_party/protobuf' 2024-12-18T03:39:50.1459891Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T03:39:50.1563211Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T03:39:50.1671746Z Entering 'third_party/psimd' 2024-12-18T03:39:50.1779393Z Entering 'third_party/pthreadpool' 2024-12-18T03:39:50.1892957Z Entering 'third_party/pybind11' 2024-12-18T03:39:50.2002672Z Entering 'third_party/python-peachpy' 2024-12-18T03:39:50.2111470Z Entering 'third_party/sleef' 2024-12-18T03:39:50.2220418Z Entering 'third_party/tensorpipe' 2024-12-18T03:39:50.2325543Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T03:39:50.2429200Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T03:39:50.2529189Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T03:39:50.2631872Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T03:39:50.2729931Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T03:39:50.2882580Z ##[endgroup] 2024-12-18T03:39:50.2960031Z [command]/usr/bin/git log -1 --format='%H' 2024-12-18T03:39:50.3014973Z '0cdf8b1d09254cfda66191d1bd01e3041c3c76f7' 2024-12-18T03:39:50.3356720Z Prepare all required actions 2024-12-18T03:39:50.3357740Z Getting action download info 2024-12-18T03:39:50.3466374Z ##[group]Run ./.github/actions/setup-rocm 2024-12-18T03:39:50.3466862Z env: 2024-12-18T03:39:50.3467190Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.3467591Z ##[endgroup] 2024-12-18T03:39:50.3504479Z ##[group]Run echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-12-18T03:39:50.3505462Z echo "DOCKER_HOST=unix:///run/user/$(id -u)/docker.sock" >> "${GITHUB_ENV}" 2024-12-18T03:39:50.3566483Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.3567082Z env: 2024-12-18T03:39:50.3567414Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.3567806Z ##[endgroup] 2024-12-18T03:39:50.3747216Z ##[group]Run set -ex 2024-12-18T03:39:50.3747702Z set -ex 2024-12-18T03:39:50.3748097Z  2024-12-18T03:39:50.3748535Z cat ~/.docker/config.json || true 2024-12-18T03:39:50.3749987Z # https://stackoverflow.com/questions/64455468/error-when-logging-into-ecr-with-docker-login-error-saving-credentials-not 2024-12-18T03:39:50.3751287Z rm -f ~/.docker/config.json 2024-12-18T03:39:50.3823836Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.3824512Z env: 2024-12-18T03:39:50.3824896Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.3825449Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:50.3826069Z ##[endgroup] 2024-12-18T03:39:50.3930314Z + cat /var/home/pytorchci/.docker/config.json 2024-12-18T03:39:50.3956740Z { 2024-12-18T03:39:50.3957152Z "auths": {} 2024-12-18T03:39:50.3961372Z + rm -f /var/home/pytorchci/.docker/config.json 2024-12-18T03:39:50.3998215Z } 2024-12-18T03:39:50.4033463Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T03:39:50.4034312Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T03:39:50.4034945Z # shellcheck disable=SC2046 2024-12-18T03:39:50.4035503Z docker stop $(docker ps -q) || true 2024-12-18T03:39:50.4036053Z # Prune all stopped containers. 2024-12-18T03:39:50.4036546Z docker container prune -f 2024-12-18T03:39:50.4090946Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.4091537Z env: 2024-12-18T03:39:50.4091867Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.4092355Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:50.4092904Z ##[endgroup] 2024-12-18T03:39:50.4836589Z "docker stop" requires at least 1 argument. 2024-12-18T03:39:50.4837296Z See 'docker stop --help'. 2024-12-18T03:39:50.4837614Z 2024-12-18T03:39:50.4837929Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-12-18T03:39:50.4838440Z 2024-12-18T03:39:50.4838641Z Stop one or more running containers 2024-12-18T03:39:50.5187553Z Total reclaimed space: 0B 2024-12-18T03:39:50.5302413Z ##[group]Run cat /etc/os-release || true 2024-12-18T03:39:50.5303091Z cat /etc/os-release || true 2024-12-18T03:39:50.5303701Z cat /etc/apt/sources.list.d/rocm.list || true 2024-12-18T03:39:50.5304348Z cat /opt/rocm/.info/version || true 2024-12-18T03:39:50.5304877Z whoami 2024-12-18T03:39:50.5372080Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.5372738Z env: 2024-12-18T03:39:50.5373107Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.5373665Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:50.5374311Z ##[endgroup] 2024-12-18T03:39:50.5502295Z PRETTY_NAME="Ubuntu 22.04.4 LTS" 2024-12-18T03:39:50.5502918Z NAME="Ubuntu" 2024-12-18T03:39:50.5503359Z VERSION_ID="22.04" 2024-12-18T03:39:50.5503854Z VERSION="22.04.4 LTS (Jammy Jellyfish)" 2024-12-18T03:39:50.5504450Z VERSION_CODENAME=jammy 2024-12-18T03:39:50.5504902Z ID=ubuntu 2024-12-18T03:39:50.5505287Z ID_LIKE=debian 2024-12-18T03:39:50.5505750Z HOME_URL="https://www.ubuntu.com/" 2024-12-18T03:39:50.5507170Z SUPPORT_URL="https://help.ubuntu.com/" 2024-12-18T03:39:50.5507920Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-12-18T03:39:50.5508933Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-12-18T03:39:50.5509850Z UBUNTU_CODENAME=jammy 2024-12-18T03:39:50.5529740Z deb [arch=amd64] https://repo.radeon.com/rocm/apt/6.2.1 jammy main 2024-12-18T03:39:50.5560124Z 6.2.1-112 2024-12-18T03:39:50.5607498Z pytorchci 2024-12-18T03:39:50.5661313Z ##[group]Run rocm-smi 2024-12-18T03:39:50.5661966Z rocm-smi 2024-12-18T03:39:50.5730579Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.5731544Z env: 2024-12-18T03:39:50.5732164Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.5733116Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:50.5733999Z ##[endgroup] 2024-12-18T03:39:50.7780469Z 2024-12-18T03:39:50.7780606Z 2024-12-18T03:39:50.7781294Z ========================================= ROCm System Management Interface ========================================= 2024-12-18T03:39:50.7782348Z =================================================== Concise Info =================================================== 2024-12-18T03:39:50.7783640Z Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2024-12-18T03:39:50.7785206Z  (DID, GUID) (Edge) (Avg) (Mem, Compute, ID)  2024-12-18T03:39:50.7786085Z ==================================================================================================================== 2024-12-18T03:39:50.7788118Z 0 4 0x740c, 57586 39.0°C 93.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-12-18T03:39:50.7789358Z 1 5 0x740c, 45873 39.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-12-18T03:39:50.7790523Z 2 2 0x740c, 51627 34.0°C 90.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-12-18T03:39:50.7792003Z 3 3 0x740c, 64489 32.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-12-18T03:39:50.7793741Z 4 8 0x740c, 30939 36.0°C 92.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-12-18T03:39:50.7795223Z 5 9 0x740c, 8466 39.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-12-18T03:39:50.7796272Z 6 6 0x740c, 41154 35.0°C 99.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2024-12-18T03:39:50.7796933Z 7 7 0x740c, 63755 33.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2024-12-18T03:39:50.7797386Z ==================================================================================================================== 2024-12-18T03:39:50.7797829Z =============================================== End of ROCm SMI Log ================================================ 2024-12-18T03:39:50.7941672Z ##[group]Run rocminfo 2024-12-18T03:39:50.7942418Z rocminfo 2024-12-18T03:39:50.8006490Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:50.8007079Z env: 2024-12-18T03:39:50.8007373Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:50.8007776Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:50.8008293Z ##[endgroup] 2024-12-18T03:39:50.9552464Z ROCk module version 6.8.5 is loaded 2024-12-18T03:39:50.9553514Z ===================== 2024-12-18T03:39:50.9554190Z HSA System Attributes 2024-12-18T03:39:50.9555369Z ===================== 2024-12-18T03:39:50.9555954Z Runtime Version: 1.14 2024-12-18T03:39:50.9556889Z Runtime Ext Version: 1.6 2024-12-18T03:39:50.9557992Z System Timestamp Freq.: 1000.000000MHz 2024-12-18T03:39:50.9558947Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-12-18T03:39:50.9561077Z Machine Model: LARGE 2024-12-18T03:39:50.9562127Z System Endianness: LITTLE 2024-12-18T03:39:50.9563524Z Mwaitx: DISABLED 2024-12-18T03:39:50.9564646Z DMAbuf Support: YES 2024-12-18T03:39:50.9565336Z 2024-12-18T03:39:50.9565670Z ========== 2024-12-18T03:39:50.9567137Z HSA Agents 2024-12-18T03:39:50.9568037Z ========== 2024-12-18T03:39:50.9568658Z ******* 2024-12-18T03:39:50.9569556Z Agent 1 2024-12-18T03:39:50.9570202Z ******* 2024-12-18T03:39:50.9571056Z Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:39:50.9572082Z Uuid: CPU-XX 2024-12-18T03:39:50.9573044Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:39:50.9574060Z Vendor Name: CPU 2024-12-18T03:39:50.9575064Z Feature: None specified 2024-12-18T03:39:50.9575877Z Profile: FULL_PROFILE 2024-12-18T03:39:50.9576874Z Float Round Mode: NEAR 2024-12-18T03:39:50.9577878Z Max Queue Number: 0(0x0) 2024-12-18T03:39:50.9578714Z Queue Min Size: 0(0x0) 2024-12-18T03:39:50.9579864Z Queue Max Size: 0(0x0) 2024-12-18T03:39:50.9581379Z Queue Type: MULTI 2024-12-18T03:39:50.9582203Z Node: 0 2024-12-18T03:39:50.9583209Z Device Type: CPU 2024-12-18T03:39:50.9584035Z Cache Info: 2024-12-18T03:39:50.9584862Z L1: 32768(0x8000) KB 2024-12-18T03:39:50.9585866Z Chip ID: 0(0x0) 2024-12-18T03:39:50.9586792Z ASIC Revision: 0(0x0) 2024-12-18T03:39:50.9587672Z Cacheline Size: 64(0x40) 2024-12-18T03:39:50.9589585Z Max Clock Freq. (MHz): 2000 2024-12-18T03:39:50.9590594Z BDFID: 0 2024-12-18T03:39:50.9591424Z Internal Node ID: 0 2024-12-18T03:39:50.9592620Z Compute Unit: 64 2024-12-18T03:39:50.9593549Z SIMDs per CU: 0 2024-12-18T03:39:50.9594477Z Shader Engines: 0 2024-12-18T03:39:50.9595588Z Shader Arrs. per Eng.: 0 2024-12-18T03:39:50.9596445Z WatchPts on Addr. Ranges:1 2024-12-18T03:39:50.9597463Z Memory Properties: 2024-12-18T03:39:50.9598339Z Features: None 2024-12-18T03:39:50.9599313Z Pool Info: 2024-12-18T03:39:50.9600046Z Pool 1 2024-12-18T03:39:50.9600867Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:50.9602462Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:39:50.9603497Z Allocatable: TRUE 2024-12-18T03:39:50.9604755Z Alloc Granule: 4KB 2024-12-18T03:39:50.9605643Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9606610Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9607850Z Accessible by all: TRUE 2024-12-18T03:39:50.9608606Z Pool 2 2024-12-18T03:39:50.9609398Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:39:50.9610821Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:39:50.9611649Z Allocatable: TRUE 2024-12-18T03:39:50.9612618Z Alloc Granule: 4KB 2024-12-18T03:39:50.9613721Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9614681Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9615579Z Accessible by all: TRUE 2024-12-18T03:39:50.9616562Z Pool 3 2024-12-18T03:39:50.9617359Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:50.9618199Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:39:50.9619335Z Allocatable: TRUE 2024-12-18T03:39:50.9620258Z Alloc Granule: 4KB 2024-12-18T03:39:50.9621169Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9622285Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9623271Z Accessible by all: TRUE 2024-12-18T03:39:50.9624031Z ISA Info: 2024-12-18T03:39:50.9624841Z ******* 2024-12-18T03:39:50.9625550Z Agent 2 2024-12-18T03:39:50.9626101Z ******* 2024-12-18T03:39:50.9626977Z Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:39:50.9628251Z Uuid: CPU-XX 2024-12-18T03:39:50.9629209Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:39:50.9630287Z Vendor Name: CPU 2024-12-18T03:39:50.9631230Z Feature: None specified 2024-12-18T03:39:50.9632036Z Profile: FULL_PROFILE 2024-12-18T03:39:50.9633486Z Float Round Mode: NEAR 2024-12-18T03:39:50.9634412Z Max Queue Number: 0(0x0) 2024-12-18T03:39:50.9635280Z Queue Min Size: 0(0x0) 2024-12-18T03:39:50.9636477Z Queue Max Size: 0(0x0) 2024-12-18T03:39:50.9637527Z Queue Type: MULTI 2024-12-18T03:39:50.9638355Z Node: 1 2024-12-18T03:39:50.9639357Z Device Type: CPU 2024-12-18T03:39:50.9640186Z Cache Info: 2024-12-18T03:39:50.9641002Z L1: 32768(0x8000) KB 2024-12-18T03:39:50.9641949Z Chip ID: 0(0x0) 2024-12-18T03:39:50.9642826Z ASIC Revision: 0(0x0) 2024-12-18T03:39:50.9643695Z Cacheline Size: 64(0x40) 2024-12-18T03:39:50.9644739Z Max Clock Freq. (MHz): 2000 2024-12-18T03:39:50.9645600Z BDFID: 0 2024-12-18T03:39:50.9646472Z Internal Node ID: 1 2024-12-18T03:39:50.9647443Z Compute Unit: 64 2024-12-18T03:39:50.9648321Z SIMDs per CU: 0 2024-12-18T03:39:50.9649212Z Shader Engines: 0 2024-12-18T03:39:50.9650306Z Shader Arrs. per Eng.: 0 2024-12-18T03:39:50.9651288Z WatchPts on Addr. Ranges:1 2024-12-18T03:39:50.9652151Z Memory Properties: 2024-12-18T03:39:50.9653050Z Features: None 2024-12-18T03:39:50.9653763Z Pool Info: 2024-12-18T03:39:50.9654531Z Pool 1 2024-12-18T03:39:50.9655819Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:50.9656687Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:39:50.9657541Z Allocatable: TRUE 2024-12-18T03:39:50.9658645Z Alloc Granule: 4KB 2024-12-18T03:39:50.9659556Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9660494Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9661630Z Accessible by all: TRUE 2024-12-18T03:39:50.9662567Z Pool 2 2024-12-18T03:39:50.9663330Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:39:50.9664437Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:39:50.9665225Z Allocatable: TRUE 2024-12-18T03:39:50.9666154Z Alloc Granule: 4KB 2024-12-18T03:39:50.9667362Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9668358Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9669194Z Accessible by all: TRUE 2024-12-18T03:39:50.9670260Z Pool 3 2024-12-18T03:39:50.9671066Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:50.9672185Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:39:50.9673598Z Allocatable: TRUE 2024-12-18T03:39:50.9674528Z Alloc Granule: 4KB 2024-12-18T03:39:50.9675363Z Alloc Recommended Granule:4KB 2024-12-18T03:39:50.9676742Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9677842Z Accessible by all: TRUE 2024-12-18T03:39:50.9678606Z ISA Info: 2024-12-18T03:39:50.9679450Z ******* 2024-12-18T03:39:50.9680106Z Agent 3 2024-12-18T03:39:50.9680669Z ******* 2024-12-18T03:39:50.9681552Z Name: gfx90a 2024-12-18T03:39:50.9682430Z Uuid: GPU-87b565030406a196 2024-12-18T03:39:50.9683266Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:50.9684520Z Vendor Name: AMD 2024-12-18T03:39:50.9685474Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:50.9686512Z Profile: BASE_PROFILE 2024-12-18T03:39:50.9687476Z Float Round Mode: NEAR 2024-12-18T03:39:50.9688360Z Max Queue Number: 128(0x80) 2024-12-18T03:39:50.9689367Z Queue Min Size: 64(0x40) 2024-12-18T03:39:50.9690307Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:50.9691156Z Queue Type: MULTI 2024-12-18T03:39:50.9692095Z Node: 2 2024-12-18T03:39:50.9693028Z Device Type: GPU 2024-12-18T03:39:50.9694052Z Cache Info: 2024-12-18T03:39:50.9695284Z L1: 16(0x10) KB 2024-12-18T03:39:50.9696306Z L2: 8192(0x2000) KB 2024-12-18T03:39:50.9697136Z Chip ID: 29708(0x740c) 2024-12-18T03:39:50.9698029Z ASIC Revision: 1(0x1) 2024-12-18T03:39:50.9699030Z Cacheline Size: 64(0x40) 2024-12-18T03:39:50.9700295Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:50.9701221Z BDFID: 12800 2024-12-18T03:39:50.9702197Z Internal Node ID: 2 2024-12-18T03:39:50.9703100Z Compute Unit: 104 2024-12-18T03:39:50.9703997Z SIMDs per CU: 4 2024-12-18T03:39:50.9704996Z Shader Engines: 8 2024-12-18T03:39:50.9705928Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:50.9707055Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:50.9708103Z Coherent Host Access: FALSE 2024-12-18T03:39:50.9708970Z Memory Properties: 2024-12-18T03:39:50.9709870Z Features: KERNEL_DISPATCH 2024-12-18T03:39:50.9710660Z Fast F16 Operation: TRUE 2024-12-18T03:39:50.9711597Z Wavefront Size: 64(0x40) 2024-12-18T03:39:50.9712797Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:50.9713596Z Workgroup Max Size per Dimension: 2024-12-18T03:39:50.9714381Z x 1024(0x400) 2024-12-18T03:39:50.9715356Z y 1024(0x400) 2024-12-18T03:39:50.9716148Z z 1024(0x400) 2024-12-18T03:39:50.9717775Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:50.9718837Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:50.9719668Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:50.9720562Z Grid Max Size per Dimension: 2024-12-18T03:39:50.9721423Z x 4294967295(0xffffffff) 2024-12-18T03:39:50.9722140Z y 4294967295(0xffffffff) 2024-12-18T03:39:50.9723009Z z 4294967295(0xffffffff) 2024-12-18T03:39:50.9724027Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:50.9731329Z Packet Processor uCode:: 83 2024-12-18T03:39:50.9732839Z SDMA engine uCode:: 8 2024-12-18T03:39:50.9734054Z IOMMU Support:: None 2024-12-18T03:39:50.9734981Z Pool Info: 2024-12-18T03:39:50.9735941Z Pool 1 2024-12-18T03:39:50.9736796Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:50.9737748Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9738748Z Allocatable: TRUE 2024-12-18T03:39:50.9739694Z Alloc Granule: 4KB 2024-12-18T03:39:50.9740681Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9741733Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9742674Z Accessible by all: FALSE 2024-12-18T03:39:50.9743556Z Pool 2 2024-12-18T03:39:50.9744455Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:50.9745381Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9746306Z Allocatable: TRUE 2024-12-18T03:39:50.9747555Z Alloc Granule: 4KB 2024-12-18T03:39:50.9748482Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9749447Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9750567Z Accessible by all: FALSE 2024-12-18T03:39:50.9751381Z Pool 3 2024-12-18T03:39:50.9752899Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:50.9754156Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9754986Z Allocatable: TRUE 2024-12-18T03:39:50.9755878Z Alloc Granule: 4KB 2024-12-18T03:39:50.9757058Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9757994Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9759086Z Accessible by all: FALSE 2024-12-18T03:39:50.9760157Z Pool 4 2024-12-18T03:39:50.9760854Z Segment: GROUP 2024-12-18T03:39:50.9761687Z Size: 64(0x40) KB 2024-12-18T03:39:50.9762792Z Allocatable: FALSE 2024-12-18T03:39:50.9763746Z Alloc Granule: 0KB 2024-12-18T03:39:50.9764586Z Alloc Recommended Granule:0KB 2024-12-18T03:39:50.9765772Z Alloc Alignment: 0KB 2024-12-18T03:39:50.9766720Z Accessible by all: FALSE 2024-12-18T03:39:50.9767445Z ISA Info: 2024-12-18T03:39:50.9768348Z ISA 1 2024-12-18T03:39:50.9769171Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:50.9770568Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:50.9771730Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:50.9772689Z Default Rounding Mode: NEAR 2024-12-18T03:39:50.9773587Z Default Rounding Mode: NEAR 2024-12-18T03:39:50.9774701Z Fast f16: TRUE 2024-12-18T03:39:50.9775670Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:50.9776429Z Workgroup Max Size per Dimension: 2024-12-18T03:39:50.9777438Z x 1024(0x400) 2024-12-18T03:39:50.9778306Z y 1024(0x400) 2024-12-18T03:39:50.9779211Z z 1024(0x400) 2024-12-18T03:39:50.9780051Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:50.9781090Z Grid Max Size per Dimension: 2024-12-18T03:39:50.9781952Z x 4294967295(0xffffffff) 2024-12-18T03:39:50.9782712Z y 4294967295(0xffffffff) 2024-12-18T03:39:50.9783563Z z 4294967295(0xffffffff) 2024-12-18T03:39:50.9784551Z FBarrier Max Size: 32 2024-12-18T03:39:50.9785339Z ******* 2024-12-18T03:39:50.9786042Z Agent 4 2024-12-18T03:39:50.9786822Z ******* 2024-12-18T03:39:50.9787497Z Name: gfx90a 2024-12-18T03:39:50.9788416Z Uuid: GPU-72b52101971d0607 2024-12-18T03:39:50.9789441Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:50.9790362Z Vendor Name: AMD 2024-12-18T03:39:50.9791407Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:50.9792551Z Profile: BASE_PROFILE 2024-12-18T03:39:50.9793493Z Float Round Mode: NEAR 2024-12-18T03:39:50.9794427Z Max Queue Number: 128(0x80) 2024-12-18T03:39:50.9795424Z Queue Min Size: 64(0x40) 2024-12-18T03:39:50.9796941Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:50.9798025Z Queue Type: MULTI 2024-12-18T03:39:50.9799162Z Node: 3 2024-12-18T03:39:50.9800073Z Device Type: GPU 2024-12-18T03:39:50.9800811Z Cache Info: 2024-12-18T03:39:50.9801676Z L1: 16(0x10) KB 2024-12-18T03:39:50.9802697Z L2: 8192(0x2000) KB 2024-12-18T03:39:50.9803461Z Chip ID: 29708(0x740c) 2024-12-18T03:39:50.9804491Z ASIC Revision: 1(0x1) 2024-12-18T03:39:50.9805463Z Cacheline Size: 64(0x40) 2024-12-18T03:39:50.9806301Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:50.9807326Z BDFID: 13568 2024-12-18T03:39:50.9808259Z Internal Node ID: 3 2024-12-18T03:39:50.9809074Z Compute Unit: 104 2024-12-18T03:39:50.9810104Z SIMDs per CU: 4 2024-12-18T03:39:50.9811055Z Shader Engines: 8 2024-12-18T03:39:50.9811939Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:50.9813384Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:50.9814573Z Coherent Host Access: FALSE 2024-12-18T03:39:50.9815501Z Memory Properties: 2024-12-18T03:39:50.9816243Z Features: KERNEL_DISPATCH 2024-12-18T03:39:50.9817193Z Fast F16 Operation: TRUE 2024-12-18T03:39:50.9818143Z Wavefront Size: 64(0x40) 2024-12-18T03:39:50.9819084Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:50.9820042Z Workgroup Max Size per Dimension: 2024-12-18T03:39:50.9820825Z x 1024(0x400) 2024-12-18T03:39:50.9821620Z y 1024(0x400) 2024-12-18T03:39:50.9822500Z z 1024(0x400) 2024-12-18T03:39:50.9823365Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:50.9824298Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:50.9825436Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:50.9826292Z Grid Max Size per Dimension: 2024-12-18T03:39:50.9827031Z x 4294967295(0xffffffff) 2024-12-18T03:39:50.9827944Z y 4294967295(0xffffffff) 2024-12-18T03:39:50.9828754Z z 4294967295(0xffffffff) 2024-12-18T03:39:50.9829660Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:50.9830748Z Packet Processor uCode:: 83 2024-12-18T03:39:50.9831713Z SDMA engine uCode:: 8 2024-12-18T03:39:50.9833115Z IOMMU Support:: None 2024-12-18T03:39:50.9833875Z Pool Info: 2024-12-18T03:39:50.9834567Z Pool 1 2024-12-18T03:39:50.9835703Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:50.9836544Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9837463Z Allocatable: TRUE 2024-12-18T03:39:50.9838573Z Alloc Granule: 4KB 2024-12-18T03:39:50.9839457Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9840818Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9841947Z Accessible by all: FALSE 2024-12-18T03:39:50.9842717Z Pool 2 2024-12-18T03:39:50.9843569Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:50.9844612Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9845417Z Allocatable: TRUE 2024-12-18T03:39:50.9846534Z Alloc Granule: 4KB 2024-12-18T03:39:50.9847624Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9848633Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9849496Z Accessible by all: FALSE 2024-12-18T03:39:50.9850473Z Pool 3 2024-12-18T03:39:50.9851301Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:50.9852127Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:50.9853168Z Allocatable: TRUE 2024-12-18T03:39:50.9854142Z Alloc Granule: 4KB 2024-12-18T03:39:50.9855008Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:50.9856090Z Alloc Alignment: 4KB 2024-12-18T03:39:50.9857439Z Accessible by all: FALSE 2024-12-18T03:39:50.9858337Z Pool 4 2024-12-18T03:39:50.9859275Z Segment: GROUP 2024-12-18T03:39:50.9860141Z Size: 64(0x40) KB 2024-12-18T03:39:50.9860936Z Allocatable: FALSE 2024-12-18T03:39:50.9862032Z Alloc Granule: 0KB 2024-12-18T03:39:50.9863005Z Alloc Recommended Granule:0KB 2024-12-18T03:39:50.9863932Z Alloc Alignment: 0KB 2024-12-18T03:39:50.9865000Z Accessible by all: FALSE 2024-12-18T03:39:50.9865842Z ISA Info: 2024-12-18T03:39:50.9866481Z ISA 1 2024-12-18T03:39:50.9867404Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:50.9868538Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:50.9869469Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:50.9870540Z Default Rounding Mode: NEAR 2024-12-18T03:39:50.9871510Z Default Rounding Mode: NEAR 2024-12-18T03:39:50.9872758Z Fast f16: TRUE 2024-12-18T03:39:50.9873808Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:50.9874735Z Workgroup Max Size per Dimension: 2024-12-18T03:39:50.9875564Z x 1024(0x400) 2024-12-18T03:39:50.9876558Z y 1024(0x400) 2024-12-18T03:39:50.9877282Z z 1024(0x400) 2024-12-18T03:39:50.9878157Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:50.9879221Z Grid Max Size per Dimension: 2024-12-18T03:39:50.9880211Z x 4294967295(0xffffffff) 2024-12-18T03:39:50.9881035Z y 4294967295(0xffffffff) 2024-12-18T03:39:50.9882011Z z 4294967295(0xffffffff) 2024-12-18T03:39:50.9882806Z FBarrier Max Size: 32 2024-12-18T03:39:50.9883936Z ******* 2024-12-18T03:39:51.0124462Z Agent 5 2024-12-18T03:39:51.0124937Z ******* 2024-12-18T03:39:51.0125385Z Name: gfx90a 2024-12-18T03:39:51.0125985Z Uuid: GPU-04e3cd7c6eb6c52d 2024-12-18T03:39:51.0126609Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0127215Z Vendor Name: AMD 2024-12-18T03:39:51.0127827Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0128434Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0129053Z Float Round Mode: NEAR 2024-12-18T03:39:51.0129656Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0130256Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0130837Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0131421Z Queue Type: MULTI 2024-12-18T03:39:51.0131973Z Node: 4 2024-12-18T03:39:51.0132508Z Device Type: GPU 2024-12-18T03:39:51.0133028Z Cache Info: 2024-12-18T03:39:51.0133469Z L1: 16(0x10) KB 2024-12-18T03:39:51.0134038Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0135049Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0135694Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0136305Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0136906Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0137461Z BDFID: 4352 2024-12-18T03:39:51.0138026Z Internal Node ID: 4 2024-12-18T03:39:51.0138618Z Compute Unit: 104 2024-12-18T03:39:51.0139188Z SIMDs per CU: 4 2024-12-18T03:39:51.0139758Z Shader Engines: 8 2024-12-18T03:39:51.0140367Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0140986Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0141648Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0142217Z Memory Properties: 2024-12-18T03:39:51.0142672Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0143241Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0143861Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0144463Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0145035Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0145527Z x 1024(0x400) 2024-12-18T03:39:51.0146042Z y 1024(0x400) 2024-12-18T03:39:51.0146540Z z 1024(0x400) 2024-12-18T03:39:51.0147078Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0147677Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0148288Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0148809Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0149245Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0149735Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0150214Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0151041Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0151749Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0152860Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0153479Z IOMMU Support:: None 2024-12-18T03:39:51.0154016Z Pool Info: 2024-12-18T03:39:51.0154424Z Pool 1 2024-12-18T03:39:51.0154957Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0155578Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0156148Z Allocatable: TRUE 2024-12-18T03:39:51.0156749Z Alloc Granule: 4KB 2024-12-18T03:39:51.0157380Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0158093Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0158748Z Accessible by all: FALSE 2024-12-18T03:39:51.0159281Z Pool 2 2024-12-18T03:39:51.0159772Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0160383Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0160981Z Allocatable: TRUE 2024-12-18T03:39:51.0161588Z Alloc Granule: 4KB 2024-12-18T03:39:51.0162567Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0163220Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0163887Z Accessible by all: FALSE 2024-12-18T03:39:51.0164512Z Pool 3 2024-12-18T03:39:51.0165089Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0165771Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0166441Z Allocatable: TRUE 2024-12-18T03:39:51.0167188Z Alloc Granule: 4KB 2024-12-18T03:39:51.0167968Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0168762Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0169528Z Accessible by all: FALSE 2024-12-18T03:39:51.0169873Z Pool 4 2024-12-18T03:39:51.0170123Z Segment: GROUP 2024-12-18T03:39:51.0170398Z Size: 64(0x40) KB 2024-12-18T03:39:51.0170665Z Allocatable: FALSE 2024-12-18T03:39:51.0170940Z Alloc Granule: 0KB 2024-12-18T03:39:51.0171241Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0171538Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0171820Z Accessible by all: FALSE 2024-12-18T03:39:51.0172073Z ISA Info: 2024-12-18T03:39:51.0172258Z ISA 1 2024-12-18T03:39:51.0172489Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0172796Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0173099Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0173388Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0173689Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0173972Z Fast f16: TRUE 2024-12-18T03:39:51.0174248Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0174653Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0174898Z x 1024(0x400) 2024-12-18T03:39:51.0175133Z y 1024(0x400) 2024-12-18T03:39:51.0175370Z z 1024(0x400) 2024-12-18T03:39:51.0175628Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0175883Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0176102Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0176339Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0176572Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0176829Z FBarrier Max Size: 32 2024-12-18T03:39:51.0177074Z ******* 2024-12-18T03:39:51.0177246Z Agent 6 2024-12-18T03:39:51.0177416Z ******* 2024-12-18T03:39:51.0177618Z Name: gfx90a 2024-12-18T03:39:51.0177878Z Uuid: GPU-e46a5b3c036a0534 2024-12-18T03:39:51.0178156Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0178442Z Vendor Name: AMD 2024-12-18T03:39:51.0178714Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0179108Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0179389Z Float Round Mode: NEAR 2024-12-18T03:39:51.0179671Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0179941Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0180211Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0180487Z Queue Type: MULTI 2024-12-18T03:39:51.0180738Z Node: 5 2024-12-18T03:39:51.0180997Z Device Type: GPU 2024-12-18T03:39:51.0181239Z Cache Info: 2024-12-18T03:39:51.0181436Z L1: 16(0x10) KB 2024-12-18T03:39:51.0181676Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0181920Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0182188Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0182469Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0182751Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0183010Z BDFID: 5120 2024-12-18T03:39:51.0183276Z Internal Node ID: 5 2024-12-18T03:39:51.0183556Z Compute Unit: 104 2024-12-18T03:39:51.0183818Z SIMDs per CU: 4 2024-12-18T03:39:51.0184090Z Shader Engines: 8 2024-12-18T03:39:51.0184370Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0184656Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0184956Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0185224Z Memory Properties: 2024-12-18T03:39:51.0185429Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0185691Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0185976Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0186257Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0186522Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0186863Z x 1024(0x400) 2024-12-18T03:39:51.0187098Z y 1024(0x400) 2024-12-18T03:39:51.0187329Z z 1024(0x400) 2024-12-18T03:39:51.0187584Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0187868Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0188140Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0188395Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0188600Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0188829Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0189062Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0189328Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0189631Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0189924Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0190205Z IOMMU Support:: None 2024-12-18T03:39:51.0190446Z Pool Info: 2024-12-18T03:39:51.0190630Z Pool 1 2024-12-18T03:39:51.0190863Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0191137Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0191549Z Allocatable: TRUE 2024-12-18T03:39:51.0191835Z Alloc Granule: 4KB 2024-12-18T03:39:51.0192129Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0192488Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0192778Z Accessible by all: FALSE 2024-12-18T03:39:51.0193030Z Pool 2 2024-12-18T03:39:51.0193260Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0193539Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0193802Z Allocatable: TRUE 2024-12-18T03:39:51.0194085Z Alloc Granule: 4KB 2024-12-18T03:39:51.0194380Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0194683Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0194977Z Accessible by all: FALSE 2024-12-18T03:39:51.0195228Z Pool 3 2024-12-18T03:39:51.0195445Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0195715Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0195988Z Allocatable: TRUE 2024-12-18T03:39:51.0196272Z Alloc Granule: 4KB 2024-12-18T03:39:51.0196561Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0196856Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0197146Z Accessible by all: FALSE 2024-12-18T03:39:51.0197391Z Pool 4 2024-12-18T03:39:51.0197611Z Segment: GROUP 2024-12-18T03:39:51.0197869Z Size: 64(0x40) KB 2024-12-18T03:39:51.0198131Z Allocatable: FALSE 2024-12-18T03:39:51.0198413Z Alloc Granule: 0KB 2024-12-18T03:39:51.0198710Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0199144Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0199458Z Accessible by all: FALSE 2024-12-18T03:39:51.0199723Z ISA Info: 2024-12-18T03:39:51.0199911Z ISA 1 2024-12-18T03:39:51.0200155Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0200473Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0200767Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0201075Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0201381Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0201659Z Fast f16: TRUE 2024-12-18T03:39:51.0201942Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0202218Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0202455Z x 1024(0x400) 2024-12-18T03:39:51.0202700Z y 1024(0x400) 2024-12-18T03:39:51.0202931Z z 1024(0x400) 2024-12-18T03:39:51.0203190Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0203441Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0203659Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0204038Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0204281Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0204546Z FBarrier Max Size: 32 2024-12-18T03:39:51.0204792Z ******* 2024-12-18T03:39:51.0204966Z Agent 7 2024-12-18T03:39:51.0205139Z ******* 2024-12-18T03:39:51.0205346Z Name: gfx90a 2024-12-18T03:39:51.0205607Z Uuid: GPU-ef2ed32bc2e35058 2024-12-18T03:39:51.0205894Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0206177Z Vendor Name: AMD 2024-12-18T03:39:51.0206449Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0206725Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0207013Z Float Round Mode: NEAR 2024-12-18T03:39:51.0207292Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0207570Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0207840Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0208109Z Queue Type: MULTI 2024-12-18T03:39:51.0208372Z Node: 6 2024-12-18T03:39:51.0208634Z Device Type: GPU 2024-12-18T03:39:51.0208875Z Cache Info: 2024-12-18T03:39:51.0209082Z L1: 16(0x10) KB 2024-12-18T03:39:51.0209331Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0209575Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0209843Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0210134Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0210409Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0210681Z BDFID: 44544 2024-12-18T03:39:51.0210952Z Internal Node ID: 6 2024-12-18T03:39:51.0211222Z Compute Unit: 104 2024-12-18T03:39:51.0211612Z SIMDs per CU: 4 2024-12-18T03:39:51.0211896Z Shader Engines: 8 2024-12-18T03:39:51.0212176Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0212473Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0212779Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0213034Z Memory Properties: 2024-12-18T03:39:51.0213253Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0213519Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0213802Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0214092Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0214357Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0214585Z x 1024(0x400) 2024-12-18T03:39:51.0214822Z y 1024(0x400) 2024-12-18T03:39:51.0215054Z z 1024(0x400) 2024-12-18T03:39:51.0215314Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0215593Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0215875Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0216124Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0216448Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0216689Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0216921Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0217186Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0217493Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0217793Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0218074Z IOMMU Support:: None 2024-12-18T03:39:51.0218322Z Pool Info: 2024-12-18T03:39:51.0218506Z Pool 1 2024-12-18T03:39:51.0218734Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0219014Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0219288Z Allocatable: TRUE 2024-12-18T03:39:51.0219578Z Alloc Granule: 4KB 2024-12-18T03:39:51.0219881Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0220187Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0220474Z Accessible by all: FALSE 2024-12-18T03:39:51.0220727Z Pool 2 2024-12-18T03:39:51.0220961Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0221231Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0221505Z Allocatable: TRUE 2024-12-18T03:39:51.0221792Z Alloc Granule: 4KB 2024-12-18T03:39:51.0222085Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0222384Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0222687Z Accessible by all: FALSE 2024-12-18T03:39:51.0222936Z Pool 3 2024-12-18T03:39:51.0223154Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0223426Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0223700Z Allocatable: TRUE 2024-12-18T03:39:51.0224089Z Alloc Granule: 4KB 2024-12-18T03:39:51.0224395Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0224701Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0224992Z Accessible by all: FALSE 2024-12-18T03:39:51.0225245Z Pool 4 2024-12-18T03:39:51.0225470Z Segment: GROUP 2024-12-18T03:39:51.0225733Z Size: 64(0x40) KB 2024-12-18T03:39:51.0226006Z Allocatable: FALSE 2024-12-18T03:39:51.0226296Z Alloc Granule: 0KB 2024-12-18T03:39:51.0226589Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0226895Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0227195Z Accessible by all: FALSE 2024-12-18T03:39:51.0227448Z ISA Info: 2024-12-18T03:39:51.0227630Z ISA 1 2024-12-18T03:39:51.0227875Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0228182Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0228482Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0228780Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0229207Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0229506Z Fast f16: TRUE 2024-12-18T03:39:51.0229801Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0230083Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0230328Z x 1024(0x400) 2024-12-18T03:39:51.0230591Z y 1024(0x400) 2024-12-18T03:39:51.0230827Z z 1024(0x400) 2024-12-18T03:39:51.0231101Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0231370Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0231589Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0231841Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0232105Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0232436Z FBarrier Max Size: 32 2024-12-18T03:39:51.0232682Z ******* 2024-12-18T03:39:51.0232874Z Agent 8 2024-12-18T03:39:51.0233047Z ******* 2024-12-18T03:39:51.0233254Z Name: gfx90a 2024-12-18T03:39:51.0233538Z Uuid: GPU-f363aff83c4a8ce4 2024-12-18T03:39:51.0233832Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0234121Z Vendor Name: AMD 2024-12-18T03:39:51.0234400Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0234683Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0234963Z Float Round Mode: NEAR 2024-12-18T03:39:51.0235263Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0235547Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0235818Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0236094Z Queue Type: MULTI 2024-12-18T03:39:51.0236360Z Node: 7 2024-12-18T03:39:51.0236775Z Device Type: GPU 2024-12-18T03:39:51.0237029Z Cache Info: 2024-12-18T03:39:51.0237239Z L1: 16(0x10) KB 2024-12-18T03:39:51.0237479Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0237734Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0238008Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0238299Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0238593Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0238865Z BDFID: 45824 2024-12-18T03:39:51.0239129Z Internal Node ID: 7 2024-12-18T03:39:51.0239413Z Compute Unit: 104 2024-12-18T03:39:51.0239696Z SIMDs per CU: 4 2024-12-18T03:39:51.0239972Z Shader Engines: 8 2024-12-18T03:39:51.0240267Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0240565Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0240859Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0241131Z Memory Properties: 2024-12-18T03:39:51.0241345Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0241752Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0242045Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0242340Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0242600Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0242829Z x 1024(0x400) 2024-12-18T03:39:51.0243073Z y 1024(0x400) 2024-12-18T03:39:51.0243303Z z 1024(0x400) 2024-12-18T03:39:51.0243562Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0243857Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0244141Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0244399Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0244612Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0244864Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0245095Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0245367Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0245676Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0245966Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0246257Z IOMMU Support:: None 2024-12-18T03:39:51.0246506Z Pool Info: 2024-12-18T03:39:51.0246685Z Pool 1 2024-12-18T03:39:51.0246918Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0247200Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0247473Z Allocatable: TRUE 2024-12-18T03:39:51.0247765Z Alloc Granule: 4KB 2024-12-18T03:39:51.0248072Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0248371Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0248666Z Accessible by all: FALSE 2024-12-18T03:39:51.0248920Z Pool 2 2024-12-18T03:39:51.0249145Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0249541Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0249827Z Allocatable: TRUE 2024-12-18T03:39:51.0250111Z Alloc Granule: 4KB 2024-12-18T03:39:51.0250411Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0250714Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0251005Z Accessible by all: FALSE 2024-12-18T03:39:51.0251268Z Pool 3 2024-12-18T03:39:51.0251509Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0251786Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0252059Z Allocatable: TRUE 2024-12-18T03:39:51.0252341Z Alloc Granule: 4KB 2024-12-18T03:39:51.0252648Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0252947Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0253238Z Accessible by all: FALSE 2024-12-18T03:39:51.0253494Z Pool 4 2024-12-18T03:39:51.0253705Z Segment: GROUP 2024-12-18T03:39:51.0253967Z Size: 64(0x40) KB 2024-12-18T03:39:51.0254355Z Allocatable: FALSE 2024-12-18T03:39:51.0254639Z Alloc Granule: 0KB 2024-12-18T03:39:51.0254937Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0255233Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0255519Z Accessible by all: FALSE 2024-12-18T03:39:51.0255778Z ISA Info: 2024-12-18T03:39:51.0255958Z ISA 1 2024-12-18T03:39:51.0256183Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0256487Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0256778Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0257062Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0257374Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0257651Z Fast f16: TRUE 2024-12-18T03:39:51.0257924Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0258191Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0258430Z x 1024(0x400) 2024-12-18T03:39:51.0258669Z y 1024(0x400) 2024-12-18T03:39:51.0258903Z z 1024(0x400) 2024-12-18T03:39:51.0259163Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0259414Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0259629Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0259874Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0260108Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0260390Z FBarrier Max Size: 32 2024-12-18T03:39:51.0260641Z ******* 2024-12-18T03:39:51.0260811Z Agent 9 2024-12-18T03:39:51.0260985Z ******* 2024-12-18T03:39:51.0261189Z Name: gfx90a 2024-12-18T03:39:51.0261446Z Uuid: GPU-1ece649c612e95a2 2024-12-18T03:39:51.0261857Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0262157Z Vendor Name: AMD 2024-12-18T03:39:51.0262429Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0262704Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0262984Z Float Round Mode: NEAR 2024-12-18T03:39:51.0263266Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0263547Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0263834Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0264111Z Queue Type: MULTI 2024-12-18T03:39:51.0264364Z Node: 8 2024-12-18T03:39:51.0264629Z Device Type: GPU 2024-12-18T03:39:51.0264878Z Cache Info: 2024-12-18T03:39:51.0265080Z L1: 16(0x10) KB 2024-12-18T03:39:51.0265322Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0265576Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0265843Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0266127Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0266544Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0266819Z BDFID: 36352 2024-12-18T03:39:51.0267114Z Internal Node ID: 8 2024-12-18T03:39:51.0267409Z Compute Unit: 104 2024-12-18T03:39:51.0267683Z SIMDs per CU: 4 2024-12-18T03:39:51.0267981Z Shader Engines: 8 2024-12-18T03:39:51.0268285Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0268584Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0268900Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0269188Z Memory Properties: 2024-12-18T03:39:51.0269403Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0269691Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0270007Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0270299Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0270581Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0270825Z x 1024(0x400) 2024-12-18T03:39:51.0271065Z y 1024(0x400) 2024-12-18T03:39:51.0271318Z z 1024(0x400) 2024-12-18T03:39:51.0271596Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0271886Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0272192Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0272586Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0272804Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0273063Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0273348Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0273624Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0273949Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0274267Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0274574Z IOMMU Support:: None 2024-12-18T03:39:51.0274979Z Pool Info: 2024-12-18T03:39:51.0275189Z Pool 1 2024-12-18T03:39:51.0275449Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0275737Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0276032Z Allocatable: TRUE 2024-12-18T03:39:51.0276336Z Alloc Granule: 4KB 2024-12-18T03:39:51.0276678Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0277056Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0277493Z Accessible by all: FALSE 2024-12-18T03:39:51.0277855Z Pool 2 2024-12-18T03:39:51.0278202Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0278581Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0278911Z Allocatable: TRUE 2024-12-18T03:39:51.0279266Z Alloc Granule: 4KB 2024-12-18T03:39:51.0279635Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0279998Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0280363Z Accessible by all: FALSE 2024-12-18T03:39:51.0280850Z Pool 3 2024-12-18T03:39:51.0281143Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0281500Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0281846Z Allocatable: TRUE 2024-12-18T03:39:51.0282190Z Alloc Granule: 4KB 2024-12-18T03:39:51.0282572Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0282959Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0283311Z Accessible by all: FALSE 2024-12-18T03:39:51.0283628Z Pool 4 2024-12-18T03:39:51.0283902Z Segment: GROUP 2024-12-18T03:39:51.0284220Z Size: 64(0x40) KB 2024-12-18T03:39:51.0284545Z Allocatable: FALSE 2024-12-18T03:39:51.0284906Z Alloc Granule: 0KB 2024-12-18T03:39:51.0285272Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0285635Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0285994Z Accessible by all: FALSE 2024-12-18T03:39:51.0286315Z ISA Info: 2024-12-18T03:39:51.0286546Z ISA 1 2024-12-18T03:39:51.0286840Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0287222Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0287582Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0287946Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0288323Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0288658Z Fast f16: TRUE 2024-12-18T03:39:51.0289004Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0289343Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0289625Z x 1024(0x400) 2024-12-18T03:39:51.0289926Z y 1024(0x400) 2024-12-18T03:39:51.0290380Z z 1024(0x400) 2024-12-18T03:39:51.0290702Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0291022Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0291298Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0291592Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0291890Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0292232Z FBarrier Max Size: 32 2024-12-18T03:39:51.0292531Z ******* 2024-12-18T03:39:51.0292757Z Agent 10 2024-12-18T03:39:51.0292977Z ******* 2024-12-18T03:39:51.0293220Z Name: gfx90a 2024-12-18T03:39:51.0293553Z Uuid: GPU-d3bee6119563da6a 2024-12-18T03:39:51.0293912Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:39:51.0294317Z Vendor Name: AMD 2024-12-18T03:39:51.0294713Z Feature: KERNEL_DISPATCH 2024-12-18T03:39:51.0295109Z Profile: BASE_PROFILE 2024-12-18T03:39:51.0295510Z Float Round Mode: NEAR 2024-12-18T03:39:51.0295909Z Max Queue Number: 128(0x80) 2024-12-18T03:39:51.0296246Z Queue Min Size: 64(0x40) 2024-12-18T03:39:51.0296672Z Queue Max Size: 131072(0x20000) 2024-12-18T03:39:51.0296955Z Queue Type: MULTI 2024-12-18T03:39:51.0297221Z Node: 9 2024-12-18T03:39:51.0297489Z Device Type: GPU 2024-12-18T03:39:51.0297735Z Cache Info: 2024-12-18T03:39:51.0297964Z L1: 16(0x10) KB 2024-12-18T03:39:51.0298223Z L2: 8192(0x2000) KB 2024-12-18T03:39:51.0298470Z Chip ID: 29708(0x740c) 2024-12-18T03:39:51.0298741Z ASIC Revision: 1(0x1) 2024-12-18T03:39:51.0299046Z Cacheline Size: 64(0x40) 2024-12-18T03:39:51.0299343Z Max Clock Freq. (MHz): 1700 2024-12-18T03:39:51.0299627Z BDFID: 37632 2024-12-18T03:39:51.0299900Z Internal Node ID: 9 2024-12-18T03:39:51.0300175Z Compute Unit: 104 2024-12-18T03:39:51.0300447Z SIMDs per CU: 4 2024-12-18T03:39:51.0300729Z Shader Engines: 8 2024-12-18T03:39:51.0301018Z Shader Arrs. per Eng.: 1 2024-12-18T03:39:51.0301316Z WatchPts on Addr. Ranges:4 2024-12-18T03:39:51.0301621Z Coherent Host Access: FALSE 2024-12-18T03:39:51.0301882Z Memory Properties: 2024-12-18T03:39:51.0302091Z Features: KERNEL_DISPATCH 2024-12-18T03:39:51.0302357Z Fast F16 Operation: TRUE 2024-12-18T03:39:51.0302642Z Wavefront Size: 64(0x40) 2024-12-18T03:39:51.0302945Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0303211Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0303433Z x 1024(0x400) 2024-12-18T03:39:51.0303676Z y 1024(0x400) 2024-12-18T03:39:51.0303912Z z 1024(0x400) 2024-12-18T03:39:51.0304287Z Max Waves Per CU: 32(0x20) 2024-12-18T03:39:51.0304584Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:39:51.0304873Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0305125Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0305340Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0305581Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0305823Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0306099Z Max fbarriers/Workgrp: 32 2024-12-18T03:39:51.0306415Z Packet Processor uCode:: 83 2024-12-18T03:39:51.0306713Z SDMA engine uCode:: 8 2024-12-18T03:39:51.0307006Z IOMMU Support:: None 2024-12-18T03:39:51.0307260Z Pool Info: 2024-12-18T03:39:51.0307459Z Pool 1 2024-12-18T03:39:51.0307700Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:39:51.0308001Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0308349Z Allocatable: TRUE 2024-12-18T03:39:51.0308641Z Alloc Granule: 4KB 2024-12-18T03:39:51.0308959Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0309402Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0309698Z Accessible by all: FALSE 2024-12-18T03:39:51.0309966Z Pool 2 2024-12-18T03:39:51.0310210Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:39:51.0310488Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0310779Z Allocatable: TRUE 2024-12-18T03:39:51.0311083Z Alloc Granule: 4KB 2024-12-18T03:39:51.0311388Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0311698Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0312001Z Accessible by all: FALSE 2024-12-18T03:39:51.0312469Z Pool 3 2024-12-18T03:39:51.0312719Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:39:51.0313004Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:39:51.0313273Z Allocatable: TRUE 2024-12-18T03:39:51.0313582Z Alloc Granule: 4KB 2024-12-18T03:39:51.0313882Z Alloc Recommended Granule:2048KB 2024-12-18T03:39:51.0314193Z Alloc Alignment: 4KB 2024-12-18T03:39:51.0314502Z Accessible by all: FALSE 2024-12-18T03:39:51.0314772Z Pool 4 2024-12-18T03:39:51.0314996Z Segment: GROUP 2024-12-18T03:39:51.0315277Z Size: 64(0x40) KB 2024-12-18T03:39:51.0315566Z Allocatable: FALSE 2024-12-18T03:39:51.0315881Z Alloc Granule: 0KB 2024-12-18T03:39:51.0316180Z Alloc Recommended Granule:0KB 2024-12-18T03:39:51.0316494Z Alloc Alignment: 0KB 2024-12-18T03:39:51.0316798Z Accessible by all: FALSE 2024-12-18T03:39:51.0317056Z ISA Info: 2024-12-18T03:39:51.0317255Z ISA 1 2024-12-18T03:39:51.0317652Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:39:51.0317986Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:39:51.0318300Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:39:51.0318609Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0318917Z Default Rounding Mode: NEAR 2024-12-18T03:39:51.0319215Z Fast f16: TRUE 2024-12-18T03:39:51.0319515Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:39:51.0319788Z Workgroup Max Size per Dimension: 2024-12-18T03:39:51.0320036Z x 1024(0x400) 2024-12-18T03:39:51.0320286Z y 1024(0x400) 2024-12-18T03:39:51.0320519Z z 1024(0x400) 2024-12-18T03:39:51.0320780Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:39:51.0321036Z Grid Max Size per Dimension: 2024-12-18T03:39:51.0321249Z x 4294967295(0xffffffff) 2024-12-18T03:39:51.0321490Z y 4294967295(0xffffffff) 2024-12-18T03:39:51.0321736Z z 4294967295(0xffffffff) 2024-12-18T03:39:51.0322000Z FBarrier Max Size: 32 2024-12-18T03:39:51.0322391Z *** Done *** 2024-12-18T03:39:51.0340377Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-12-18T03:39:51.0340725Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2024-12-18T03:39:51.0341257Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T03:39:51.0341759Z if [[ $ngpu -eq 0 ]]; then 2024-12-18T03:39:51.0342037Z  echo "Error: Failed to detect any GPUs on the runner" 2024-12-18T03:39:51.0342313Z  echo "$msg" 2024-12-18T03:39:51.0342490Z  exit 1 2024-12-18T03:39:51.0342661Z fi 2024-12-18T03:39:51.0342836Z if [[ $ngpu -eq 1 ]]; then 2024-12-18T03:39:51.0343153Z  echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs" 2024-12-18T03:39:51.0343483Z  echo "$msg" 2024-12-18T03:39:51.0343659Z  exit 1 2024-12-18T03:39:51.0343831Z fi 2024-12-18T03:39:51.0374595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:51.0374879Z env: 2024-12-18T03:39:51.0375040Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.0375270Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.0375529Z ##[endgroup] 2024-12-18T03:39:51.2147860Z Prepare all required actions 2024-12-18T03:39:51.2208297Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-12-18T03:39:51.2208975Z with: 2024-12-18T03:39:51.2209396Z diskspace-cutoff: 70 2024-12-18T03:39:51.2209840Z env: 2024-12-18T03:39:51.2210238Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.2210808Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.2211447Z ##[endgroup] 2024-12-18T03:39:51.2288094Z ##[group]Run set -ex 2024-12-18T03:39:51.2288549Z set -ex 2024-12-18T03:39:51.2288928Z diskspace_cutoff=70 2024-12-18T03:39:51.2289522Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-12-18T03:39:51.2290488Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T03:39:51.2291922Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T03:39:51.2293116Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-12-18T03:39:51.2293708Z  docker system prune -af 2024-12-18T03:39:51.2294893Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T03:39:51.2295804Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-12-18T03:39:51.2296745Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-12-18T03:39:51.2297558Z  echo "$msg" 2024-12-18T03:39:51.2297971Z  exit 1 2024-12-18T03:39:51.2298363Z  else 2024-12-18T03:39:51.2298788Z  difference=$((diskspace - diskspace_new)) 2024-12-18T03:39:51.2299430Z  echo "Diskspace saved: $difference percent" 2024-12-18T03:39:51.2299960Z  fi 2024-12-18T03:39:51.2300288Z fi 2024-12-18T03:39:51.2362339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:51.2362949Z env: 2024-12-18T03:39:51.2363278Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.2363813Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.2364379Z ##[endgroup] 2024-12-18T03:39:51.2461008Z + diskspace_cutoff=70 2024-12-18T03:39:51.2470447Z ++ docker info -f '{{.DockerRootDir}}' 2024-12-18T03:39:51.3460230Z + docker_root_dir=/media/4TB/docker-rootless 2024-12-18T03:39:51.3475709Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-12-18T03:39:51.3477721Z ++ sed -n 2p 2024-12-18T03:39:51.3510378Z ++ sed s/%// 2024-12-18T03:39:51.3510908Z ++ sed 's/ //' 2024-12-18T03:39:51.3529889Z + diskspace=56 2024-12-18T03:39:51.3531130Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-12-18T03:39:51.3532339Z + [[ 56 -ge 70 ]] 2024-12-18T03:39:51.3599597Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:39:51.3600544Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:39:51.3601337Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-12-18T03:39:51.3669975Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:51.3670651Z env: 2024-12-18T03:39:51.3671031Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.3671585Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.3672204Z ##[endgroup] 2024-12-18T03:39:51.3905406Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-12-18T03:39:51.3906586Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2024-12-18T03:39:51.3907934Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon" >> "${GITHUB_ENV}" 2024-12-18T03:39:51.3964943Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:51.3965558Z env: 2024-12-18T03:39:51.3965893Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.3966402Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.3966975Z ##[endgroup] 2024-12-18T03:39:51.4179815Z ##[group]Run aws-actions/configure-aws-credentials@v4 2024-12-18T03:39:51.4180506Z with: 2024-12-18T03:39:51.4181165Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2024-12-18T03:39:51.4181945Z aws-region: us-east-1 2024-12-18T03:39:51.4182140Z role-duration-seconds: 18000 2024-12-18T03:39:51.4182325Z audience: sts.amazonaws.com 2024-12-18T03:39:51.4182487Z env: 2024-12-18T03:39:51.4182666Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:51.4182972Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:51.4183539Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:51.4183703Z ##[endgroup] 2024-12-18T03:39:51.7349538Z Assuming role with OIDC 2024-12-18T03:39:51.9313585Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2024-12-18T03:39:52.0051892Z ##[group]Run aws-actions/amazon-ecr-login@v2 2024-12-18T03:39:52.0052614Z with: 2024-12-18T03:39:52.0053064Z mask-password: true 2024-12-18T03:39:52.0053575Z registry-type: private 2024-12-18T03:39:52.0054142Z skip-logout: false 2024-12-18T03:39:52.0054671Z env: 2024-12-18T03:39:52.0055150Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:52.0055885Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:52.0057142Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:52.0058178Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:52.0058728Z AWS_REGION: us-east-1 2024-12-18T03:39:52.0059412Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:52.0060137Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:52.0071081Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:52.0071584Z ##[endgroup] 2024-12-18T03:39:52.3610760Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.7674823Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@release/2.6 2024-12-18T03:39:52.7675787Z with: 2024-12-18T03:39:52.7676967Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.7678291Z docker-build-dir: .ci/docker 2024-12-18T03:39:52.7678822Z working-directory: . 2024-12-18T03:39:52.7679482Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.7680197Z force-push: false 2024-12-18T03:39:52.7680629Z env: 2024-12-18T03:39:52.7681533Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:52.7682148Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:52.7683191Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:52.7684132Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:52.7684651Z AWS_REGION: us-east-1 2024-12-18T03:39:52.7685454Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:52.7686174Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:52.7697089Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:52.7697583Z ##[endgroup] 2024-12-18T03:39:52.7736701Z ##[group]Run set -ex 2024-12-18T03:39:52.7737271Z set -ex 2024-12-18T03:39:52.7737688Z  2024-12-18T03:39:52.7738413Z # If the docker build directory or the build script doesn't exist, the action will 2024-12-18T03:39:52.7739575Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-12-18T03:39:52.7740565Z # job could then download the pre-built image as usual 2024-12-18T03:39:52.7741498Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-12-18T03:39:52.7742336Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7743168Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7743893Z  2024-12-18T03:39:52.7744548Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-12-18T03:39:52.7745337Z  exit 0 2024-12-18T03:39:52.7745764Z else 2024-12-18T03:39:52.7746253Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7746845Z fi 2024-12-18T03:39:52.7747245Z  2024-12-18T03:39:52.7747867Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-12-18T03:39:52.7748933Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-12-18T03:39:52.7749909Z  # use it as it is, but first let's extract the tag 2024-12-18T03:39:52.7750763Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-12-18T03:39:52.7751672Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7752744Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7753488Z else 2024-12-18T03:39:52.7754062Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-12-18T03:39:52.7754882Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7756005Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.7756989Z fi 2024-12-18T03:39:52.7823489Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:52.7824202Z env: 2024-12-18T03:39:52.7824619Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:52.7825232Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:52.7826288Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:52.7827253Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:52.7827783Z AWS_REGION: us-east-1 2024-12-18T03:39:52.7828407Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:52.7829600Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:52.7840311Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:52.7840810Z REPO_NAME: pytorch 2024-12-18T03:39:52.7841994Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.7843272Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T03:39:52.7843944Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.7844637Z ##[endgroup] 2024-12-18T03:39:52.7954466Z + [[ ! -d .ci/docker ]] 2024-12-18T03:39:52.7955271Z + [[ ! -f .ci/docker/build.sh ]] 2024-12-18T03:39:52.7956759Z + echo skip=false 2024-12-18T03:39:52.7963958Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-12-18T03:39:52.7981863Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.7983754Z ++ awk -F '[:,]' '{print $2}' 2024-12-18T03:39:52.8045251Z + DOCKER_TAG=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.8046113Z + echo docker-tag=45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.8047600Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.8120726Z ##[group]Run set +e 2024-12-18T03:39:52.8121429Z set +e 2024-12-18T03:39:52.8121931Z set -x 2024-12-18T03:39:52.8122456Z  2024-12-18T03:39:52.8122911Z login() { 2024-12-18T03:39:52.8123942Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T03:39:52.8125068Z } 2024-12-18T03:39:52.8125521Z  2024-12-18T03:39:52.8125989Z retry () { 2024-12-18T03:39:52.8126523Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T03:39:52.8127100Z } 2024-12-18T03:39:52.8127476Z  2024-12-18T03:39:52.8127904Z retry login "${DOCKER_REGISTRY}" 2024-12-18T03:39:52.8128467Z  2024-12-18T03:39:52.8128869Z START_TIME=$(date +%s) 2024-12-18T03:39:52.8129399Z # Wait up to 90 minutes 2024-12-18T03:39:52.8130053Z while [[ $(( $(date +%s) - 5400 )) -lt $START_TIME ]]; do 2024-12-18T03:39:52.8130918Z  # Check if image already exists, if it does then skip building it 2024-12-18T03:39:52.8131877Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-12-18T03:39:52.8132646Z  exit 0 2024-12-18T03:39:52.8133147Z  fi 2024-12-18T03:39:52.8133609Z  2024-12-18T03:39:52.8134403Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2024-12-18T03:39:52.8135594Z  # use this to differentiate between the Docker build and regular build jobs. For the 2024-12-18T03:39:52.8136727Z  # latter, it will wait for the Docker images to become available before continuing 2024-12-18T03:39:52.8137629Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2024-12-18T03:39:52.8138340Z  # It's a Docker build job, let's build the image 2024-12-18T03:39:52.8138948Z  break 2024-12-18T03:39:52.8139373Z  else 2024-12-18T03:39:52.8139979Z  # It's a regular build job, wait for the image to become available 2024-12-18T03:39:52.8140696Z  sleep 300 2024-12-18T03:39:52.8141137Z  fi 2024-12-18T03:39:52.8141544Z done 2024-12-18T03:39:52.8141934Z  2024-12-18T03:39:52.8142555Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-12-18T03:39:52.8143537Z # be empty. The default action would be to continue rebuild the image 2024-12-18T03:39:52.8144907Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-12-18T03:39:52.8145740Z  # if we're on the base branch then use the parent commit 2024-12-18T03:39:52.8146444Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-12-18T03:39:52.8147011Z else 2024-12-18T03:39:52.8147593Z  # otherwise we're on a PR, so use the most recent base commit 2024-12-18T03:39:52.8148404Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-12-18T03:39:52.8149041Z fi 2024-12-18T03:39:52.8149422Z  2024-12-18T03:39:52.8149838Z if [[ -z "${MERGE_BASE}" ]]; then 2024-12-18T03:39:52.8150462Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.8151428Z  2024-12-18T03:39:52.8152555Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-12-18T03:39:52.8153526Z  exit 0 2024-12-18T03:39:52.8154004Z fi 2024-12-18T03:39:52.8154459Z  2024-12-18T03:39:52.8155139Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-12-18T03:39:52.8156477Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-12-18T03:39:52.8157509Z  exit 1 2024-12-18T03:39:52.8157931Z fi 2024-12-18T03:39:52.8158322Z  2024-12-18T03:39:52.8158976Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-12-18T03:39:52.8160142Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-12-18T03:39:52.8161375Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-12-18T03:39:52.8162981Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-12-18T03:39:52.8164609Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-12-18T03:39:52.8165737Z fi 2024-12-18T03:39:52.8166187Z  2024-12-18T03:39:52.8166668Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-12-18T03:39:52.8238213Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:52.8238905Z env: 2024-12-18T03:39:52.8239305Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:52.8239890Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:52.8240935Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:52.8242076Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:52.8242688Z AWS_REGION: us-east-1 2024-12-18T03:39:52.8243466Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:52.8244392Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:52.8255559Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:52.8256063Z DOCKER_BUILD_DIR: .ci/docker 2024-12-18T03:39:52.8256680Z BASE_REVISION: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:39:52.8258044Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.8259356Z DOCKER_TAG: 45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:52.8260132Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.8260822Z DOCKER_PUSH: 2024-12-18T03:39:52.8261239Z ##[endgroup] 2024-12-18T03:39:52.8370129Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.8370925Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:52.8379220Z + aws ecr get-login-password --region us-east-1 2024-12-18T03:39:52.8383203Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:53.9122820Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2024-12-18T03:39:53.9124025Z Configure a credential helper to remove this warning. See 2024-12-18T03:39:53.9125754Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2024-12-18T03:39:53.9126496Z 2024-12-18T03:39:53.9127427Z Login Succeeded 2024-12-18T03:39:53.9190897Z ++ date +%s 2024-12-18T03:39:53.9207884Z + START_TIME=1734493193 2024-12-18T03:39:53.9216338Z ++ date +%s 2024-12-18T03:39:53.9247107Z + [[ 1734487793 -lt 1734493193 ]] 2024-12-18T03:39:53.9248600Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:54.5972938Z { 2024-12-18T03:39:54.5973483Z "schemaVersion": 2, 2024-12-18T03:39:54.5974317Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-12-18T03:39:54.5975941Z "config": { 2024-12-18T03:39:54.5976669Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-12-18T03:39:54.5977511Z "size": 24144, 2024-12-18T03:39:54.5978280Z "digest": "sha256:07b91116eb1063321da7a63f31f1ef61d34f7dc8b7d56ec47c24f12af9f38edb" 2024-12-18T03:39:54.5979130Z }, 2024-12-18T03:39:54.5979491Z "layers": [ 2024-12-18T03:39:54.5979912Z { 2024-12-18T03:39:54.5980517Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5981301Z "size": 28583948, 2024-12-18T03:39:54.5982055Z "digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1" 2024-12-18T03:39:54.5983011Z }, 2024-12-18T03:39:54.5983431Z { 2024-12-18T03:39:54.5984120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5984995Z "size": 1825, 2024-12-18T03:39:54.5985890Z "digest": "sha256:7e5cdcc8d39cbfc3cbf5fe048da71a36c2c80d4a1e184cccc8c9f02e7e076f64" 2024-12-18T03:39:54.5986919Z }, 2024-12-18T03:39:54.5987329Z { 2024-12-18T03:39:54.5987907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5988638Z "size": 312642362, 2024-12-18T03:39:54.5989390Z "digest": "sha256:231ab5ed5e979cc01fda602a87a43fd305466b7cc01b98a76690ffa0b716f312" 2024-12-18T03:39:54.5990204Z }, 2024-12-18T03:39:54.5990546Z { 2024-12-18T03:39:54.5991120Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5991838Z "size": 863, 2024-12-18T03:39:54.5992830Z "digest": "sha256:fcbefe9ad79ece7f701532eae83e9de139fe1c0949a7813e55dd3a53003af9cd" 2024-12-18T03:39:54.5993858Z }, 2024-12-18T03:39:54.5994267Z { 2024-12-18T03:39:54.5994966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5995839Z "size": 106, 2024-12-18T03:39:54.5996596Z "digest": "sha256:af67ad6d287ebd8cbde55a749a5bbc06170aea40e4902c93cd34922e10c8c408" 2024-12-18T03:39:54.5997448Z }, 2024-12-18T03:39:54.5997811Z { 2024-12-18T03:39:54.5998388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.5999241Z "size": 704, 2024-12-18T03:39:54.5999981Z "digest": "sha256:a87dd7b9e020ba8d0d077d74c105bbb7fe8cdd8f1e6ae0c98b84c3e2c06ace60" 2024-12-18T03:39:54.6000815Z }, 2024-12-18T03:39:54.6001162Z { 2024-12-18T03:39:54.6001729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6002450Z "size": 1257, 2024-12-18T03:39:54.6003321Z "digest": "sha256:aa67f5368a1ddc10ff5b57e69f4ebcf2f83e0f32dfe4648e2a21aee041f107f8" 2024-12-18T03:39:54.6004304Z }, 2024-12-18T03:39:54.6004708Z { 2024-12-18T03:39:54.6005370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6006223Z "size": 3708, 2024-12-18T03:39:54.6007074Z "digest": "sha256:e71aede6d46007bf2a44f7cd14800cf5486c895c0d90a0b18ea9c30e6fff4571" 2024-12-18T03:39:54.6008049Z }, 2024-12-18T03:39:54.6008458Z { 2024-12-18T03:39:54.6009132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6009875Z "size": 1860, 2024-12-18T03:39:54.6010586Z "digest": "sha256:ee65ff0efe5e4f68873b0389e60a87f13f1a1bb0df1178fd70e5e83e59753fab" 2024-12-18T03:39:54.6011403Z }, 2024-12-18T03:39:54.6011749Z { 2024-12-18T03:39:54.6012706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6013439Z "size": 701, 2024-12-18T03:39:54.6014140Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:39:54.6014988Z }, 2024-12-18T03:39:54.6015395Z { 2024-12-18T03:39:54.6016070Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6016921Z "size": 2717169179, 2024-12-18T03:39:54.6017792Z "digest": "sha256:8362d69f07765b32e976c14dd6625372f9d81f263db1dc4ef06f46d99571f992" 2024-12-18T03:39:54.6018668Z }, 2024-12-18T03:39:54.6019012Z { 2024-12-18T03:39:54.6019574Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6020744Z "size": 380, 2024-12-18T03:39:54.6021440Z "digest": "sha256:0d29ca60b09574f6c0756a7025b88000a59b1009992fa1478637f5bdc5784ad2" 2024-12-18T03:39:54.6022241Z }, 2024-12-18T03:39:54.6022587Z { 2024-12-18T03:39:54.6023194Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6023933Z "size": 12135, 2024-12-18T03:39:54.6024635Z "digest": "sha256:6697412e0463180656f0051e70d45098fb97bbdb91515623a3ca0d0785ed26c3" 2024-12-18T03:39:54.6025428Z }, 2024-12-18T03:39:54.6025783Z { 2024-12-18T03:39:54.6026348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6027069Z "size": 504, 2024-12-18T03:39:54.6027780Z "digest": "sha256:3fe85e92871bbecf1a51782c8349d745adf952dabe41e4023f16496e13c89d6e" 2024-12-18T03:39:54.6028597Z }, 2024-12-18T03:39:54.6028949Z { 2024-12-18T03:39:54.6029509Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6030244Z "size": 121477504, 2024-12-18T03:39:54.6030968Z "digest": "sha256:db8079b9a247c260d493898268f96390bcef45198f5799ea28a82894417d3409" 2024-12-18T03:39:54.6031768Z }, 2024-12-18T03:39:54.6032115Z { 2024-12-18T03:39:54.6032916Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6033774Z "size": 109, 2024-12-18T03:39:54.6034573Z "digest": "sha256:1e1cc236bfa694641ad0d2d7fb1a76caa46f0b5364084a30d576a593cdf09838" 2024-12-18T03:39:54.6035389Z }, 2024-12-18T03:39:54.6035735Z { 2024-12-18T03:39:54.6036298Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6037011Z "size": 489, 2024-12-18T03:39:54.6037720Z "digest": "sha256:5594c816d49f271fafd93fedc956a82586a1ddb33f6a81c6c47324b2bc675e6f" 2024-12-18T03:39:54.6038538Z }, 2024-12-18T03:39:54.6038885Z { 2024-12-18T03:39:54.6039445Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6040167Z "size": 296, 2024-12-18T03:39:54.6040868Z "digest": "sha256:d0419af25312d587b2c3532b144859e2155c14d2ca73864f20041dd565b0c9c7" 2024-12-18T03:39:54.6041661Z }, 2024-12-18T03:39:54.6042008Z { 2024-12-18T03:39:54.6042621Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6043472Z "size": 103, 2024-12-18T03:39:54.6044313Z "digest": "sha256:4a7e3c6b1985b14cd275d8480a86dabccb21244b38cca3771881b2f87a880a7c" 2024-12-18T03:39:54.6045277Z }, 2024-12-18T03:39:54.6045693Z { 2024-12-18T03:39:54.6046376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6047223Z "size": 1474, 2024-12-18T03:39:54.6047992Z "digest": "sha256:e7e49899a1507f1355d8dd6a16d5beb03f1ced22428dc3b796a8c453fba8b41d" 2024-12-18T03:39:54.6048801Z }, 2024-12-18T03:39:54.6049148Z { 2024-12-18T03:39:54.6049710Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6050429Z "size": 430331313, 2024-12-18T03:39:54.6051167Z "digest": "sha256:907b5d385c1450d33d646dcce152dec74855431d5aa6d427f39d74f87ab7a183" 2024-12-18T03:39:54.6051995Z }, 2024-12-18T03:39:54.6052347Z { 2024-12-18T03:39:54.6052913Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6053769Z "size": 163, 2024-12-18T03:39:54.6055001Z "digest": "sha256:e09946f27e30eab5749280587f84a64accfe85ccfaabf422c1fae010dd492cbd" 2024-12-18T03:39:54.6055998Z }, 2024-12-18T03:39:54.6056368Z { 2024-12-18T03:39:54.6056931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6057644Z "size": 1640, 2024-12-18T03:39:54.6058579Z "digest": "sha256:7294a9f24454fce9dc046e77ab28a73149429dde57793009bb3223eb424713de" 2024-12-18T03:39:54.6059487Z }, 2024-12-18T03:39:54.6059830Z { 2024-12-18T03:39:54.6060399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6061129Z "size": 7880578701, 2024-12-18T03:39:54.6061890Z "digest": "sha256:5cbdd1d7b36d7b6e2d6428e2857ec4f8614e3becbdb0ec6e3b64f3cf63584e59" 2024-12-18T03:39:54.6063067Z }, 2024-12-18T03:39:54.6063413Z { 2024-12-18T03:39:54.6063975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6064691Z "size": 105, 2024-12-18T03:39:54.6065387Z "digest": "sha256:f528471f2fcaa24ae54a666344201877b06d22561708fd21987ac10c0873d6a0" 2024-12-18T03:39:54.6066182Z }, 2024-12-18T03:39:54.6066545Z { 2024-12-18T03:39:54.6067112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6067826Z "size": 1116, 2024-12-18T03:39:54.6068520Z "digest": "sha256:e1276511038b9b5b700d3a69400a0692e260be884871347905717fbec3009ced" 2024-12-18T03:39:54.6069313Z }, 2024-12-18T03:39:54.6069665Z { 2024-12-18T03:39:54.6070228Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6070947Z "size": 318568913, 2024-12-18T03:39:54.6071683Z "digest": "sha256:f3d817e0be62e96a38e3a6596705a35dfa864a2658a4d4e92890f192eaa1103e" 2024-12-18T03:39:54.6072838Z }, 2024-12-18T03:39:54.6073186Z { 2024-12-18T03:39:54.6073757Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6074582Z "size": 111, 2024-12-18T03:39:54.6075440Z "digest": "sha256:babfeddb429861eef35685a83ff84f1b120f8eda1b6a1c552453065dc0bd6cb3" 2024-12-18T03:39:54.6076383Z }, 2024-12-18T03:39:54.6076733Z { 2024-12-18T03:39:54.6077304Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6078027Z "size": 1555, 2024-12-18T03:39:54.6078746Z "digest": "sha256:e00cbec1ecdd51cc9b1b2471a6bf61f3e93cc03540114c13375e35cb48169f90" 2024-12-18T03:39:54.6079575Z }, 2024-12-18T03:39:54.6079925Z { 2024-12-18T03:39:54.6080491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6081210Z "size": 107, 2024-12-18T03:39:54.6081918Z "digest": "sha256:60175e320c6277d6be778429e32b3dd271c508af81bb52bdfec8c5c6edb55a66" 2024-12-18T03:39:54.6082732Z }, 2024-12-18T03:39:54.6083098Z { 2024-12-18T03:39:54.6083663Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6084468Z "size": 166, 2024-12-18T03:39:54.6085314Z "digest": "sha256:e4fd8cd3183ce9d5e635650cd867e61f68b6edc31b37e605f9c038a2064e42c0" 2024-12-18T03:39:54.6086283Z }, 2024-12-18T03:39:54.6086686Z { 2024-12-18T03:39:54.6087358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6088209Z "size": 2842293, 2024-12-18T03:39:54.6089067Z "digest": "sha256:f79cba221c4638d554fcf21bde3a813a5ddd7d4900f5928e43c92c9307e9d58d" 2024-12-18T03:39:54.6089905Z }, 2024-12-18T03:39:54.6090256Z { 2024-12-18T03:39:54.6090816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6091537Z "size": 107, 2024-12-18T03:39:54.6092243Z "digest": "sha256:e425c457ab3e0a6637feed1c3ef95776f4c43fc59336a9da4e0a7f19858529c2" 2024-12-18T03:39:54.6093065Z }, 2024-12-18T03:39:54.6093417Z { 2024-12-18T03:39:54.6094001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6094746Z "size": 566, 2024-12-18T03:39:54.6095594Z "digest": "sha256:6d2cbef9a0f53dff8893f40509e211ffe284a5d8fb4e03923b9c9646b06ad2e7" 2024-12-18T03:39:54.6096579Z }, 2024-12-18T03:39:54.6096986Z { 2024-12-18T03:39:54.6098018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6098769Z "size": 43195564, 2024-12-18T03:39:54.6099511Z "digest": "sha256:2c4472ad27f9258eee43c37d8fdd0ec0db693fc1d33d03902c90d85850a2fa74" 2024-12-18T03:39:54.6100331Z }, 2024-12-18T03:39:54.6100680Z { 2024-12-18T03:39:54.6101242Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6101961Z "size": 106, 2024-12-18T03:39:54.6102669Z "digest": "sha256:48ffbd6d25729ef5a8cbfc2950686efcd5cc5854803572b079a40e7d59b81481" 2024-12-18T03:39:54.6103495Z }, 2024-12-18T03:39:54.6103843Z { 2024-12-18T03:39:54.6104409Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6105459Z "size": 295, 2024-12-18T03:39:54.6106171Z "digest": "sha256:cf56c8fabbc0842e7f589f295894c2b6eb980e7f5ec8f2232b8c105f503a46e5" 2024-12-18T03:39:54.6106993Z }, 2024-12-18T03:39:54.6107341Z { 2024-12-18T03:39:54.6107908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6108646Z "size": 88293, 2024-12-18T03:39:54.6109372Z "digest": "sha256:b1c0b18891861b71d1c07dfb5e1063225b3ee635b5cf8d088bc62755ae2a3140" 2024-12-18T03:39:54.6110185Z }, 2024-12-18T03:39:54.6110535Z { 2024-12-18T03:39:54.6111107Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6111826Z "size": 106, 2024-12-18T03:39:54.6112657Z "digest": "sha256:3021fd4a208aeeab27bda896abedaba113f2a3845b3b037c57f855a429bcd61a" 2024-12-18T03:39:54.6113484Z }, 2024-12-18T03:39:54.6113831Z { 2024-12-18T03:39:54.6114390Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6115118Z "size": 1402, 2024-12-18T03:39:54.6115895Z "digest": "sha256:a6f5eec4a2df539ca39d87dfd6ba37ef1c60304eda99ed48b657823b8b5a83aa" 2024-12-18T03:39:54.6116872Z }, 2024-12-18T03:39:54.6117276Z { 2024-12-18T03:39:54.6117911Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6118626Z "size": 701, 2024-12-18T03:39:54.6119340Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:39:54.6120145Z }, 2024-12-18T03:39:54.6120489Z { 2024-12-18T03:39:54.6121053Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6121769Z "size": 137, 2024-12-18T03:39:54.6122483Z "digest": "sha256:89e8bdba21ffada6ab13d8b625b5349b0cc0398741ffd1c5155fdea44633255a" 2024-12-18T03:39:54.6123398Z }, 2024-12-18T03:39:54.6123806Z { 2024-12-18T03:39:54.6124473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6125318Z "size": 120, 2024-12-18T03:39:54.6126169Z "digest": "sha256:7239a56eb3344ab4d10ba3a41fc1e7af5559c747ee876bb8660f0237ed85a2f0" 2024-12-18T03:39:54.6127150Z }, 2024-12-18T03:39:54.6127557Z { 2024-12-18T03:39:54.6128207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6128932Z "size": 5014581981, 2024-12-18T03:39:54.6129673Z "digest": "sha256:22093b6a474afa19ed0f46a0491aa69688d1f0b3954fb7614a8f8e2538f90a43" 2024-12-18T03:39:54.6130482Z }, 2024-12-18T03:39:54.6130830Z { 2024-12-18T03:39:54.6131401Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6132123Z "size": 174, 2024-12-18T03:39:54.6132832Z "digest": "sha256:48aa755f3a17db6703cc7bff22b93fe216ee7f1112fb2e8330c2e1f7737822c2" 2024-12-18T03:39:54.6133689Z }, 2024-12-18T03:39:54.6134104Z { 2024-12-18T03:39:54.6134775Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6135624Z "size": 209, 2024-12-18T03:39:54.6136439Z "digest": "sha256:789ba2fe5a5187199259980262efd9e420b90ff489372e2cd60c98470a3db40d" 2024-12-18T03:39:54.6137322Z }, 2024-12-18T03:39:54.6137670Z { 2024-12-18T03:39:54.6138240Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6138963Z "size": 701, 2024-12-18T03:39:54.6139667Z "digest": "sha256:82c83b062d947fa6269438bde3cb9af1fdf56b6945be283407336d4cc3f16676" 2024-12-18T03:39:54.6140817Z }, 2024-12-18T03:39:54.6141199Z { 2024-12-18T03:39:54.6141782Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6142511Z "size": 633, 2024-12-18T03:39:54.6143214Z "digest": "sha256:e2e30e00626e69012828cfe72a0d31832f6ff31805e36d44356537d5f7fcb9ba" 2024-12-18T03:39:54.6144029Z }, 2024-12-18T03:39:54.6144377Z { 2024-12-18T03:39:54.6144942Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6145664Z "size": 296742339, 2024-12-18T03:39:54.6146385Z "digest": "sha256:c631a2525143e662e72b9b97c2e87b09a43a1038995e76b403dad6ba88966e41" 2024-12-18T03:39:54.6147710Z }, 2024-12-18T03:39:54.6148056Z { 2024-12-18T03:39:54.6148616Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6149339Z "size": 1932, 2024-12-18T03:39:54.6150061Z "digest": "sha256:08d3deeeb0fbaac4992048b95790661de095ddd8925926cfa2f2ec217bf4e0f7" 2024-12-18T03:39:54.6150889Z }, 2024-12-18T03:39:54.6151247Z { 2024-12-18T03:39:54.6151816Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6152628Z "size": 15983403, 2024-12-18T03:39:54.6153374Z "digest": "sha256:f098b0de0dbe9076c39d2aa97b15a08de76d7c33be71cc1069255184e8716618" 2024-12-18T03:39:54.6154199Z }, 2024-12-18T03:39:54.6154543Z { 2024-12-18T03:39:54.6155118Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-12-18T03:39:54.6155839Z "size": 54145664, 2024-12-18T03:39:54.6156675Z "digest": "sha256:4e73d9ed9f9510302a97c5ce44701037bd8453e91d3dbe54738434120cf00300" 2024-12-18T03:39:54.6157656Z } 2024-12-18T03:39:54.6158067Z ] 2024-12-18T03:39:54.6158435Z } 2024-12-18T03:39:54.6158806Z + exit 0 2024-12-18T03:39:54.6211076Z ##[group]Run set -eux 2024-12-18T03:39:54.6211540Z set -eux 2024-12-18T03:39:54.6212904Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2024-12-18T03:39:54.6277239Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:54.6277880Z env: 2024-12-18T03:39:54.6278248Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:54.6278771Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:54.6279691Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:54.6280536Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:54.6281016Z AWS_REGION: us-east-1 2024-12-18T03:39:54.6281618Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:54.6282255Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:54.6291392Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:54.6291823Z ##[endgroup] 2024-12-18T03:39:54.6398150Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2024-12-18T03:39:54.6400344Z + jq --raw-output .SecretString 2024-12-18T03:39:54.6405363Z + jq -r .docker_hub_readonly_token 2024-12-18T03:39:54.6408940Z + docker login --username pytorchbot --password-stdin 2024-12-18T03:39:55.3693045Z 2024-12-18T03:39:55.3696095Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action 2024-12-18T03:39:55.4850631Z Error: Cannot perform an interactive login from a non TTY device 2024-12-18T03:39:55.4937810Z ##[error]Process completed with exit code 1. 2024-12-18T03:39:55.5094768Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@release/2.6 2024-12-18T03:39:55.5095667Z with: 2024-12-18T03:39:55.5096798Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:55.5098242Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:55.5098940Z env: 2024-12-18T03:39:55.5099344Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:55.5099939Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:55.5100968Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:55.5101916Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:55.5102454Z AWS_REGION: us-east-1 2024-12-18T03:39:55.5103068Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:55.5103764Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:55.5114476Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:55.5115530Z ##[endgroup] 2024-12-18T03:39:55.5143154Z ##[group]Run set -x 2024-12-18T03:39:55.5143613Z set -x 2024-12-18T03:39:55.5143994Z set +e 2024-12-18T03:39:55.5144355Z  2024-12-18T03:39:55.5144701Z login() { 2024-12-18T03:39:55.5145479Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-12-18T03:39:55.5146316Z } 2024-12-18T03:39:55.5146682Z  2024-12-18T03:39:55.5147038Z retry () { 2024-12-18T03:39:55.5147491Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-12-18T03:39:55.5148005Z } 2024-12-18T03:39:55.5148353Z  2024-12-18T03:39:55.5148741Z retry login "${DOCKER_REGISTRY}" 2024-12-18T03:39:55.5149256Z  2024-12-18T03:39:55.5149604Z set -e 2024-12-18T03:39:55.5150158Z # ignore output since only exit code is used for conditional 2024-12-18T03:39:55.5150946Z # only pull docker image if it's not available locally 2024-12-18T03:39:55.5151805Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-12-18T03:39:55.5152761Z  retry docker pull "${DOCKER_IMAGE}" 2024-12-18T03:39:55.5153280Z fi 2024-12-18T03:39:55.5209028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:39:55.5209663Z env: 2024-12-18T03:39:55.5210022Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:55.5210554Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:55.5211474Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:55.5212307Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:55.5212773Z AWS_REGION: us-east-1 2024-12-18T03:39:55.5213298Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:55.5213922Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:55.5223017Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:55.5224091Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:55.5225371Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:55.5226007Z ##[endgroup] 2024-12-18T03:39:55.5319260Z + set +e 2024-12-18T03:39:55.5319997Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:55.5320816Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:55.5326893Z + aws ecr get-login-password --region us-east-1 2024-12-18T03:39:55.5327920Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T03:39:56.5984285Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2024-12-18T03:39:56.5985453Z Configure a credential helper to remove this warning. See 2024-12-18T03:39:56.5986504Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2024-12-18T03:39:56.5987201Z 2024-12-18T03:39:56.5987824Z Login Succeeded 2024-12-18T03:39:56.6042343Z + set -e 2024-12-18T03:39:56.6043741Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:39:56.6555411Z Prepare all required actions 2024-12-18T03:39:56.6556321Z Getting action download info 2024-12-18T03:39:56.8257068Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-12-18T03:39:57.5329833Z Download action repository 'actions/download-artifact@v4' (SHA:fa0a91b85d4f404e444e00e005971372dc801d16) 2024-12-18T03:39:58.1586366Z ##[group]Run ./.github/actions/download-build-artifacts 2024-12-18T03:39:58.1587114Z with: 2024-12-18T03:39:58.1587609Z name: linux-focal-rocm6.2-py3.10 2024-12-18T03:39:58.1588233Z s3-bucket: gha-artifacts 2024-12-18T03:39:58.1588747Z env: 2024-12-18T03:39:58.1589173Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:58.1589799Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:58.1591415Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:58.1592790Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:58.1593465Z AWS_REGION: us-east-1 2024-12-18T03:39:58.1594403Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:58.1595290Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:58.1606377Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:58.1606890Z ##[endgroup] 2024-12-18T03:39:58.1667644Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T03:39:58.1668316Z with: 2024-12-18T03:39:58.1668785Z name: linux-focal-rocm6.2-py3.10 2024-12-18T03:39:58.1669399Z s3-bucket: gha-artifacts 2024-12-18T03:39:58.1669933Z region: us-east-1 2024-12-18T03:39:58.1670386Z env: 2024-12-18T03:39:58.1670808Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:39:58.1671422Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:39:58.1672937Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:39:58.1673976Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:39:58.1674527Z AWS_REGION: us-east-1 2024-12-18T03:39:58.1675147Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:39:58.1675887Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:39:58.1687431Z AWS_SESSION_TOKEN: *** 2024-12-18T03:39:58.1687866Z ##[endgroup] 2024-12-18T03:39:58.6481108Z (node:652138) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T03:39:58.6482030Z 2024-12-18T03:39:58.6482389Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T03:39:58.6483350Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T03:39:58.6484362Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T03:39:58.8078390Z Found 1 objects with prefix pytorch/pytorch/12383255654/linux-focal-rocm6.2-py3.10/ 2024-12-18T03:39:58.8079789Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T03:40:08.8838805Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-12-18T03:40:08.8850718Z Artifact download has finished successfully 2024-12-18T03:40:08.9485311Z ##[group]Run unzip -o artifacts.zip 2024-12-18T03:40:08.9486081Z unzip -o artifacts.zip 2024-12-18T03:40:08.9558231Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:08.9558963Z env: 2024-12-18T03:40:08.9559376Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:08.9559965Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:08.9560995Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:08.9561932Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:08.9562456Z AWS_REGION: us-east-1 2024-12-18T03:40:08.9563064Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:08.9563751Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:08.9575191Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:08.9575726Z ##[endgroup] 2024-12-18T03:40:08.9719542Z Archive: artifacts.zip 2024-12-18T03:40:08.9721977Z creating: dist/ 2024-12-18T03:40:11.6794404Z inflating: dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T03:40:11.6919573Z inflating: dist/.ninja_log 2024-12-18T03:40:11.6920277Z creating: build/custom_test_artifacts/ 2024-12-18T03:40:11.6921046Z creating: build/custom_test_artifacts/custom-op-build/ 2024-12-18T03:40:11.6921969Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-12-18T03:40:11.6923029Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:40:11.6924341Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:40:11.6925518Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-12-18T03:40:11.6926683Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:40:11.6928831Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:40:11.6930060Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:40:11.6931438Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:40:11.6932826Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:40:11.6934057Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:40:11.6935284Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:40:11.6936693Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:40:11.6938152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:40:11.6939608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:40:11.6941066Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:40:11.6942501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:40:11.6943950Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:40:11.6945179Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:40:11.6946249Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:40:11.6947374Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-12-18T03:40:11.6948543Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-12-18T03:40:11.6949818Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-12-18T03:40:11.6951289Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-12-18T03:40:11.6952875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-12-18T03:40:11.6954171Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-12-18T03:40:11.6955497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-12-18T03:40:11.6956845Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-12-18T03:40:11.6958192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-12-18T03:40:11.6959534Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-12-18T03:40:11.6960881Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-12-18T03:40:11.6965482Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-12-18T03:40:11.7100749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-12-18T03:40:11.7102191Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-12-18T03:40:11.7103589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-12-18T03:40:11.7105155Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-12-18T03:40:11.7106613Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-12-18T03:40:11.7107963Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-12-18T03:40:11.7109359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-12-18T03:40:11.7111134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-12-18T03:40:11.7112699Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-12-18T03:40:11.7114081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-12-18T03:40:11.7115461Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-12-18T03:40:11.7125211Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-12-18T03:40:11.7202971Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-12-18T03:40:11.7204643Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:40:11.7206134Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:40:11.7207422Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-12-18T03:40:11.7208605Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-12-18T03:40:11.7209737Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-12-18T03:40:11.7210828Z inflating: build/custom_test_artifacts/custom-op-build/hip_new_types.cc 2024-12-18T03:40:11.7211868Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-12-18T03:40:11.7212857Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-12-18T03:40:11.7213881Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-12-18T03:40:11.7320003Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-12-18T03:40:11.7377277Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-12-18T03:40:11.7378336Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-12-18T03:40:11.7379260Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-12-18T03:40:11.7380340Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:40:11.7381574Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:40:11.7382750Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-12-18T03:40:11.7383903Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:40:11.7385128Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:40:11.7386321Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:40:11.7387692Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:40:11.7389061Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:40:11.7390866Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:40:11.7392095Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:40:11.7393697Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:40:11.7395149Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:40:11.7396547Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:40:11.7397960Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:40:11.7399415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:40:11.7401192Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:40:11.7402412Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:40:11.7403461Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:40:11.7404545Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-12-18T03:40:11.7405704Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-12-18T03:40:11.7407009Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-12-18T03:40:11.7408476Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-12-18T03:40:11.7409899Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-12-18T03:40:11.7411242Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-12-18T03:40:11.7412600Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-12-18T03:40:11.7413971Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-12-18T03:40:11.7415346Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-12-18T03:40:11.7416694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-12-18T03:40:11.7418059Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-12-18T03:40:11.7420870Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-12-18T03:40:11.7480452Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-12-18T03:40:11.7481980Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:40:11.7483367Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:40:11.7484588Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-12-18T03:40:11.7485723Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-12-18T03:40:11.7486814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-12-18T03:40:11.7487884Z inflating: build/custom_test_artifacts/jit-hook-build/hip_new_types.cc 2024-12-18T03:40:11.7488887Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-12-18T03:40:11.7489847Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-12-18T03:40:11.7490818Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-12-18T03:40:11.7531839Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-12-18T03:40:11.7533228Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-12-18T03:40:11.7534201Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-12-18T03:40:11.7535308Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-12-18T03:40:11.7538151Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-12-18T03:40:11.7539580Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-12-18T03:40:11.7540866Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-12-18T03:40:11.7542239Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-12-18T03:40:11.7543551Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-12-18T03:40:11.7545806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-12-18T03:40:11.7547309Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-12-18T03:40:11.7548626Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-12-18T03:40:11.7549935Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-12-18T03:40:11.7551473Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-12-18T03:40:11.7553218Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-12-18T03:40:11.7554737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-12-18T03:40:11.7556269Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-12-18T03:40:11.7557845Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-12-18T03:40:11.7559386Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-12-18T03:40:11.7560698Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-12-18T03:40:11.7561841Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-12-18T03:40:11.7563020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-12-18T03:40:11.7564279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-12-18T03:40:11.7565702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-12-18T03:40:11.7567296Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-12-18T03:40:11.7568851Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-12-18T03:40:11.7570289Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-12-18T03:40:11.7571779Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-12-18T03:40:11.7573292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-12-18T03:40:11.7574785Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-12-18T03:40:11.7576275Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-12-18T03:40:11.7577763Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-12-18T03:40:11.7579392Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-12-18T03:40:11.7676330Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-12-18T03:40:11.7678009Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-12-18T03:40:11.7679580Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-12-18T03:40:11.7681267Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-12-18T03:40:11.7682860Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-12-18T03:40:11.7684371Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-12-18T03:40:11.7686348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-12-18T03:40:11.7687992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-12-18T03:40:11.7689581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-12-18T03:40:11.7691143Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-12-18T03:40:11.7692699Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-12-18T03:40:11.7700748Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-12-18T03:40:11.7751594Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-12-18T03:40:11.7753509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-12-18T03:40:11.7755023Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-12-18T03:40:11.7756424Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-12-18T03:40:11.7757650Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-12-18T03:40:11.7758852Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-12-18T03:40:11.7760001Z inflating: build/custom_test_artifacts/custom-backend-build/hip_new_types.cc 2024-12-18T03:40:11.7761076Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-12-18T03:40:11.7762117Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-12-18T03:40:11.7763180Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-12-18T03:40:11.7850056Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-12-18T03:40:11.7889139Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-12-18T03:40:11.7889997Z creating: build/lib/ 2024-12-18T03:40:11.7970637Z inflating: build/lib/libprotobuf-lite.a 2024-12-18T03:40:11.7979216Z inflating: build/lib/libpthreadpool.a 2024-12-18T03:40:11.7986591Z inflating: build/lib/libcpuinfo.a 2024-12-18T03:40:11.7993806Z inflating: build/lib/libcpuinfo_internals.a 2024-12-18T03:40:11.8411362Z inflating: build/lib/libprotobuf.a 2024-12-18T03:40:11.8871708Z inflating: build/lib/libprotoc.a 2024-12-18T03:40:11.8872562Z inflating: build/lib/libclog.a 2024-12-18T03:40:11.8890142Z inflating: build/lib/libpytorch_qnnpack.a 2024-12-18T03:40:11.8891547Z inflating: build/lib/libnnpack_reference_layers.a 2024-12-18T03:40:11.8908387Z inflating: build/lib/libnnpack.a 2024-12-18T03:40:11.9078846Z inflating: build/lib/libmicrokernels-prod.a 2024-12-18T03:40:11.9866699Z inflating: build/lib/libmicrokernels-all.a 2024-12-18T03:40:11.9929497Z inflating: build/lib/libgtest.a 2024-12-18T03:40:11.9945908Z inflating: build/lib/libgmock.a 2024-12-18T03:40:11.9946746Z inflating: build/lib/libgtest_main.a 2024-12-18T03:40:11.9947391Z inflating: build/lib/libgmock_main.a 2024-12-18T03:40:12.0014871Z inflating: build/lib/libbenchmark.a 2024-12-18T03:40:12.0015607Z inflating: build/lib/libbenchmark_main.a 2024-12-18T03:40:12.0096563Z inflating: build/lib/libXNNPACK.a 2024-12-18T03:40:12.0103590Z inflating: build/lib/libittnotify.a 2024-12-18T03:40:12.0162071Z inflating: build/lib/libasmjit.a 2024-12-18T03:40:12.1341001Z inflating: build/lib/libfbgemm.a 2024-12-18T03:40:12.1366073Z inflating: build/lib/libtensorpipe_uv.a 2024-12-18T03:40:12.1895690Z inflating: build/lib/libtensorpipe.a 2024-12-18T03:40:12.1934577Z inflating: build/lib/libonnx_proto.a 2024-12-18T03:40:12.2038403Z inflating: build/lib/libgloo.a 2024-12-18T03:40:12.2399892Z inflating: build/lib/libgloo_hip.a 2024-12-18T03:40:12.3073169Z inflating: build/lib/libonnx.a 2024-12-18T03:40:12.3091843Z inflating: build/lib/libfmt.a 2024-12-18T03:40:12.3388854Z inflating: build/lib/libkineto.a 2024-12-18T03:40:13.2931330Z inflating: build/lib/libdnnl.a 2024-12-18T03:40:13.3020631Z inflating: build/lib/libc10.so 2024-12-18T03:40:13.3021394Z inflating: build/lib/libtorch_global_deps.so 2024-12-18T03:40:13.3064796Z inflating: build/lib/libc10_hip.so 2024-12-18T03:40:13.3065520Z inflating: build/lib/libcaffe2_nvrtc.so 2024-12-18T03:40:15.5971063Z inflating: build/lib/libtorch_cpu.so 2024-12-18T03:40:15.5974893Z inflating: build/lib/libunbox_lib.a 2024-12-18T03:40:15.5978620Z inflating: build/lib/libshm.so 2024-12-18T03:40:16.8034410Z inflating: build/lib/libtorch_hip.so 2024-12-18T03:40:16.8035128Z inflating: build/lib/libtorch.so 2024-12-18T03:40:16.8053617Z inflating: build/lib/libjitbackend_test.so 2024-12-18T03:40:16.8117468Z inflating: build/lib/libtorchbind_test.so 2024-12-18T03:40:16.8139540Z inflating: build/lib/libbackend_with_compiler.so 2024-12-18T03:40:16.8162238Z inflating: build/lib/libaoti_custom_ops.so 2024-12-18T03:40:17.0038585Z inflating: build/lib/libtorch_python.so 2024-12-18T03:40:17.0069414Z inflating: build/lib/libnnapi_backend.so 2024-12-18T03:40:17.0070106Z creating: build/bin/ 2024-12-18T03:40:17.0070627Z creating: build/bin/CMakeFiles/ 2024-12-18T03:40:17.0071245Z inflating: build/bin/cmake_install.cmake 2024-12-18T03:40:17.0071910Z inflating: build/bin/CTestTestfile.cmake 2024-12-18T03:40:17.0473724Z inflating: build/bin/protoc-3.13.0.0 2024-12-18T03:40:17.0875129Z inflating: build/bin/protoc 2024-12-18T03:40:17.0923178Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-12-18T03:40:17.0972436Z inflating: build/bin/c10_DeviceGuard_test 2024-12-18T03:40:17.1021869Z inflating: build/bin/c10_Device_test 2024-12-18T03:40:17.1078305Z inflating: build/bin/c10_DispatchKeySet_test 2024-12-18T03:40:17.1129140Z inflating: build/bin/c10_Scalar_test 2024-12-18T03:40:17.1176021Z inflating: build/bin/c10_StreamGuard_test 2024-12-18T03:40:17.1224628Z inflating: build/bin/c10_SymInt_test 2024-12-18T03:40:17.1276414Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-12-18T03:40:17.1329356Z inflating: build/bin/c10_InlineStreamGuard_test 2024-12-18T03:40:17.1382997Z inflating: build/bin/c10_SizesAndStrides_test 2024-12-18T03:40:17.1449402Z inflating: build/bin/c10_cow_test 2024-12-18T03:40:17.1496370Z inflating: build/bin/c10_ConstexprCrc_test 2024-12-18T03:40:17.1546748Z inflating: build/bin/c10_Bitset_test 2024-12-18T03:40:17.1594504Z inflating: build/bin/c10_ArrayRef_test 2024-12-18T03:40:17.1647871Z inflating: build/bin/c10_LeftRight_test 2024-12-18T03:40:17.1696391Z inflating: build/bin/c10_DeadlockDetection_test 2024-12-18T03:40:17.1744577Z inflating: build/bin/c10_Half_test 2024-12-18T03:40:17.1797494Z inflating: build/bin/c10_Metaprogramming_test 2024-12-18T03:40:17.1848353Z inflating: build/bin/c10_NetworkFlow_test 2024-12-18T03:40:17.1895753Z inflating: build/bin/c10_Synchronized_test 2024-12-18T03:40:17.1944785Z inflating: build/bin/c10_TypeIndex_test 2024-12-18T03:40:17.1997973Z inflating: build/bin/c10_ThreadLocal_test 2024-12-18T03:40:17.2046837Z inflating: build/bin/c10_TypeList_test 2024-12-18T03:40:17.2093756Z inflating: build/bin/c10_TypeTraits_test 2024-12-18T03:40:17.2142973Z inflating: build/bin/c10_accumulate_test 2024-12-18T03:40:17.2196490Z inflating: build/bin/c10_bfloat16_test 2024-12-18T03:40:17.2244443Z inflating: build/bin/c10_bit_cast_test 2024-12-18T03:40:17.2297007Z inflating: build/bin/c10_complex_test 2024-12-18T03:40:17.2350943Z inflating: build/bin/c10_complex_math_test 2024-12-18T03:40:17.2398531Z inflating: build/bin/c10_error_test 2024-12-18T03:40:17.2448842Z inflating: build/bin/c10_exception_test 2024-12-18T03:40:17.2497124Z inflating: build/bin/c10_flags_test 2024-12-18T03:40:17.2545475Z inflating: build/bin/c10_generic_math_test 2024-12-18T03:40:17.2596339Z inflating: build/bin/c10_lazy_test 2024-12-18T03:40:17.2644904Z inflating: build/bin/c10_irange_test 2024-12-18T03:40:17.2801106Z inflating: build/bin/c10_intrusive_ptr_test 2024-12-18T03:40:17.2855279Z inflating: build/bin/c10_logging_test 2024-12-18T03:40:17.2927822Z inflating: build/bin/c10_optional_test 2024-12-18T03:40:17.2987208Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-12-18T03:40:17.3038316Z inflating: build/bin/c10_registry_test 2024-12-18T03:40:17.3183915Z inflating: build/bin/c10_small_vector_test 2024-12-18T03:40:17.3233264Z inflating: build/bin/c10_string_util_test 2024-12-18T03:40:17.3282895Z inflating: build/bin/c10_ssize_test 2024-12-18T03:40:17.3339617Z inflating: build/bin/c10_string_view_test 2024-12-18T03:40:17.3388013Z inflating: build/bin/c10_tempfile_test 2024-12-18T03:40:17.3431314Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-12-18T03:40:17.3484860Z inflating: build/bin/c10_typeid_test 2024-12-18T03:40:17.3531350Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2024-12-18T03:40:17.3578285Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2024-12-18T03:40:17.3624819Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2024-12-18T03:40:17.3671568Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2024-12-18T03:40:17.3718249Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2024-12-18T03:40:17.3765071Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2024-12-18T03:40:17.3811503Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2024-12-18T03:40:17.3858600Z inflating: build/bin/c10_hip_HIPTest 2024-12-18T03:40:17.4224445Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-12-18T03:40:17.4599324Z inflating: build/bin/vec_test_all_types_AVX512 2024-12-18T03:40:17.4987494Z inflating: build/bin/vec_test_all_types_AVX2 2024-12-18T03:40:17.5038577Z inflating: build/bin/test_edge_op_registration 2024-12-18T03:40:17.5089504Z inflating: build/bin/BackoffTest 2024-12-18T03:40:17.5140197Z inflating: build/bin/FileStoreTest 2024-12-18T03:40:17.5192973Z inflating: build/bin/TCPStoreTest 2024-12-18T03:40:17.5243790Z inflating: build/bin/HashStoreTest 2024-12-18T03:40:17.5306080Z inflating: build/bin/ProcessGroupGlooTest 2024-12-18T03:40:17.5307675Z inflating: build/bin/example_allreduce 2024-12-18T03:40:17.5362148Z inflating: build/bin/static_runtime_bench 2024-12-18T03:40:17.5596874Z inflating: build/bin/static_runtime_test 2024-12-18T03:40:17.5666939Z inflating: build/bin/Dict_test 2024-12-18T03:40:17.5713937Z inflating: build/bin/hip_half_test 2024-12-18T03:40:17.5763396Z inflating: build/bin/Dimname_test 2024-12-18T03:40:17.5825343Z inflating: build/bin/MaybeOwned_test 2024-12-18T03:40:17.5880728Z inflating: build/bin/NamedTensor_test 2024-12-18T03:40:17.5936512Z inflating: build/bin/apply_utils_test 2024-12-18T03:40:17.5992880Z inflating: build/bin/atest 2024-12-18T03:40:17.6052854Z inflating: build/bin/basic 2024-12-18T03:40:17.6104948Z inflating: build/bin/broadcast_test 2024-12-18T03:40:17.6154086Z inflating: build/bin/cpu_allocator_test 2024-12-18T03:40:17.6205090Z inflating: build/bin/cpu_profiling_allocator_test 2024-12-18T03:40:17.6261931Z inflating: build/bin/cpu_generator_test 2024-12-18T03:40:17.6348938Z inflating: build/bin/cpu_rng_test 2024-12-18T03:40:17.6396956Z inflating: build/bin/dispatch_key_set_test 2024-12-18T03:40:17.6445590Z inflating: build/bin/dlconvertor_test 2024-12-18T03:40:17.6500634Z inflating: build/bin/extension_backend_test 2024-12-18T03:40:17.6553630Z inflating: build/bin/half_test 2024-12-18T03:40:17.6643481Z inflating: build/bin/ivalue_test 2024-12-18T03:40:17.6690950Z inflating: build/bin/lazy_tensor_test 2024-12-18T03:40:17.6742713Z inflating: build/bin/math_kernel_test 2024-12-18T03:40:17.6791267Z inflating: build/bin/operator_name_test 2024-12-18T03:40:17.6842952Z inflating: build/bin/memory_format_test 2024-12-18T03:40:17.6893642Z inflating: build/bin/memory_overlapping_test 2024-12-18T03:40:17.6947362Z inflating: build/bin/native_test 2024-12-18T03:40:17.6997962Z inflating: build/bin/mobile_memory_cleanup 2024-12-18T03:40:17.7047303Z inflating: build/bin/operators_test 2024-12-18T03:40:17.7096853Z inflating: build/bin/packedtensoraccessor_test 2024-12-18T03:40:17.7160547Z inflating: build/bin/pow_test 2024-12-18T03:40:17.7208158Z inflating: build/bin/reduce_ops_test 2024-12-18T03:40:17.7263055Z inflating: build/bin/quantized_test 2024-12-18T03:40:17.7311715Z inflating: build/bin/reportMemoryUsage_test 2024-12-18T03:40:17.7366407Z inflating: build/bin/scalar_tensor_test 2024-12-18T03:40:17.7415635Z inflating: build/bin/StorageUtils_test 2024-12-18T03:40:17.7471731Z inflating: build/bin/scalar_test 2024-12-18T03:40:17.7522477Z inflating: build/bin/stride_properties_test 2024-12-18T03:40:17.7598021Z inflating: build/bin/tensor_iterator_test 2024-12-18T03:40:17.7649939Z inflating: build/bin/test_parallel 2024-12-18T03:40:17.7703023Z inflating: build/bin/type_ptr_test 2024-12-18T03:40:17.7704519Z inflating: build/bin/thread_init_test 2024-12-18T03:40:17.7705959Z inflating: build/bin/verify_api_visibility 2024-12-18T03:40:17.7757060Z inflating: build/bin/undefined_tensor_test 2024-12-18T03:40:17.7813865Z inflating: build/bin/type_test 2024-12-18T03:40:17.7880227Z inflating: build/bin/legacy_vmap_test 2024-12-18T03:40:17.7929399Z inflating: build/bin/weakref_test 2024-12-18T03:40:17.8030697Z inflating: build/bin/List_test 2024-12-18T03:40:17.8079867Z inflating: build/bin/wrapdim_test 2024-12-18T03:40:17.8136859Z inflating: build/bin/IListRef_test 2024-12-18T03:40:17.8186240Z inflating: build/bin/xla_tensor_test 2024-12-18T03:40:17.8299958Z inflating: build/bin/kernel_function_legacy_test 2024-12-18T03:40:17.8390632Z inflating: build/bin/kernel_function_test 2024-12-18T03:40:17.8510759Z inflating: build/bin/kernel_lambda_legacy_test 2024-12-18T03:40:17.8608148Z inflating: build/bin/kernel_lambda_test 2024-12-18T03:40:17.8671992Z inflating: build/bin/KernelFunction_test 2024-12-18T03:40:17.8730558Z inflating: build/bin/kernel_stackbased_test 2024-12-18T03:40:17.8779545Z inflating: build/bin/CppSignature_test 2024-12-18T03:40:17.8869966Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-12-18T03:40:17.8916715Z inflating: build/bin/op_allowlist_test 2024-12-18T03:40:17.8977431Z inflating: build/bin/inline_container_test 2024-12-18T03:40:17.9258822Z inflating: build/bin/op_registration_test 2024-12-18T03:40:17.9311828Z inflating: build/bin/backend_fallback_test 2024-12-18T03:40:17.9359422Z inflating: build/bin/hip_complex_math_test 2024-12-18T03:40:17.9409308Z inflating: build/bin/hip_apply_test 2024-12-18T03:40:17.9456298Z inflating: build/bin/hip_complex_test 2024-12-18T03:40:17.9502895Z inflating: build/bin/hip_distributions_test 2024-12-18T03:40:17.9549800Z inflating: build/bin/hip_generator_test 2024-12-18T03:40:17.9598564Z inflating: build/bin/hip_dlconvertor_test 2024-12-18T03:40:17.9645561Z inflating: build/bin/hip_integer_divider_test 2024-12-18T03:40:17.9692257Z inflating: build/bin/hip_optional_test 2024-12-18T03:40:17.9739649Z inflating: build/bin/hip_packedtensoraccessor_test 2024-12-18T03:40:17.9786238Z inflating: build/bin/hip_vectorized_test 2024-12-18T03:40:18.0315197Z inflating: build/bin/test_jit 2024-12-18T03:40:18.1043942Z inflating: build/bin/test_tensorexpr 2024-12-18T03:40:18.1056783Z inflating: build/bin/tutorial_tensorexpr 2024-12-18T03:40:18.1109266Z inflating: build/bin/test_dist_autograd 2024-12-18T03:40:18.1174540Z inflating: build/bin/test_cpp_rpc 2024-12-18T03:40:18.1176024Z inflating: build/bin/parallel_benchmark 2024-12-18T03:40:18.2281595Z inflating: build/bin/test_api 2024-12-18T03:40:18.2289534Z inflating: build/bin/aot_model_compiler_test 2024-12-18T03:40:18.2352963Z inflating: build/bin/test_mobile_nnc 2024-12-18T03:40:18.2676567Z inflating: build/bin/test_lazy 2024-12-18T03:40:18.2680337Z inflating: build/bin/torch_shm_manager 2024-12-18T03:40:18.2681026Z creating: .additional_ci_files/ 2024-12-18T03:40:18.2762922Z inflating: .additional_ci_files/test-times.json 2024-12-18T03:40:18.3085481Z inflating: .additional_ci_files/test-class-times.json 2024-12-18T03:40:18.3148447Z ##[group]Run rm artifacts.zip 2024-12-18T03:40:18.3149017Z rm artifacts.zip 2024-12-18T03:40:18.3219451Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:18.3220142Z env: 2024-12-18T03:40:18.3220543Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:18.3221152Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:18.3222194Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:18.3223139Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:18.3223653Z AWS_REGION: us-east-1 2024-12-18T03:40:18.3224229Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:18.3224920Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:18.3235421Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:18.3235903Z ##[endgroup] 2024-12-18T03:40:18.5867107Z ##[group]Run df -H 2024-12-18T03:40:18.5867586Z df -H 2024-12-18T03:40:18.5932713Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:18.5933467Z env: 2024-12-18T03:40:18.5933904Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:18.5934536Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:18.5935635Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:18.5936649Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:18.5937223Z AWS_REGION: us-east-1 2024-12-18T03:40:18.5937850Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:18.5938583Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:18.5949610Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:18.5950125Z ##[endgroup] 2024-12-18T03:40:18.6126054Z Filesystem Size Used Avail Use% Mounted on 2024-12-18T03:40:18.6126805Z tmpfs 109G 33M 109G 1% /run 2024-12-18T03:40:18.6127503Z /dev/nvme0n1p2 944G 63G 833G 7% / 2024-12-18T03:40:18.6128215Z tmpfs 542G 33k 542G 1% /dev/shm 2024-12-18T03:40:18.6128906Z tmpfs 5.3M 0 5.3M 0% /run/lock 2024-12-18T03:40:18.6129727Z /dev/nvme0n1p1 536M 6.4M 530M 2% /boot/efi 2024-12-18T03:40:18.6131027Z /dev/nvme1n1 3.8T 2.0T 1.7T 56% /media/4TB 2024-12-18T03:40:18.6131882Z tmpfs 109G 33k 109G 1% /run/user/1307800118 2024-12-18T03:40:18.6132606Z 172.18.148.8:/export/amd2 5.5T 260G 5.3T 5% /mnt 2024-12-18T03:40:18.6133511Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch 108G 468M 107G 1% /home/amd-pytorch 2024-12-18T03:40:18.6134617Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage 110T 72T 39T 65% /groups 2024-12-18T03:40:18.6135699Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch 5.5T 375G 5.2T 7% /scratch 2024-12-18T03:40:18.6136841Z pure1.jax.cs.cpe.ice.amd.com:/homes/mkunredd 108G 1.1M 108G 1% /home/mkunredd 2024-12-18T03:40:18.6137983Z pure1.jax.cs.cpe.ice.amd.com:/homes/okakarpa 108G 137M 108G 1% /home/okakarpa 2024-12-18T03:40:18.6192024Z Prepare all required actions 2024-12-18T03:40:18.6192828Z Getting action download info 2024-12-18T03:40:18.8373924Z ##[group]Run ./.github/actions/download-td-artifacts 2024-12-18T03:40:18.8374260Z with: 2024-12-18T03:40:18.8374466Z env: 2024-12-18T03:40:18.8374670Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:18.8404701Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:18.8405346Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:18.8405875Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:18.8406171Z AWS_REGION: us-east-1 2024-12-18T03:40:18.8406501Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:18.8406897Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:18.8412250Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:18.8412529Z ##[endgroup] 2024-12-18T03:40:18.8476945Z ##[group]Run seemethere/download-artifact-s3@v4 2024-12-18T03:40:18.8477558Z with: 2024-12-18T03:40:18.8477978Z name: td_results 2024-12-18T03:40:18.8478499Z s3-bucket: gha-artifacts 2024-12-18T03:40:18.8478988Z region: us-east-1 2024-12-18T03:40:18.8479406Z env: 2024-12-18T03:40:18.8479808Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:18.8480386Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:18.8481448Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:18.8482559Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:18.8483168Z AWS_REGION: us-east-1 2024-12-18T03:40:18.8483841Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:18.8484663Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:18.8491721Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:18.8491978Z ##[endgroup] 2024-12-18T03:40:19.3013383Z (node:652197) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-12-18T03:40:19.3014400Z 2024-12-18T03:40:19.3014789Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-12-18T03:40:19.3015908Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-12-18T03:40:19.3016997Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-12-18T03:40:19.4499102Z Found 1 objects with prefix pytorch/pytorch/12383255654/td_results/ 2024-12-18T03:40:19.4500451Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T03:40:19.6316743Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2024-12-18T03:40:19.6328441Z Artifact download has finished successfully 2024-12-18T03:40:19.6867558Z ##[group]Run mkdir -p .additional_ci_files 2024-12-18T03:40:19.6868261Z mkdir -p .additional_ci_files 2024-12-18T03:40:19.6869058Z mv td_results.json .additional_ci_files/td_results.json || true 2024-12-18T03:40:19.6932997Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:19.6933723Z env: 2024-12-18T03:40:19.6934182Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:19.6934804Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:19.6935850Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:19.6936795Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:19.6937320Z AWS_REGION: us-east-1 2024-12-18T03:40:19.6937924Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:19.6938608Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:19.6948971Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:19.6949454Z ##[endgroup] 2024-12-18T03:40:19.7182695Z ##[group]Run .github/scripts/parse_ref.py 2024-12-18T03:40:19.7183421Z .github/scripts/parse_ref.py 2024-12-18T03:40:19.7244180Z shell: /usr/bin/bash -e {0} 2024-12-18T03:40:19.7244816Z env: 2024-12-18T03:40:19.7245312Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:19.7245841Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:19.7246534Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:19.7247461Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:19.7247800Z AWS_REGION: us-east-1 2024-12-18T03:40:19.7248169Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:19.7248590Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:19.7257204Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:19.7257673Z ##[endgroup] 2024-12-18T03:40:19.7835314Z Prepare all required actions 2024-12-18T03:40:19.7895749Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-12-18T03:40:19.7896414Z with: 2024-12-18T03:40:19.7897189Z github-token: *** 2024-12-18T03:40:19.7897664Z env: 2024-12-18T03:40:19.7898093Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:19.7898710Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:19.7899790Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:19.7900786Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:19.7901338Z AWS_REGION: us-east-1 2024-12-18T03:40:19.7901937Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:19.7902675Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:19.7913841Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:19.7914344Z ##[endgroup] 2024-12-18T03:40:19.7945667Z ##[group]Run set -eux 2024-12-18T03:40:19.7946189Z set -eux 2024-12-18T03:40:19.7947082Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-12-18T03:40:19.8018635Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:19.8019258Z env: 2024-12-18T03:40:19.8019631Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:19.8020165Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:19.8021068Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:19.8021893Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:19.8022356Z AWS_REGION: us-east-1 2024-12-18T03:40:19.8022868Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:19.8023474Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:19.8032819Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:19.8033466Z GITHUB_TOKEN: *** 2024-12-18T03:40:19.8033861Z ##[endgroup] 2024-12-18T03:40:19.8128769Z + python3 .github/scripts/get_workflow_job_id.py 12383255654 gpu9489.jax.cs.cpe.ice.amd.com 2024-12-18T03:40:21.4360850Z setting job-id=34566686857 2024-12-18T03:40:21.4362128Z setting job-name=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:21.4695757Z Prepare all required actions 2024-12-18T03:40:21.4696525Z Getting action download info 2024-12-18T03:40:21.6510876Z ##[group]Run ./.github/actions/filter-test-configs 2024-12-18T03:40:21.6511529Z with: 2024-12-18T03:40:21.6512650Z github-token: *** 2024-12-18T03:40:21.6515138Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]} 2024-12-18T03:40:21.6518229Z job-name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:21.6519238Z env: 2024-12-18T03:40:21.6519648Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:21.6520326Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:21.6521353Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:21.6522310Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:21.6522836Z AWS_REGION: us-east-1 2024-12-18T03:40:21.6523377Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:21.6524047Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:21.6534359Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:21.6534846Z ##[endgroup] 2024-12-18T03:40:21.6601624Z ##[group]Run nick-fields/retry@v3.0.0 2024-12-18T03:40:21.6602622Z with: 2024-12-18T03:40:21.6603009Z shell: bash 2024-12-18T03:40:21.6603422Z timeout_minutes: 10 2024-12-18T03:40:21.6603869Z max_attempts: 5 2024-12-18T03:40:21.6604304Z retry_wait_seconds: 30 2024-12-18T03:40:21.6605678Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T03:40:21.6607116Z polling_interval_seconds: 1 2024-12-18T03:40:21.6607630Z warning_on_retry: true 2024-12-18T03:40:21.6608103Z continue_on_error: false 2024-12-18T03:40:21.6608574Z env: 2024-12-18T03:40:21.6608960Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:21.6609532Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:21.6610566Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:21.6611500Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:21.6612026Z AWS_REGION: us-east-1 2024-12-18T03:40:21.6612592Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:21.6613281Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:21.6623625Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:21.6624330Z GITHUB_TOKEN: *** 2024-12-18T03:40:21.6624783Z ##[endgroup] 2024-12-18T03:40:21.7900706Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-12-18T03:40:22.0673348Z Defaulting to user installation because normal site-packages is not writeable 2024-12-18T03:40:22.2327184Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2024-12-18T03:40:22.2330702Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1) 2024-12-18T03:40:22.2420849Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2024-12-18T03:40:22.2424380Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2024-12-18T03:40:22.2427860Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2024-12-18T03:40:22.2437663Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2024-12-18T03:40:22.7900961Z Command completed after 1 attempt(s). 2024-12-18T03:40:22.8036614Z ##[group]Run set -x 2024-12-18T03:40:22.8037633Z set -x 2024-12-18T03:40:22.8038092Z  2024-12-18T03:40:22.8038810Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T03:40:22.8039656Z # in runner workspace 2024-12-18T03:40:22.8040363Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-12-18T03:40:22.8109787Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:22.8110491Z env: 2024-12-18T03:40:22.8110899Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:22.8111532Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:22.8112916Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:22.8113914Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:22.8114447Z AWS_REGION: us-east-1 2024-12-18T03:40:22.8115046Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:22.8115736Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:22.8126141Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:22.8126632Z ##[endgroup] 2024-12-18T03:40:22.8231565Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-12-18T03:40:22.8683628Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T03:40:22.8684406Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-12-18T03:40:22.8685045Z echo "Job name: ${JOB_NAME}" 2024-12-18T03:40:22.8685591Z  2024-12-18T03:40:22.8686281Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-12-18T03:40:22.8687572Z # in runner workspace 2024-12-18T03:40:22.8688375Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-12-18T03:40:22.8689264Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-12-18T03:40:22.8689874Z  --job-name "${JOB_NAME}" \ 2024-12-18T03:40:22.8692507Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}" \ 2024-12-18T03:40:22.8695119Z  --selected-test-configs "" \ 2024-12-18T03:40:22.8695720Z  --pr-number "${PR_NUMBER}" \ 2024-12-18T03:40:22.8696280Z  --tag "${TAG}" \ 2024-12-18T03:40:22.8696830Z  --event-name "${EVENT_NAME}" \ 2024-12-18T03:40:22.8697406Z  --schedule "${SCHEDULE}" \ 2024-12-18T03:40:22.8697965Z  --branch "${HEAD_BRANCH}" 2024-12-18T03:40:22.8767287Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:22.8768013Z env: 2024-12-18T03:40:22.8768426Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:22.8769050Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:22.8770082Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:22.8771023Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:22.8771543Z AWS_REGION: us-east-1 2024-12-18T03:40:22.8772125Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:22.8772815Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:22.8783219Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:22.8783971Z GITHUB_TOKEN: *** 2024-12-18T03:40:22.8784928Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:22.8785991Z PR_NUMBER: 2024-12-18T03:40:22.8786407Z TAG: 2024-12-18T03:40:22.8786799Z EVENT_NAME: push 2024-12-18T03:40:22.8787237Z SCHEDULE: 2024-12-18T03:40:22.8787640Z HEAD_BRANCH: 2024-12-18T03:40:22.8788069Z ##[endgroup] 2024-12-18T03:40:22.8885730Z Workflow: periodic 2024-12-18T03:40:22.8887023Z Job name: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:23.3067416Z ##[group]Run echo "Filtered matrix:" 2024-12-18T03:40:23.3067994Z echo "Filtered matrix:" 2024-12-18T03:40:23.3070195Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu", "owners": ["module:rocm", "oncall:distributed"]}]}" 2024-12-18T03:40:23.3072640Z  2024-12-18T03:40:23.3072992Z echo 2024-12-18T03:40:23.3073451Z echo "Is the current job unstable? False" 2024-12-18T03:40:23.3073998Z  2024-12-18T03:40:23.3074337Z echo 2024-12-18T03:40:23.3074758Z echo "Is keep-going label set? False" 2024-12-18T03:40:23.3075302Z  2024-12-18T03:40:23.3075659Z echo 2024-12-18T03:40:23.3076042Z echo "Renabled issues? " 2024-12-18T03:40:23.3138250Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:23.3138887Z env: 2024-12-18T03:40:23.3139262Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:23.3139805Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:23.3140731Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:23.3141594Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:23.3142501Z AWS_REGION: us-east-1 2024-12-18T03:40:23.3143156Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:23.3143781Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:23.3153001Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:23.3153435Z ##[endgroup] 2024-12-18T03:40:23.3248413Z Filtered matrix: 2024-12-18T03:40:23.3250935Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu, owners: [module:rocm, oncall:distributed]}]} 2024-12-18T03:40:23.3253306Z 2024-12-18T03:40:23.3253596Z Is the current job unstable? False 2024-12-18T03:40:23.3253991Z 2024-12-18T03:40:23.3254223Z Is keep-going label set? False 2024-12-18T03:40:23.3254585Z 2024-12-18T03:40:23.3254781Z Renabled issues? 2024-12-18T03:40:23.3325812Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T03:40:23.3326822Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-12-18T03:40:23.3388716Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T03:40:23.3389465Z env: 2024-12-18T03:40:23.3389886Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:23.3390483Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:23.3391506Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:23.3392796Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:23.3393321Z AWS_REGION: us-east-1 2024-12-18T03:40:23.3393879Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:23.3394564Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:23.3404880Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:23.3405362Z JOB_TIMEOUT: 300 2024-12-18T03:40:23.3405800Z ##[endgroup] 2024-12-18T03:40:23.3582664Z ##[group]Run set -x 2024-12-18T03:40:23.3583218Z set -x 2024-12-18T03:40:23.3583620Z  2024-12-18T03:40:23.3584057Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-12-18T03:40:23.3584702Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-12-18T03:40:23.3585361Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-12-18T03:40:23.3585950Z  TEST_COMMAND=.ci/caffe2/test.sh 2024-12-18T03:40:23.3586445Z else 2024-12-18T03:40:23.3586854Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T03:40:23.3587339Z fi 2024-12-18T03:40:23.3587675Z  2024-12-18T03:40:23.3588216Z # detached container should get cleaned up by teardown_ec2_linux 2024-12-18T03:40:23.3589056Z # TODO: Stop building test binaries as part of the build phase 2024-12-18T03:40:23.3589788Z # Used for GPU_FLAG since that doesn't play nice 2024-12-18T03:40:23.3590470Z # shellcheck disable=SC2086,SC2090 2024-12-18T03:40:23.3591019Z container_name=$(docker run \ 2024-12-18T03:40:23.3591540Z  ${GPU_FLAG:-} \ 2024-12-18T03:40:23.3592013Z  -e BUILD_ENVIRONMENT \ 2024-12-18T03:40:23.3592817Z  -e PR_NUMBER \ 2024-12-18T03:40:23.3593264Z  -e GITHUB_ACTIONS \ 2024-12-18T03:40:23.3593725Z  -e GITHUB_REPOSITORY \ 2024-12-18T03:40:23.3594193Z  -e GITHUB_WORKFLOW \ 2024-12-18T03:40:23.3594659Z  -e GITHUB_JOB \ 2024-12-18T03:40:23.3595092Z  -e GITHUB_RUN_ID \ 2024-12-18T03:40:23.3595529Z  -e GITHUB_RUN_NUMBER \ 2024-12-18T03:40:23.3595999Z  -e GITHUB_RUN_ATTEMPT \ 2024-12-18T03:40:23.3596460Z  -e JOB_ID \ 2024-12-18T03:40:23.3597051Z  -e JOB_NAME \ 2024-12-18T03:40:23.3597668Z  -e BRANCH \ 2024-12-18T03:40:23.3598143Z  -e SHA1 \ 2024-12-18T03:40:23.3598669Z  -e AWS_DEFAULT_REGION \ 2024-12-18T03:40:23.3599269Z  -e IN_WHEEL_TEST \ 2024-12-18T03:40:23.3599883Z  -e SHARD_NUMBER \ 2024-12-18T03:40:23.3600452Z  -e TEST_CONFIG \ 2024-12-18T03:40:23.3601333Z  -e NUM_TEST_SHARDS \ 2024-12-18T03:40:23.3601821Z  -e REENABLED_ISSUES \ 2024-12-18T03:40:23.3602350Z  -e CONTINUE_THROUGH_ERROR \ 2024-12-18T03:40:23.3602915Z  -e VERBOSE_TEST_LOGS \ 2024-12-18T03:40:23.3603424Z  -e TEST_SHOWLOCALS \ 2024-12-18T03:40:23.3603923Z  -e NO_TEST_TIMEOUT \ 2024-12-18T03:40:23.3604408Z  -e NO_TD \ 2024-12-18T03:40:23.3604949Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-12-18T03:40:23.3605495Z  -e SCCACHE_BUCKET \ 2024-12-18T03:40:23.3605987Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-12-18T03:40:23.3606636Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-12-18T03:40:23.3607260Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-12-18T03:40:23.3607820Z  -e TESTS_TO_INCLUDE \ 2024-12-18T03:40:23.3608401Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-12-18T03:40:23.3609018Z  --ulimit stack=10485760:83886080 \ 2024-12-18T03:40:23.3609535Z  --ulimit core=0 \ 2024-12-18T03:40:23.3610032Z  --security-opt seccomp=unconfined \ 2024-12-18T03:40:23.3610580Z  --cap-add=SYS_PTRACE \ 2024-12-18T03:40:23.3611052Z  --shm-size="8g" \ 2024-12-18T03:40:23.3611485Z  --tty \ 2024-12-18T03:40:23.3611871Z  --detach \ 2024-12-18T03:40:23.3612303Z  --name="${container_name}" \ 2024-12-18T03:40:23.3612804Z  --user jenkins \ 2024-12-18T03:40:23.3613352Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-12-18T03:40:23.3613974Z  -w /var/lib/jenkins/workspace \ 2024-12-18T03:40:23.3614495Z  "${DOCKER_IMAGE}" 2024-12-18T03:40:23.3614933Z ) 2024-12-18T03:40:23.3615343Z # save container name for later step 2024-12-18T03:40:23.3616388Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2024-12-18T03:40:23.3617578Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2024-12-18T03:40:23.3619062Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2024-12-18T03:40:23.3673838Z shell: /usr/bin/bash -e {0} 2024-12-18T03:40:23.3674311Z env: 2024-12-18T03:40:23.3674676Z GIT_DEFAULT_BRANCH: main 2024-12-18T03:40:23.3675222Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T03:40:23.3676165Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T03:40:23.3677021Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T03:40:23.3677492Z AWS_REGION: us-east-1 2024-12-18T03:40:23.3677987Z AWS_ACCESS_KEY_ID: *** 2024-12-18T03:40:23.3678602Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T03:40:23.3687812Z AWS_SESSION_TOKEN: *** 2024-12-18T03:40:23.3688339Z BUILD_ENVIRONMENT: linux-focal-rocm6.2-py3.10 2024-12-18T03:40:23.3688919Z PR_NUMBER: 2024-12-18T03:40:23.3689339Z GITHUB_REPOSITORY: pytorch/pytorch 2024-12-18T03:40:23.3689856Z GITHUB_WORKFLOW: periodic 2024-12-18T03:40:23.3690282Z GITHUB_JOB: test 2024-12-18T03:40:23.3690672Z GITHUB_RUN_ID: 12383255654 2024-12-18T03:40:23.3691114Z GITHUB_RUN_NUMBER: 15427 2024-12-18T03:40:23.3691545Z GITHUB_RUN_ATTEMPT: 1 2024-12-18T03:40:23.3691952Z JOB_ID: 34566686857 2024-12-18T03:40:23.3692779Z JOB_NAME: linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:23.3693679Z BRANCH: release/2.6 2024-12-18T03:40:23.3694146Z SHA1: 0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:23.3694703Z CONTINUE_THROUGH_ERROR: False 2024-12-18T03:40:23.3695171Z VERBOSE_TEST_LOGS: False 2024-12-18T03:40:23.3695603Z TEST_SHOWLOCALS: False 2024-12-18T03:40:23.3696032Z NO_TEST_TIMEOUT: False 2024-12-18T03:40:23.3696441Z NO_TD: False 2024-12-18T03:40:23.3696821Z TEST_CONFIG: distributed 2024-12-18T03:40:23.3697639Z SHARD_NUMBER: 1 2024-12-18T03:40:23.3698019Z NUM_TEST_SHARDS: 3 2024-12-18T03:40:23.3698411Z REENABLED_ISSUES: 2024-12-18T03:40:23.3698896Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-12-18T03:40:23.3700081Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:40:23.3701374Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:40:23.3702096Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-12-18T03:40:23.3702606Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-12-18T03:40:23.3703082Z TESTS_TO_INCLUDE: 2024-12-18T03:40:23.3703475Z ##[endgroup] 2024-12-18T03:40:23.3796919Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2024-12-18T03:40:23.3797663Z + [[ linux-focal-rocm6.2-py3.10 == *onnx* ]] 2024-12-18T03:40:23.3798375Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-12-18T03:40:23.3820285Z +++ nproc --ignore=2 2024-12-18T03:40:23.3857293Z ++ docker run --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e SCCACHE_BUCKET -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE --env-file=/tmp/github_env_12383255654 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-rocm-n-py3:45e1356b47a284893081276eff3000b7b534f3b1 2024-12-18T03:40:23.7736446Z + container_name=32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T03:40:23.7737771Z + echo CONTAINER_NAME=32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T03:40:23.7739987Z + docker exec -t 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2024-12-18T03:40:36.3483729Z Processing ./dist/torch-2.6.0a0+git0cdf8b1-cp310-cp310-linux_x86_64.whl 2024-12-18T03:40:36.7605129Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.16.1) 2024-12-18T03:40:36.7607400Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (4.12.2) 2024-12-18T03:40:36.7609493Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2.8.8) 2024-12-18T03:40:36.7611483Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (3.1.4) 2024-12-18T03:40:36.7613797Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (2024.10.0) 2024-12-18T03:40:36.7619959Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.6.0a0+git0cdf8b1) (1.13.1) 2024-12-18T03:40:36.7634570Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.6.0a0+git0cdf8b1) (1.3.0) 2024-12-18T03:40:36.7937876Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.6.0a0+git0cdf8b1) (3.0.2) 2024-12-18T03:40:37.0991196Z Installing collected packages: torch 2024-12-18T03:40:45.7954827Z Successfully installed torch-2.6.0a0+git0cdf8b1 2024-12-18T03:40:45.8518144Z + export TERM=vt100 2024-12-18T03:40:45.8518701Z + TERM=vt100 2024-12-18T03:40:45.8528020Z ++ dirname .ci/pytorch/test.sh 2024-12-18T03:40:45.8556192Z + source .ci/pytorch/common.sh 2024-12-18T03:40:45.8567921Z +++ dirname .ci/pytorch/common.sh 2024-12-18T03:40:45.8602038Z ++ source .ci/pytorch/common_utils.sh 2024-12-18T03:40:45.8603097Z +++ declare -f -t trap_add 2024-12-18T03:40:45.8609864Z ++ set -ex 2024-12-18T03:40:45.8610500Z ++ [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:40:45.8611175Z ++ unset HIP_PLATFORM 2024-12-18T03:40:45.8611721Z ++ export PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:40:45.8612326Z ++ PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:40:45.8612895Z ++ export HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:40:45.8613490Z ++ HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:40:45.8614027Z ++ export HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:40:45.8614618Z ++ HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:40:45.8615177Z ++ BUILD_TEST_LIBTORCH=0 2024-12-18T03:40:45.8615753Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-12-18T03:40:45.8616411Z + echo 'Environment variables:' 2024-12-18T03:40:45.8616964Z Environment variables: 2024-12-18T03:40:45.8617447Z + env 2024-12-18T03:40:45.8645947Z INSTALLED_DB=yes 2024-12-18T03:40:45.8646917Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:40:45.8648024Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-12-18T03:40:45.8648824Z CONTINUE_THROUGH_ERROR=False 2024-12-18T03:40:45.8649571Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-12-18T03:40:45.8650311Z HOSTNAME=32ffe0408e4f 2024-12-18T03:40:45.8651499Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.8656254Z GITHUB_ACTION=__self 2024-12-18T03:40:45.8657051Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T03:40:45.8657676Z GITHUB_RUN_NUMBER=15427 2024-12-18T03:40:45.8658213Z TEST_CONFIG=distributed 2024-12-18T03:40:45.8658756Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T03:40:45.8659361Z AWS_DEFAULT_REGION=us-east-1 2024-12-18T03:40:45.8659913Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T03:40:45.8660444Z GITHUB_REF_TYPE=branch 2024-12-18T03:40:45.8666428Z *** 2024-12-18T03:40:45.8666946Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T03:40:45.8667539Z GITHUB_ACTIONS=true 2024-12-18T03:40:45.8668100Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.8668837Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.8669909Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/release/2.6 2024-12-18T03:40:45.8670920Z VERBOSE_TEST_LOGS=False 2024-12-18T03:40:45.8671452Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T03:40:45.8672008Z SHARD_NUMBER=1 2024-12-18T03:40:45.8672694Z GITHUB_REF_PROTECTED=true 2024-12-18T03:40:45.8673213Z HOME=/var/lib/jenkins 2024-12-18T03:40:45.8673764Z GITHUB_API_URL=https://api.github.com 2024-12-18T03:40:45.8674428Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T03:40:45.8674994Z LANG=C.UTF-8 2024-12-18T03:40:45.8675445Z PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:40:45.8675965Z NUM_TEST_SHARDS=3 2024-12-18T03:40:45.8677134Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.8678878Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:45.8679957Z MAGMA_HOME=/opt/rocm/magma 2024-12-18T03:40:45.8681119Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.8682313Z HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:40:45.8683186Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T03:40:45.8684168Z GITHUB_EVENT_NAME=push 2024-12-18T03:40:45.8684665Z GITHUB_RUN_ID=12383255654 2024-12-18T03:40:45.8686597Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.8687968Z GITHUB_ACTOR=malfet 2024-12-18T03:40:45.8688426Z PR_NUMBER= 2024-12-18T03:40:45.8688855Z GITHUB_RUN_ATTEMPT=1 2024-12-18T03:40:45.8689361Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T03:40:45.8690020Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T03:40:45.8690677Z TERM=vt100 2024-12-18T03:40:45.8691120Z INSTALLED_VISION=yes 2024-12-18T03:40:45.8691597Z BRANCH=release/2.6 2024-12-18T03:40:45.8692043Z TESTS_TO_INCLUDE= 2024-12-18T03:40:45.8693035Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-12-18T03:40:45.8694185Z GITHUB_SERVER_URL=https://github.com 2024-12-18T03:40:45.8694804Z PYTORCH_ROCM_ARCH=gfx90a 2024-12-18T03:40:45.8695326Z REENABLED_ISSUES= 2024-12-18T03:40:45.8695767Z SHLVL=1 2024-12-18T03:40:45.8696151Z MAX_JOBS=126 2024-12-18T03:40:45.8696586Z GITHUB_ACTOR_ID=2453524 2024-12-18T03:40:45.8697244Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.8697987Z GITHUB_REF_NAME=release/2.6 2024-12-18T03:40:45.8698498Z ROCM_PATH=/opt/rocm 2024-12-18T03:40:45.8699214Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:40:45.8700013Z GITHUB_JOB=test 2024-12-18T03:40:45.8700450Z NO_TEST_TIMEOUT=False 2024-12-18T03:40:45.8700958Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T03:40:45.8701509Z LC_ALL=C.UTF-8 2024-12-18T03:40:45.8701959Z GITHUB_RETENTION_DAYS=90 2024-12-18T03:40:45.8702489Z GITHUB_ACTION_REPOSITORY= 2024-12-18T03:40:45.8704308Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:40:45.8706471Z GITHUB_BASE_REF= 2024-12-18T03:40:45.8706951Z CI=true 2024-12-18T03:40:45.8707386Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:40:45.8707964Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T03:40:45.8708516Z JOB_ID=34566686857 2024-12-18T03:40:45.8708997Z INSTALLED_PROTOBUF=yes 2024-12-18T03:40:45.8709480Z GITHUB_HEAD_REF= 2024-12-18T03:40:45.8709935Z GITHUB_ACTION_REF= 2024-12-18T03:40:45.8710494Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T03:40:45.8711218Z TEST_SHOWLOCALS=False 2024-12-18T03:40:45.8711728Z GITHUB_WORKFLOW=periodic 2024-12-18T03:40:45.8712598Z DEBIAN_FRONTEND=noninteractive 2024-12-18T03:40:45.8713863Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.8715104Z NO_TD=False 2024-12-18T03:40:45.8715533Z OLDPWD=/var/lib/jenkins 2024-12-18T03:40:45.8716006Z _=/usr/bin/env 2024-12-18T03:40:45.8716642Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-12-18T03:40:45.9117528Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-12-18T03:40:45.9118843Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T03:40:45.9119991Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-12-18T03:40:45.9121076Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-12-18T03:40:45.9121898Z + BUILD_DIR=build 2024-12-18T03:40:45.9122391Z + BUILD_RENAMED_DIR=build_renamed 2024-12-18T03:40:45.9122977Z + BUILD_BIN_DIR=build/bin 2024-12-18T03:40:45.9123485Z + SHARD_NUMBER=1 2024-12-18T03:40:45.9123932Z + NUM_TEST_SHARDS=3 2024-12-18T03:40:45.9124396Z + export VALGRIND=ON 2024-12-18T03:40:45.9124863Z + VALGRIND=ON 2024-12-18T03:40:45.9125371Z + [[ linux-focal-rocm6.2-py3.10 == *clang9* ]] 2024-12-18T03:40:45.9126043Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-12-18T03:40:45.9126635Z + [[ 0 == \1 ]] 2024-12-18T03:40:45.9127079Z + [[ False == \1 ]] 2024-12-18T03:40:45.9127599Z + [[ linux-focal-rocm6.2-py3.10 != *bazel* ]] 2024-12-18T03:40:45.9128960Z ++ realpath build/custom_test_artifacts 2024-12-18T03:40:45.9163994Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2024-12-18T03:40:45.9165037Z + [[ -n '' ]] 2024-12-18T03:40:45.9165531Z + echo 'Environment variables' 2024-12-18T03:40:45.9166099Z Environment variables 2024-12-18T03:40:45.9166565Z + env 2024-12-18T03:40:45.9184838Z INSTALLED_DB=yes 2024-12-18T03:40:45.9185784Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T03:40:45.9186767Z AOTRITON_INSTALLED_PREFIX=/opt/rocm/aotriton 2024-12-18T03:40:45.9187487Z CONTINUE_THROUGH_ERROR=False 2024-12-18T03:40:45.9188136Z BUILD_ENVIRONMENT=linux-focal-rocm6.2-py3.10 2024-12-18T03:40:45.9188799Z HOSTNAME=32ffe0408e4f 2024-12-18T03:40:45.9189984Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.9191237Z GITHUB_ACTION=__self 2024-12-18T03:40:45.9191765Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-12-18T03:40:45.9192522Z GITHUB_RUN_NUMBER=15427 2024-12-18T03:40:45.9193041Z TEST_CONFIG=distributed 2024-12-18T03:40:45.9193556Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-12-18T03:40:45.9194163Z AWS_DEFAULT_REGION=us-east-1 2024-12-18T03:40:45.9194714Z GITHUB_TRIGGERING_ACTOR=malfet 2024-12-18T03:40:45.9195251Z GITHUB_REF_TYPE=branch 2024-12-18T03:40:45.9195843Z *** 2024-12-18T03:40:45.9196289Z GITHUB_REPOSITORY_ID=65600975 2024-12-18T03:40:45.9196842Z GITHUB_ACTIONS=true 2024-12-18T03:40:45.9197384Z SHA1=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.9198104Z GITHUB_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.9199151Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/release/2.6 2024-12-18T03:40:45.9200108Z VERBOSE_TEST_LOGS=False 2024-12-18T03:40:45.9200626Z GITHUB_REF=refs/heads/release/2.6 2024-12-18T03:40:45.9201181Z SHARD_NUMBER=1 2024-12-18T03:40:45.9202118Z GITHUB_REF_PROTECTED=true 2024-12-18T03:40:45.9202645Z HOME=/var/lib/jenkins 2024-12-18T03:40:45.9203174Z GITHUB_API_URL=https://api.github.com 2024-12-18T03:40:45.9203826Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-12-18T03:40:45.9204386Z LANG=C.UTF-8 2024-12-18T03:40:45.9204828Z PYTORCH_TEST_WITH_ROCM=1 2024-12-18T03:40:45.9205327Z NUM_TEST_SHARDS=3 2024-12-18T03:40:45.9206471Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.9208214Z JOB_NAME=linux-focal-rocm6.2-py3.10 / test (distributed, 1, 3, linux.rocm.gpu, module:rocm, oncall:distributed) 2024-12-18T03:40:45.9209283Z MAGMA_HOME=/opt/rocm/magma 2024-12-18T03:40:45.9210430Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.9211614Z HSAKMT_DEBUG_LEVEL=4 2024-12-18T03:40:45.9212505Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2024-12-18T03:40:45.9213485Z GITHUB_EVENT_NAME=push 2024-12-18T03:40:45.9213974Z GITHUB_RUN_ID=12383255654 2024-12-18T03:40:45.9215211Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.9216506Z GITHUB_ACTOR=malfet 2024-12-18T03:40:45.9216954Z PR_NUMBER= 2024-12-18T03:40:45.9217374Z GITHUB_RUN_ATTEMPT=1 2024-12-18T03:40:45.9217829Z VALGRIND=ON 2024-12-18T03:40:45.9218279Z ANACONDA_PYTHON_VERSION=3.10 2024-12-18T03:40:45.9218929Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-12-18T03:40:45.9219589Z TERM=vt100 2024-12-18T03:40:45.9220011Z INSTALLED_VISION=yes 2024-12-18T03:40:45.9220482Z BRANCH=release/2.6 2024-12-18T03:40:45.9220926Z TESTS_TO_INCLUDE= 2024-12-18T03:40:45.9221867Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2024-12-18T03:40:45.9222974Z GITHUB_SERVER_URL=https://github.com 2024-12-18T03:40:45.9223593Z PYTORCH_ROCM_ARCH=gfx90a 2024-12-18T03:40:45.9224089Z REENABLED_ISSUES= 2024-12-18T03:40:45.9224524Z SHLVL=1 2024-12-18T03:40:45.9225294Z MAX_JOBS=126 2024-12-18T03:40:45.9225725Z GITHUB_ACTOR_ID=2453524 2024-12-18T03:40:45.9226377Z GITHUB_WORKFLOW_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7 2024-12-18T03:40:45.9227111Z GITHUB_REF_NAME=release/2.6 2024-12-18T03:40:45.9227622Z ROCM_PATH=/opt/rocm 2024-12-18T03:40:45.9228333Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-12-18T03:40:45.9229140Z GITHUB_JOB=test 2024-12-18T03:40:45.9229587Z NO_TEST_TIMEOUT=False 2024-12-18T03:40:45.9230101Z GITHUB_REPOSITORY=pytorch/pytorch 2024-12-18T03:40:45.9230644Z LC_ALL=C.UTF-8 2024-12-18T03:40:45.9231084Z GITHUB_RETENTION_DAYS=90 2024-12-18T03:40:45.9231597Z GITHUB_ACTION_REPOSITORY= 2024-12-18T03:40:45.9233765Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:40:45.9235604Z GITHUB_BASE_REF= 2024-12-18T03:40:45.9236079Z CI=true 2024-12-18T03:40:45.9236516Z HSA_FORCE_FINE_GRAIN_PCIE=1 2024-12-18T03:40:45.9237069Z GITHUB_REPOSITORY_OWNER=pytorch 2024-12-18T03:40:45.9237592Z JOB_ID=34566686857 2024-12-18T03:40:45.9238046Z INSTALLED_PROTOBUF=yes 2024-12-18T03:40:45.9238525Z GITHUB_HEAD_REF= 2024-12-18T03:40:45.9238961Z GITHUB_ACTION_REF= 2024-12-18T03:40:45.9239512Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-12-18T03:40:45.9240183Z TEST_SHOWLOCALS=False 2024-12-18T03:40:45.9240677Z GITHUB_WORKFLOW=periodic 2024-12-18T03:40:45.9241211Z DEBIAN_FRONTEND=noninteractive 2024-12-18T03:40:45.9242427Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_29e59f84-fd0d-4b84-8a02-bf60be63f8b8 2024-12-18T03:40:45.9243661Z NO_TD=False 2024-12-18T03:40:45.9244088Z OLDPWD=/var/lib/jenkins 2024-12-18T03:40:45.9244552Z _=/usr/bin/env 2024-12-18T03:40:45.9245002Z + echo 'Testing pytorch' 2024-12-18T03:40:45.9245860Z Testing pytorch 2024-12-18T03:40:45.9246342Z + export LANG=C.UTF-8 2024-12-18T03:40:45.9246810Z + LANG=C.UTF-8 2024-12-18T03:40:45.9247241Z + PR_NUMBER= 2024-12-18T03:40:45.9247700Z + [[ distributed == \d\e\f\a\u\l\t ]] 2024-12-18T03:40:45.9248318Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-12-18T03:40:45.9248976Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:40:45.9249610Z + export HIP_VISIBLE_DEVICES=0,1 2024-12-18T03:40:45.9250161Z + HIP_VISIBLE_DEVICES=0,1 2024-12-18T03:40:45.9250667Z + [[ distributed == \s\l\o\w ]] 2024-12-18T03:40:45.9251303Z + [[ linux-focal-rocm6.2-py3.10 == *slow-gradcheck* ]] 2024-12-18T03:40:45.9252028Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T03:40:45.9252662Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:40:45.9253314Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T03:40:45.9253963Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-12-18T03:40:45.9254594Z + [[ distributed == *crossref* ]] 2024-12-18T03:40:45.9255227Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T03:40:45.9255854Z + export VALGRIND=OFF 2024-12-18T03:40:45.9256345Z + VALGRIND=OFF 2024-12-18T03:40:45.9256773Z + rocminfo 2024-12-18T03:40:45.9417926Z ROCk module version 6.8.5 is loaded 2024-12-18T03:40:46.0766872Z ===================== 2024-12-18T03:40:46.0767592Z HSA System Attributes 2024-12-18T03:40:46.0768175Z ===================== 2024-12-18T03:40:46.0768707Z Runtime Version: 1.14 2024-12-18T03:40:46.0769264Z Runtime Ext Version: 1.6 2024-12-18T03:40:46.0769926Z System Timestamp Freq.: 1000.000000MHz 2024-12-18T03:40:46.0771006Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2024-12-18T03:40:46.0772118Z Machine Model: LARGE 2024-12-18T03:40:46.0772911Z System Endianness: LITTLE 2024-12-18T03:40:46.0773606Z Mwaitx: DISABLED 2024-12-18T03:40:46.0774214Z DMAbuf Support: YES 2024-12-18T03:40:46.0774570Z 2024-12-18T03:40:46.0774748Z ========== 2024-12-18T03:40:46.0775313Z HSA Agents 2024-12-18T03:40:46.0776402Z ========== 2024-12-18T03:40:46.0776875Z ******* 2024-12-18T03:40:46.0777344Z Agent 1 2024-12-18T03:40:46.0777818Z ******* 2024-12-18T03:40:46.0778402Z Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.0779124Z Uuid: CPU-XX 2024-12-18T03:40:46.0779876Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.0780675Z Vendor Name: CPU 2024-12-18T03:40:46.0781445Z Feature: None specified 2024-12-18T03:40:46.0782201Z Profile: FULL_PROFILE 2024-12-18T03:40:46.0783008Z Float Round Mode: NEAR 2024-12-18T03:40:46.0783852Z Max Queue Number: 0(0x0) 2024-12-18T03:40:46.0784604Z Queue Min Size: 0(0x0) 2024-12-18T03:40:46.0785344Z Queue Max Size: 0(0x0) 2024-12-18T03:40:46.0786070Z Queue Type: MULTI 2024-12-18T03:40:46.0786758Z Node: 0 2024-12-18T03:40:46.0787463Z Device Type: CPU 2024-12-18T03:40:46.0788115Z Cache Info: 2024-12-18T03:40:46.0788706Z L1: 32768(0x8000) KB 2024-12-18T03:40:46.0789421Z Chip ID: 0(0x0) 2024-12-18T03:40:46.0790137Z ASIC Revision: 0(0x0) 2024-12-18T03:40:46.0790916Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.0792098Z Max Clock Freq. (MHz): 2000 2024-12-18T03:40:46.0793091Z BDFID: 0 2024-12-18T03:40:46.0793828Z Internal Node ID: 0 2024-12-18T03:40:46.0794578Z Compute Unit: 64 2024-12-18T03:40:46.0795305Z SIMDs per CU: 0 2024-12-18T03:40:46.0796034Z Shader Engines: 0 2024-12-18T03:40:46.0796802Z Shader Arrs. per Eng.: 0 2024-12-18T03:40:46.0797584Z WatchPts on Addr. Ranges:1 2024-12-18T03:40:46.0798297Z Memory Properties: 2024-12-18T03:40:46.0798831Z Features: None 2024-12-18T03:40:46.0799362Z Pool Info: 2024-12-18T03:40:46.0799854Z Pool 1 2024-12-18T03:40:46.0800488Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.0801258Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:40:46.0801997Z Allocatable: TRUE 2024-12-18T03:40:46.0802760Z Alloc Granule: 4KB 2024-12-18T03:40:46.0803569Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0804369Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0805150Z Accessible by all: TRUE 2024-12-18T03:40:46.0805817Z Pool 2 2024-12-18T03:40:46.0806440Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:40:46.0807167Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:40:46.0807884Z Allocatable: TRUE 2024-12-18T03:40:46.0808629Z Alloc Granule: 4KB 2024-12-18T03:40:46.0809439Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0810232Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0811395Z Accessible by all: TRUE 2024-12-18T03:40:46.0812060Z Pool 3 2024-12-18T03:40:46.0812682Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.0813418Z Size: 528196736(0x1f7ba480) KB 2024-12-18T03:40:46.0814141Z Allocatable: TRUE 2024-12-18T03:40:46.0814892Z Alloc Granule: 4KB 2024-12-18T03:40:46.0815681Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0816475Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0817247Z Accessible by all: TRUE 2024-12-18T03:40:46.0817914Z ISA Info: 2024-12-18T03:40:46.0818401Z ******* 2024-12-18T03:40:46.0818879Z Agent 2 2024-12-18T03:40:46.0819347Z ******* 2024-12-18T03:40:46.0819907Z Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.0820932Z Uuid: CPU-XX 2024-12-18T03:40:46.0821695Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.0822465Z Vendor Name: CPU 2024-12-18T03:40:46.0823191Z Feature: None specified 2024-12-18T03:40:46.0823919Z Profile: FULL_PROFILE 2024-12-18T03:40:46.0824661Z Float Round Mode: NEAR 2024-12-18T03:40:46.0825413Z Max Queue Number: 0(0x0) 2024-12-18T03:40:46.0826165Z Queue Min Size: 0(0x0) 2024-12-18T03:40:46.0827240Z Queue Max Size: 0(0x0) 2024-12-18T03:40:46.0827978Z Queue Type: MULTI 2024-12-18T03:40:46.0828691Z Node: 1 2024-12-18T03:40:46.0829382Z Device Type: CPU 2024-12-18T03:40:46.0830033Z Cache Info: 2024-12-18T03:40:46.0830579Z L1: 32768(0x8000) KB 2024-12-18T03:40:46.0831271Z Chip ID: 0(0x0) 2024-12-18T03:40:46.0831977Z ASIC Revision: 0(0x0) 2024-12-18T03:40:46.0833088Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.0833853Z Max Clock Freq. (MHz): 2000 2024-12-18T03:40:46.0834568Z BDFID: 0 2024-12-18T03:40:46.0835310Z Internal Node ID: 1 2024-12-18T03:40:46.0836050Z Compute Unit: 64 2024-12-18T03:40:46.0836773Z SIMDs per CU: 0 2024-12-18T03:40:46.0837511Z Shader Engines: 0 2024-12-18T03:40:46.0838265Z Shader Arrs. per Eng.: 0 2024-12-18T03:40:46.0839036Z WatchPts on Addr. Ranges:1 2024-12-18T03:40:46.0839738Z Memory Properties: 2024-12-18T03:40:46.0840263Z Features: None 2024-12-18T03:40:46.0840780Z Pool Info: 2024-12-18T03:40:46.0841278Z Pool 1 2024-12-18T03:40:46.0841912Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.0842658Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:40:46.0843400Z Allocatable: TRUE 2024-12-18T03:40:46.0844162Z Alloc Granule: 4KB 2024-12-18T03:40:46.0845339Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0846151Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0846940Z Accessible by all: TRUE 2024-12-18T03:40:46.0847611Z Pool 2 2024-12-18T03:40:46.0848232Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2024-12-18T03:40:46.0848958Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:40:46.0849673Z Allocatable: TRUE 2024-12-18T03:40:46.0850423Z Alloc Granule: 4KB 2024-12-18T03:40:46.0851210Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0852028Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0852816Z Accessible by all: TRUE 2024-12-18T03:40:46.0853501Z Pool 3 2024-12-18T03:40:46.0854101Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.0854821Z Size: 528455496(0x1f7f9748) KB 2024-12-18T03:40:46.0855533Z Allocatable: TRUE 2024-12-18T03:40:46.0856282Z Alloc Granule: 4KB 2024-12-18T03:40:46.0857081Z Alloc Recommended Granule:4KB 2024-12-18T03:40:46.0857869Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0858643Z Accessible by all: TRUE 2024-12-18T03:40:46.0859317Z ISA Info: 2024-12-18T03:40:46.0859794Z ******* 2024-12-18T03:40:46.0860267Z Agent 3 2024-12-18T03:40:46.0861066Z ******* 2024-12-18T03:40:46.0861612Z Name: gfx90a 2024-12-18T03:40:46.0862324Z Uuid: GPU-87b565030406a196 2024-12-18T03:40:46.0863074Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.0863837Z Vendor Name: AMD 2024-12-18T03:40:46.0864567Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.0865302Z Profile: BASE_PROFILE 2024-12-18T03:40:46.0866044Z Float Round Mode: NEAR 2024-12-18T03:40:46.0866802Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.0867539Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.0868263Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.0868999Z Queue Type: MULTI 2024-12-18T03:40:46.0869685Z Node: 2 2024-12-18T03:40:46.0870382Z Device Type: GPU 2024-12-18T03:40:46.0871036Z Cache Info: 2024-12-18T03:40:46.0871573Z L1: 16(0x10) KB 2024-12-18T03:40:46.0872212Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.0873011Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.0873718Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.0874466Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.0875220Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.0875928Z BDFID: 12800 2024-12-18T03:40:46.0876639Z Internal Node ID: 2 2024-12-18T03:40:46.0877383Z Compute Unit: 104 2024-12-18T03:40:46.0878108Z SIMDs per CU: 4 2024-12-18T03:40:46.0879195Z Shader Engines: 8 2024-12-18T03:40:46.0879956Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.0880725Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.0881518Z Coherent Host Access: FALSE 2024-12-18T03:40:46.0882215Z Memory Properties: 2024-12-18T03:40:46.0882765Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.0883471Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.0884231Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.0885009Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.0885710Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.0886310Z x 1024(0x400) 2024-12-18T03:40:46.0886931Z y 1024(0x400) 2024-12-18T03:40:46.0887547Z z 1024(0x400) 2024-12-18T03:40:46.0888223Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.0888982Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.0889735Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.0890410Z Grid Max Size per Dimension: 2024-12-18T03:40:46.0890967Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.0891593Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.0892225Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.0892945Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.0894205Z Packet Processor uCode:: 83 2024-12-18T03:40:46.0895010Z SDMA engine uCode:: 8 2024-12-18T03:40:46.0895781Z IOMMU Support:: None 2024-12-18T03:40:46.0896447Z Pool Info: 2024-12-18T03:40:46.0896947Z Pool 1 2024-12-18T03:40:46.0897572Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.0898313Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0899059Z Allocatable: TRUE 2024-12-18T03:40:46.0899826Z Alloc Granule: 4KB 2024-12-18T03:40:46.0900615Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0901420Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0902199Z Accessible by all: FALSE 2024-12-18T03:40:46.0902876Z Pool 2 2024-12-18T03:40:46.0903487Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.0904250Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0905000Z Allocatable: TRUE 2024-12-18T03:40:46.0905765Z Alloc Granule: 4KB 2024-12-18T03:40:46.0906564Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0907370Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0908147Z Accessible by all: FALSE 2024-12-18T03:40:46.0908813Z Pool 3 2024-12-18T03:40:46.0909416Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.0910120Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0910855Z Allocatable: TRUE 2024-12-18T03:40:46.0911610Z Alloc Granule: 4KB 2024-12-18T03:40:46.0912828Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0913619Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0914390Z Accessible by all: FALSE 2024-12-18T03:40:46.0915059Z Pool 4 2024-12-18T03:40:46.0915641Z Segment: GROUP 2024-12-18T03:40:46.0916326Z Size: 64(0x40) KB 2024-12-18T03:40:46.0917035Z Allocatable: FALSE 2024-12-18T03:40:46.0917787Z Alloc Granule: 0KB 2024-12-18T03:40:46.0918564Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.0919383Z Alloc Alignment: 0KB 2024-12-18T03:40:46.0920150Z Accessible by all: FALSE 2024-12-18T03:40:46.0920829Z ISA Info: 2024-12-18T03:40:46.0921317Z ISA 1 2024-12-18T03:40:46.0921964Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.0922792Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.0923585Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.0924368Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.0925169Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.0925912Z Fast f16: TRUE 2024-12-18T03:40:46.0926656Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.0927368Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.0928336Z x 1024(0x400) 2024-12-18T03:40:46.0928993Z y 1024(0x400) 2024-12-18T03:40:46.0929624Z z 1024(0x400) 2024-12-18T03:40:46.0930312Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.0930999Z Grid Max Size per Dimension: 2024-12-18T03:40:46.0931588Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.0932230Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.0932868Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.0933582Z FBarrier Max Size: 32 2024-12-18T03:40:46.0934244Z ******* 2024-12-18T03:40:46.0934720Z Agent 4 2024-12-18T03:40:46.0935185Z ******* 2024-12-18T03:40:46.0935724Z Name: gfx90a 2024-12-18T03:40:46.0936441Z Uuid: GPU-72b52101971d0607 2024-12-18T03:40:46.0937203Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.0937980Z Vendor Name: AMD 2024-12-18T03:40:46.0938717Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.0939456Z Profile: BASE_PROFILE 2024-12-18T03:40:46.0940208Z Float Round Mode: NEAR 2024-12-18T03:40:46.0940961Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.0941690Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.0942418Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.0943136Z Queue Type: MULTI 2024-12-18T03:40:46.0943832Z Node: 3 2024-12-18T03:40:46.0944527Z Device Type: GPU 2024-12-18T03:40:46.0945516Z Cache Info: 2024-12-18T03:40:46.0946050Z L1: 16(0x10) KB 2024-12-18T03:40:46.0946689Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.0947402Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.0948230Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.0949049Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.0949798Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.0950496Z BDFID: 13568 2024-12-18T03:40:46.0951209Z Internal Node ID: 3 2024-12-18T03:40:46.0951939Z Compute Unit: 104 2024-12-18T03:40:46.0952949Z SIMDs per CU: 4 2024-12-18T03:40:46.0953683Z Shader Engines: 8 2024-12-18T03:40:46.0954588Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.0955494Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.0956358Z Coherent Host Access: FALSE 2024-12-18T03:40:46.0957056Z Memory Properties: 2024-12-18T03:40:46.0957612Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.0958310Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.0959065Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.0959832Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.0960525Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.0961127Z x 1024(0x400) 2024-12-18T03:40:46.0962123Z y 1024(0x400) 2024-12-18T03:40:46.0962742Z z 1024(0x400) 2024-12-18T03:40:46.0963444Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.0964207Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.0964959Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.0965632Z Grid Max Size per Dimension: 2024-12-18T03:40:46.0966196Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.0966822Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.0967526Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.0968371Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.0969263Z Packet Processor uCode:: 83 2024-12-18T03:40:46.0970302Z SDMA engine uCode:: 8 2024-12-18T03:40:46.0971126Z IOMMU Support:: None 2024-12-18T03:40:46.0971924Z Pool Info: 2024-12-18T03:40:46.0972516Z Pool 1 2024-12-18T03:40:46.0973212Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.0973958Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0974689Z Allocatable: TRUE 2024-12-18T03:40:46.0975459Z Alloc Granule: 4KB 2024-12-18T03:40:46.0976255Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0977058Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0977827Z Accessible by all: FALSE 2024-12-18T03:40:46.0978494Z Pool 2 2024-12-18T03:40:46.0979133Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.0979857Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0980937Z Allocatable: TRUE 2024-12-18T03:40:46.0981679Z Alloc Granule: 4KB 2024-12-18T03:40:46.0982464Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0983274Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0984049Z Accessible by all: FALSE 2024-12-18T03:40:46.0984714Z Pool 3 2024-12-18T03:40:46.0985361Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.0986091Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.0986814Z Allocatable: TRUE 2024-12-18T03:40:46.0987571Z Alloc Granule: 4KB 2024-12-18T03:40:46.0988364Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.0989169Z Alloc Alignment: 4KB 2024-12-18T03:40:46.0989932Z Accessible by all: FALSE 2024-12-18T03:40:46.0990602Z Pool 4 2024-12-18T03:40:46.0991182Z Segment: GROUP 2024-12-18T03:40:46.0991864Z Size: 64(0x40) KB 2024-12-18T03:40:46.0992712Z Allocatable: FALSE 2024-12-18T03:40:46.0993488Z Alloc Granule: 0KB 2024-12-18T03:40:46.0994265Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.0995053Z Alloc Alignment: 0KB 2024-12-18T03:40:46.0996157Z Accessible by all: FALSE 2024-12-18T03:40:46.0996836Z ISA Info: 2024-12-18T03:40:46.0997328Z ISA 1 2024-12-18T03:40:46.0997960Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.0998796Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.0999582Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1000364Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1001161Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1001894Z Fast f16: TRUE 2024-12-18T03:40:46.1002630Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1003341Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1003986Z x 1024(0x400) 2024-12-18T03:40:46.1004637Z y 1024(0x400) 2024-12-18T03:40:46.1005259Z z 1024(0x400) 2024-12-18T03:40:46.1005955Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1006633Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1007222Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1007963Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1008699Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1009409Z FBarrier Max Size: 32 2024-12-18T03:40:46.1010067Z ******* 2024-12-18T03:40:46.1010533Z Agent 5 2024-12-18T03:40:46.1010988Z ******* 2024-12-18T03:40:46.1011517Z Name: gfx90a 2024-12-18T03:40:46.1012230Z Uuid: GPU-04e3cd7c6eb6c52d 2024-12-18T03:40:46.1012988Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1014154Z Vendor Name: AMD 2024-12-18T03:40:46.1014884Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1015612Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1016353Z Float Round Mode: NEAR 2024-12-18T03:40:46.1017107Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1017838Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1018561Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1019284Z Queue Type: MULTI 2024-12-18T03:40:46.1019956Z Node: 4 2024-12-18T03:40:46.1020648Z Device Type: GPU 2024-12-18T03:40:46.1021300Z Cache Info: 2024-12-18T03:40:46.1021852Z L1: 16(0x10) KB 2024-12-18T03:40:46.1022493Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1023159Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1023871Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1024630Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1025380Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1026091Z BDFID: 4352 2024-12-18T03:40:46.1026809Z Internal Node ID: 4 2024-12-18T03:40:46.1027541Z Compute Unit: 104 2024-12-18T03:40:46.1028576Z SIMDs per CU: 4 2024-12-18T03:40:46.1029327Z Shader Engines: 8 2024-12-18T03:40:46.1030103Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1030880Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1031666Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1032662Z Memory Properties: 2024-12-18T03:40:46.1033235Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1033931Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1034688Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1035446Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1036143Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1036728Z x 1024(0x400) 2024-12-18T03:40:46.1037358Z y 1024(0x400) 2024-12-18T03:40:46.1037940Z z 1024(0x400) 2024-12-18T03:40:46.1038519Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1039193Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1039855Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1040540Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1041106Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1041761Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1042334Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1042951Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1043647Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1044315Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1044969Z IOMMU Support:: None 2024-12-18T03:40:46.1045530Z Pool Info: 2024-12-18T03:40:46.1046279Z Pool 1 2024-12-18T03:40:46.1046813Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1047452Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1048075Z Allocatable: TRUE 2024-12-18T03:40:46.1048720Z Alloc Granule: 4KB 2024-12-18T03:40:46.1049405Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1050089Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1050747Z Accessible by all: FALSE 2024-12-18T03:40:46.1051310Z Pool 2 2024-12-18T03:40:46.1051829Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1052460Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1053078Z Allocatable: TRUE 2024-12-18T03:40:46.1053725Z Alloc Granule: 4KB 2024-12-18T03:40:46.1054390Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1055056Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1055708Z Accessible by all: FALSE 2024-12-18T03:40:46.1056276Z Pool 3 2024-12-18T03:40:46.1056780Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1057383Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1057979Z Allocatable: TRUE 2024-12-18T03:40:46.1058618Z Alloc Granule: 4KB 2024-12-18T03:40:46.1059559Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1060237Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1060897Z Accessible by all: FALSE 2024-12-18T03:40:46.1061461Z Pool 4 2024-12-18T03:40:46.1061952Z Segment: GROUP 2024-12-18T03:40:46.1062532Z Size: 64(0x40) KB 2024-12-18T03:40:46.1063126Z Allocatable: FALSE 2024-12-18T03:40:46.1063759Z Alloc Granule: 0KB 2024-12-18T03:40:46.1064423Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1065092Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1065746Z Accessible by all: FALSE 2024-12-18T03:40:46.1066320Z ISA Info: 2024-12-18T03:40:46.1066733Z ISA 1 2024-12-18T03:40:46.1067283Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1067986Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1068649Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1069310Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1069983Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1070617Z Fast f16: TRUE 2024-12-18T03:40:46.1071273Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1071870Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1072508Z x 1024(0x400) 2024-12-18T03:40:46.1073083Z y 1024(0x400) 2024-12-18T03:40:46.1073609Z z 1024(0x400) 2024-12-18T03:40:46.1074511Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1075093Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1075584Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1076126Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1076654Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1077253Z FBarrier Max Size: 32 2024-12-18T03:40:46.1077812Z ******* 2024-12-18T03:40:46.1078212Z Agent 6 2024-12-18T03:40:46.1078611Z ******* 2024-12-18T03:40:46.1079068Z Name: gfx90a 2024-12-18T03:40:46.1079681Z Uuid: GPU-e46a5b3c036a0534 2024-12-18T03:40:46.1080325Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1080970Z Vendor Name: AMD 2024-12-18T03:40:46.1081595Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1082212Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1082837Z Float Round Mode: NEAR 2024-12-18T03:40:46.1083473Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1084099Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1084714Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1085321Z Queue Type: MULTI 2024-12-18T03:40:46.1085885Z Node: 5 2024-12-18T03:40:46.1086459Z Device Type: GPU 2024-12-18T03:40:46.1087311Z Cache Info: 2024-12-18T03:40:46.1087865Z L1: 16(0x10) KB 2024-12-18T03:40:46.1088511Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1089118Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1089713Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1090335Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1090957Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1091550Z BDFID: 5120 2024-12-18T03:40:46.1092162Z Internal Node ID: 5 2024-12-18T03:40:46.1092829Z Compute Unit: 104 2024-12-18T03:40:46.1093535Z SIMDs per CU: 4 2024-12-18T03:40:46.1094269Z Shader Engines: 8 2024-12-18T03:40:46.1094990Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1095668Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1096337Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1096931Z Memory Properties: 2024-12-18T03:40:46.1097412Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1098006Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1098646Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1099293Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1099883Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1100375Z x 1024(0x400) 2024-12-18T03:40:46.1100893Z y 1024(0x400) 2024-12-18T03:40:46.1101420Z z 1024(0x400) 2024-12-18T03:40:46.1101986Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1102880Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1103514Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1104089Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1104556Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1105095Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1105626Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1106233Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1106914Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1107603Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1108356Z IOMMU Support:: None 2024-12-18T03:40:46.1109031Z Pool Info: 2024-12-18T03:40:46.1109455Z Pool 1 2024-12-18T03:40:46.1109989Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1110622Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1111235Z Allocatable: TRUE 2024-12-18T03:40:46.1111880Z Alloc Granule: 4KB 2024-12-18T03:40:46.1112640Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1113319Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1114020Z Accessible by all: FALSE 2024-12-18T03:40:46.1114690Z Pool 2 2024-12-18T03:40:46.1115303Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1116330Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1116938Z Allocatable: TRUE 2024-12-18T03:40:46.1117582Z Alloc Granule: 4KB 2024-12-18T03:40:46.1118239Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1118901Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1119553Z Accessible by all: FALSE 2024-12-18T03:40:46.1120113Z Pool 3 2024-12-18T03:40:46.1120618Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1121216Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1121817Z Allocatable: TRUE 2024-12-18T03:40:46.1122448Z Alloc Granule: 4KB 2024-12-18T03:40:46.1123120Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1123787Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1124447Z Accessible by all: FALSE 2024-12-18T03:40:46.1125004Z Pool 4 2024-12-18T03:40:46.1125497Z Segment: GROUP 2024-12-18T03:40:46.1126086Z Size: 64(0x40) KB 2024-12-18T03:40:46.1126689Z Allocatable: FALSE 2024-12-18T03:40:46.1127321Z Alloc Granule: 0KB 2024-12-18T03:40:46.1127980Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1128650Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1129306Z Accessible by all: FALSE 2024-12-18T03:40:46.1129888Z ISA Info: 2024-12-18T03:40:46.1130312Z ISA 1 2024-12-18T03:40:46.1130848Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1131892Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1132563Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1133235Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1133913Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1134537Z Fast f16: TRUE 2024-12-18T03:40:46.1135156Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1135749Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1136268Z x 1024(0x400) 2024-12-18T03:40:46.1136797Z y 1024(0x400) 2024-12-18T03:40:46.1137323Z z 1024(0x400) 2024-12-18T03:40:46.1137896Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1138479Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1138963Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1139501Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1140031Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1140628Z FBarrier Max Size: 32 2024-12-18T03:40:46.1141189Z ******* 2024-12-18T03:40:46.1141594Z Agent 7 2024-12-18T03:40:46.1141984Z ******* 2024-12-18T03:40:46.1142432Z Name: gfx90a 2024-12-18T03:40:46.1143022Z Uuid: GPU-ef2ed32bc2e35058 2024-12-18T03:40:46.1143905Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1144560Z Vendor Name: AMD 2024-12-18T03:40:46.1145196Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1145812Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1146441Z Float Round Mode: NEAR 2024-12-18T03:40:46.1147077Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1147768Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1148493Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1149188Z Queue Type: MULTI 2024-12-18T03:40:46.1149766Z Node: 6 2024-12-18T03:40:46.1150366Z Device Type: GPU 2024-12-18T03:40:46.1150922Z Cache Info: 2024-12-18T03:40:46.1151393Z L1: 16(0x10) KB 2024-12-18T03:40:46.1151931Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1152713Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1153328Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1154008Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1154751Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1155456Z BDFID: 44544 2024-12-18T03:40:46.1156140Z Internal Node ID: 6 2024-12-18T03:40:46.1156764Z Compute Unit: 104 2024-12-18T03:40:46.1157375Z SIMDs per CU: 4 2024-12-18T03:40:46.1157996Z Shader Engines: 8 2024-12-18T03:40:46.1158653Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1159313Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1160292Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1160885Z Memory Properties: 2024-12-18T03:40:46.1161355Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1161933Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1162576Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1163215Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1163796Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1164290Z x 1024(0x400) 2024-12-18T03:40:46.1164816Z y 1024(0x400) 2024-12-18T03:40:46.1165342Z z 1024(0x400) 2024-12-18T03:40:46.1165945Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1166591Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1167237Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1167802Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1168282Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1168839Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1169375Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1169983Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1170682Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1171348Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1171988Z IOMMU Support:: None 2024-12-18T03:40:46.1172544Z Pool Info: 2024-12-18T03:40:46.1173273Z Pool 1 2024-12-18T03:40:46.1173866Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1174635Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1175344Z Allocatable: TRUE 2024-12-18T03:40:46.1175994Z Alloc Granule: 4KB 2024-12-18T03:40:46.1176681Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1177365Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1178028Z Accessible by all: FALSE 2024-12-18T03:40:46.1178597Z Pool 2 2024-12-18T03:40:46.1179126Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1179748Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1180369Z Allocatable: TRUE 2024-12-18T03:40:46.1181009Z Alloc Granule: 4KB 2024-12-18T03:40:46.1181685Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1182358Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1183023Z Accessible by all: FALSE 2024-12-18T03:40:46.1183604Z Pool 3 2024-12-18T03:40:46.1184121Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1184737Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1185355Z Allocatable: TRUE 2024-12-18T03:40:46.1186005Z Alloc Granule: 4KB 2024-12-18T03:40:46.1186678Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1187357Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1188015Z Accessible by all: FALSE 2024-12-18T03:40:46.1188836Z Pool 4 2024-12-18T03:40:46.1189328Z Segment: GROUP 2024-12-18T03:40:46.1189907Z Size: 64(0x40) KB 2024-12-18T03:40:46.1190511Z Allocatable: FALSE 2024-12-18T03:40:46.1191146Z Alloc Granule: 0KB 2024-12-18T03:40:46.1191807Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1192592Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1193251Z Accessible by all: FALSE 2024-12-18T03:40:46.1193820Z ISA Info: 2024-12-18T03:40:46.1194235Z ISA 1 2024-12-18T03:40:46.1194781Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1195491Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1196169Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1196832Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1197507Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1198137Z Fast f16: TRUE 2024-12-18T03:40:46.1198769Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1199376Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1199911Z x 1024(0x400) 2024-12-18T03:40:46.1200466Z y 1024(0x400) 2024-12-18T03:40:46.1201000Z z 1024(0x400) 2024-12-18T03:40:46.1201873Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1202459Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1202963Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1203519Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1204065Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1204665Z FBarrier Max Size: 32 2024-12-18T03:40:46.1205225Z ******* 2024-12-18T03:40:46.1205628Z Agent 8 2024-12-18T03:40:46.1206027Z ******* 2024-12-18T03:40:46.1206484Z Name: gfx90a 2024-12-18T03:40:46.1207091Z Uuid: GPU-f363aff83c4a8ce4 2024-12-18T03:40:46.1207734Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1208390Z Vendor Name: AMD 2024-12-18T03:40:46.1209015Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1209652Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1210284Z Float Round Mode: NEAR 2024-12-18T03:40:46.1210922Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1211554Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1212174Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1212784Z Queue Type: MULTI 2024-12-18T03:40:46.1213358Z Node: 7 2024-12-18T03:40:46.1213936Z Device Type: GPU 2024-12-18T03:40:46.1214478Z Cache Info: 2024-12-18T03:40:46.1214935Z L1: 16(0x10) KB 2024-12-18T03:40:46.1215485Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1216045Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1216949Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1217579Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1218207Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1218798Z BDFID: 45824 2024-12-18T03:40:46.1219395Z Internal Node ID: 7 2024-12-18T03:40:46.1220019Z Compute Unit: 104 2024-12-18T03:40:46.1220632Z SIMDs per CU: 4 2024-12-18T03:40:46.1221255Z Shader Engines: 8 2024-12-18T03:40:46.1221899Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1222570Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1223244Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1223851Z Memory Properties: 2024-12-18T03:40:46.1224323Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1224916Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1225559Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1226211Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1226801Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1227313Z x 1024(0x400) 2024-12-18T03:40:46.1227841Z y 1024(0x400) 2024-12-18T03:40:46.1228361Z z 1024(0x400) 2024-12-18T03:40:46.1228938Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1229857Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1230504Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1231083Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1231556Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1232092Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1232728Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1233337Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1234069Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1234844Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1235603Z IOMMU Support:: None 2024-12-18T03:40:46.1236236Z Pool Info: 2024-12-18T03:40:46.1236658Z Pool 1 2024-12-18T03:40:46.1237199Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1237834Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1238479Z Allocatable: TRUE 2024-12-18T03:40:46.1239126Z Alloc Granule: 4KB 2024-12-18T03:40:46.1239793Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1240470Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1241125Z Accessible by all: FALSE 2024-12-18T03:40:46.1241691Z Pool 2 2024-12-18T03:40:46.1242212Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1242827Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1243425Z Allocatable: TRUE 2024-12-18T03:40:46.1244061Z Alloc Granule: 4KB 2024-12-18T03:40:46.1244726Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1245706Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1246354Z Accessible by all: FALSE 2024-12-18T03:40:46.1246913Z Pool 3 2024-12-18T03:40:46.1247419Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1248090Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1248801Z Allocatable: TRUE 2024-12-18T03:40:46.1249512Z Alloc Granule: 4KB 2024-12-18T03:40:46.1250173Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1250839Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1251497Z Accessible by all: FALSE 2024-12-18T03:40:46.1252062Z Pool 4 2024-12-18T03:40:46.1252562Z Segment: GROUP 2024-12-18T03:40:46.1253144Z Size: 64(0x40) KB 2024-12-18T03:40:46.1253739Z Allocatable: FALSE 2024-12-18T03:40:46.1254376Z Alloc Granule: 0KB 2024-12-18T03:40:46.1255043Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1255716Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1256369Z Accessible by all: FALSE 2024-12-18T03:40:46.1256929Z ISA Info: 2024-12-18T03:40:46.1257336Z ISA 1 2024-12-18T03:40:46.1257872Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1258849Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1259525Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1260195Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1260873Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1261504Z Fast f16: TRUE 2024-12-18T03:40:46.1262125Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1262720Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1263242Z x 1024(0x400) 2024-12-18T03:40:46.1263786Z y 1024(0x400) 2024-12-18T03:40:46.1264311Z z 1024(0x400) 2024-12-18T03:40:46.1264889Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1265474Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1265954Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1266504Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1267043Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1267640Z FBarrier Max Size: 32 2024-12-18T03:40:46.1268191Z ******* 2024-12-18T03:40:46.1268595Z Agent 9 2024-12-18T03:40:46.1268990Z ******* 2024-12-18T03:40:46.1269438Z Name: gfx90a 2024-12-18T03:40:46.1270035Z Uuid: GPU-1ece649c612e95a2 2024-12-18T03:40:46.1270668Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1271313Z Vendor Name: AMD 2024-12-18T03:40:46.1271943Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1272786Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1273781Z Float Round Mode: NEAR 2024-12-18T03:40:46.1274427Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1275055Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1275670Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1276287Z Queue Type: MULTI 2024-12-18T03:40:46.1276861Z Node: 8 2024-12-18T03:40:46.1277445Z Device Type: GPU 2024-12-18T03:40:46.1277985Z Cache Info: 2024-12-18T03:40:46.1278437Z L1: 16(0x10) KB 2024-12-18T03:40:46.1278979Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1279544Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1280144Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1280788Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1281420Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1282019Z BDFID: 36352 2024-12-18T03:40:46.1282616Z Internal Node ID: 8 2024-12-18T03:40:46.1283228Z Compute Unit: 104 2024-12-18T03:40:46.1283829Z SIMDs per CU: 4 2024-12-18T03:40:46.1284442Z Shader Engines: 8 2024-12-18T03:40:46.1285080Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1285733Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1286689Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1287301Z Memory Properties: 2024-12-18T03:40:46.1287780Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1288363Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1289012Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1289668Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1290258Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1290763Z x 1024(0x400) 2024-12-18T03:40:46.1291295Z y 1024(0x400) 2024-12-18T03:40:46.1291821Z z 1024(0x400) 2024-12-18T03:40:46.1292394Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1293028Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1293663Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1294228Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1294706Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1295237Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1295767Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1296373Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1297059Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1297724Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1298357Z IOMMU Support:: None 2024-12-18T03:40:46.1298902Z Pool Info: 2024-12-18T03:40:46.1299316Z Pool 1 2024-12-18T03:40:46.1299838Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1300475Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1301099Z Allocatable: TRUE 2024-12-18T03:40:46.1302013Z Alloc Granule: 4KB 2024-12-18T03:40:46.1302691Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1303381Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1304042Z Accessible by all: FALSE 2024-12-18T03:40:46.1304612Z Pool 2 2024-12-18T03:40:46.1305133Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1305760Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1306369Z Allocatable: TRUE 2024-12-18T03:40:46.1307005Z Alloc Granule: 4KB 2024-12-18T03:40:46.1307677Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1308365Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1309035Z Accessible by all: FALSE 2024-12-18T03:40:46.1309601Z Pool 3 2024-12-18T03:40:46.1310115Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1310717Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1311327Z Allocatable: TRUE 2024-12-18T03:40:46.1311969Z Alloc Granule: 4KB 2024-12-18T03:40:46.1312724Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1313393Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1314042Z Accessible by all: FALSE 2024-12-18T03:40:46.1314897Z Pool 4 2024-12-18T03:40:46.1315412Z Segment: GROUP 2024-12-18T03:40:46.1316017Z Size: 64(0x40) KB 2024-12-18T03:40:46.1316620Z Allocatable: FALSE 2024-12-18T03:40:46.1317257Z Alloc Granule: 0KB 2024-12-18T03:40:46.1317920Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1318590Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1319239Z Accessible by all: FALSE 2024-12-18T03:40:46.1319813Z ISA Info: 2024-12-18T03:40:46.1320227Z ISA 1 2024-12-18T03:40:46.1320758Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1321452Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1322130Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1322793Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1323474Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1324100Z Fast f16: TRUE 2024-12-18T03:40:46.1324721Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1325330Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1325857Z x 1024(0x400) 2024-12-18T03:40:46.1326394Z y 1024(0x400) 2024-12-18T03:40:46.1326919Z z 1024(0x400) 2024-12-18T03:40:46.1327499Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1328078Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1328577Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1329116Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1329962Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1330563Z FBarrier Max Size: 32 2024-12-18T03:40:46.1331128Z ******* 2024-12-18T03:40:46.1331531Z Agent 10 2024-12-18T03:40:46.1331932Z ******* 2024-12-18T03:40:46.1332387Z Name: gfx90a 2024-12-18T03:40:46.1332978Z Uuid: GPU-d3bee6119563da6a 2024-12-18T03:40:46.1333619Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.1334269Z Vendor Name: AMD 2024-12-18T03:40:46.1334890Z Feature: KERNEL_DISPATCH 2024-12-18T03:40:46.1335532Z Profile: BASE_PROFILE 2024-12-18T03:40:46.1336165Z Float Round Mode: NEAR 2024-12-18T03:40:46.1336819Z Max Queue Number: 128(0x80) 2024-12-18T03:40:46.1337444Z Queue Min Size: 64(0x40) 2024-12-18T03:40:46.1338064Z Queue Max Size: 131072(0x20000) 2024-12-18T03:40:46.1338675Z Queue Type: MULTI 2024-12-18T03:40:46.1339255Z Node: 9 2024-12-18T03:40:46.1339842Z Device Type: GPU 2024-12-18T03:40:46.1340390Z Cache Info: 2024-12-18T03:40:46.1340849Z L1: 16(0x10) KB 2024-12-18T03:40:46.1341385Z L2: 8192(0x2000) KB 2024-12-18T03:40:46.1342220Z Chip ID: 29708(0x740c) 2024-12-18T03:40:46.1342827Z ASIC Revision: 1(0x1) 2024-12-18T03:40:46.1343479Z Cacheline Size: 64(0x40) 2024-12-18T03:40:46.1344115Z Max Clock Freq. (MHz): 1700 2024-12-18T03:40:46.1344706Z BDFID: 37632 2024-12-18T03:40:46.1345301Z Internal Node ID: 9 2024-12-18T03:40:46.1345924Z Compute Unit: 104 2024-12-18T03:40:46.1346531Z SIMDs per CU: 4 2024-12-18T03:40:46.1347138Z Shader Engines: 8 2024-12-18T03:40:46.1347780Z Shader Arrs. per Eng.: 1 2024-12-18T03:40:46.1348435Z WatchPts on Addr. Ranges:4 2024-12-18T03:40:46.1349118Z Coherent Host Access: FALSE 2024-12-18T03:40:46.1349706Z Memory Properties: 2024-12-18T03:40:46.1350185Z Features: KERNEL_DISPATCH 2024-12-18T03:40:46.1350787Z Fast F16 Operation: TRUE 2024-12-18T03:40:46.1351436Z Wavefront Size: 64(0x40) 2024-12-18T03:40:46.1352079Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1352921Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1353455Z x 1024(0x400) 2024-12-18T03:40:46.1353983Z y 1024(0x400) 2024-12-18T03:40:46.1354499Z z 1024(0x400) 2024-12-18T03:40:46.1355077Z Max Waves Per CU: 32(0x20) 2024-12-18T03:40:46.1355723Z Max Work-item Per CU: 2048(0x800) 2024-12-18T03:40:46.1356373Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1356947Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1357421Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1358352Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1358977Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1359697Z Max fbarriers/Workgrp: 32 2024-12-18T03:40:46.1360513Z Packet Processor uCode:: 83 2024-12-18T03:40:46.1361290Z SDMA engine uCode:: 8 2024-12-18T03:40:46.1362065Z IOMMU Support:: None 2024-12-18T03:40:46.1362723Z Pool Info: 2024-12-18T03:40:46.1363224Z Pool 1 2024-12-18T03:40:46.1363851Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2024-12-18T03:40:46.1364247Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1364593Z Allocatable: TRUE 2024-12-18T03:40:46.1364914Z Alloc Granule: 4KB 2024-12-18T03:40:46.1365237Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1365561Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1365875Z Accessible by all: FALSE 2024-12-18T03:40:46.1366148Z Pool 2 2024-12-18T03:40:46.1366399Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2024-12-18T03:40:46.1366697Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1366986Z Allocatable: TRUE 2024-12-18T03:40:46.1367289Z Alloc Granule: 4KB 2024-12-18T03:40:46.1367736Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1368067Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1368382Z Accessible by all: FALSE 2024-12-18T03:40:46.1368652Z Pool 3 2024-12-18T03:40:46.1368897Z Segment: GLOBAL; FLAGS: FINE GRAINED 2024-12-18T03:40:46.1369185Z Size: 67092480(0x3ffc000) KB 2024-12-18T03:40:46.1369475Z Allocatable: TRUE 2024-12-18T03:40:46.1369773Z Alloc Granule: 4KB 2024-12-18T03:40:46.1370092Z Alloc Recommended Granule:2048KB 2024-12-18T03:40:46.1370412Z Alloc Alignment: 4KB 2024-12-18T03:40:46.1370724Z Accessible by all: FALSE 2024-12-18T03:40:46.1370995Z Pool 4 2024-12-18T03:40:46.1371238Z Segment: GROUP 2024-12-18T03:40:46.1371525Z Size: 64(0x40) KB 2024-12-18T03:40:46.1371820Z Allocatable: FALSE 2024-12-18T03:40:46.1372124Z Alloc Granule: 0KB 2024-12-18T03:40:46.1372443Z Alloc Recommended Granule:0KB 2024-12-18T03:40:46.1372762Z Alloc Alignment: 0KB 2024-12-18T03:40:46.1373078Z Accessible by all: FALSE 2024-12-18T03:40:46.1373350Z ISA Info: 2024-12-18T03:40:46.1373550Z ISA 1 2024-12-18T03:40:46.1373810Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2024-12-18T03:40:46.1374143Z Machine Models: HSA_MACHINE_MODEL_LARGE 2024-12-18T03:40:46.1374467Z Profiles: HSA_PROFILE_BASE 2024-12-18T03:40:46.1374780Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1375228Z Default Rounding Mode: NEAR 2024-12-18T03:40:46.1375529Z Fast f16: TRUE 2024-12-18T03:40:46.1375829Z Workgroup Max Size: 1024(0x400) 2024-12-18T03:40:46.1376124Z Workgroup Max Size per Dimension: 2024-12-18T03:40:46.1376380Z x 1024(0x400) 2024-12-18T03:40:46.1376645Z y 1024(0x400) 2024-12-18T03:40:46.1376904Z z 1024(0x400) 2024-12-18T03:40:46.1377189Z Grid Max Size: 4294967295(0xffffffff) 2024-12-18T03:40:46.1377470Z Grid Max Size per Dimension: 2024-12-18T03:40:46.1377719Z x 4294967295(0xffffffff) 2024-12-18T03:40:46.1377984Z y 4294967295(0xffffffff) 2024-12-18T03:40:46.1378254Z z 4294967295(0xffffffff) 2024-12-18T03:40:46.1378543Z FBarrier Max Size: 32 2024-12-18T03:40:46.1378814Z *** Done *** 2024-12-18T03:40:46.1379020Z + rocminfo 2024-12-18T03:40:46.1379207Z + grep -E 'Name:.*\sgfx|Marketing' 2024-12-18T03:40:46.2376837Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.2377644Z Marketing Name: AMD EPYC 7713 64-Core Processor 2024-12-18T03:40:46.2378249Z Name: gfx90a 2024-12-18T03:40:46.2378857Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2379423Z Name: gfx90a 2024-12-18T03:40:46.2379998Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2380947Z Name: gfx90a 2024-12-18T03:40:46.2381542Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2382127Z Name: gfx90a 2024-12-18T03:40:46.2382839Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2383562Z Name: gfx90a 2024-12-18T03:40:46.2384287Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2385005Z Name: gfx90a 2024-12-18T03:40:46.2385729Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2386456Z Name: gfx90a 2024-12-18T03:40:46.2387179Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2387899Z Name: gfx90a 2024-12-18T03:40:46.2388624Z Marketing Name: AMD Instinct MI250X/MI250 2024-12-18T03:40:46.2576666Z + [[ linux-focal-rocm6.2-py3.10 == *xpu* ]] 2024-12-18T03:40:46.2577395Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-12-18T03:40:46.2578065Z + pip_install --user ninja==1.10.2 2024-12-18T03:40:46.2578844Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T03:40:46.2579824Z + python3 -m pip install --progress-bar off --user ninja==1.10.2 2024-12-18T03:40:46.8254444Z Collecting ninja==1.10.2 2024-12-18T03:40:46.9257082Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-12-18T03:40:46.9593455Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-12-18T03:40:47.2986454Z Installing collected packages: ninja 2024-12-18T03:40:47.3053388Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-12-18T03:40:47.3055273Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-12-18T03:40:47.3073500Z Successfully installed ninja-1.10.2 2024-12-18T03:40:47.3605678Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:40:47.3609380Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:40:47.3611515Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-12-18T03:40:47.3612173Z + install_tlparse 2024-12-18T03:40:47.3612701Z + pip_install --user tlparse==0.3.25 2024-12-18T03:40:47.3613498Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2024-12-18T03:40:47.3614462Z + python3 -m pip install --progress-bar off --user tlparse==0.3.25 2024-12-18T03:40:47.8932093Z Collecting tlparse==0.3.25 2024-12-18T03:40:47.9795674Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-12-18T03:40:48.0134701Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-12-18T03:40:48.4449415Z Installing collected packages: tlparse 2024-12-18T03:40:48.4742126Z Successfully installed tlparse-0.3.25 2024-12-18T03:40:48.5329606Z ++ python -m site --user-base 2024-12-18T03:40:48.5836259Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-12-18T03:40:48.5838941Z + [[ linux-focal-rocm6.2-py3.10 == *asan* ]] 2024-12-18T03:40:48.5839623Z + [[ linux-focal-rocm6.2-py3.10 == *-debug* ]] 2024-12-18T03:40:48.5841200Z + [[ linux-focal-rocm6.2-py3.10 != *-bazel-* ]] 2024-12-18T03:40:48.5842183Z + echo 'We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass' 2024-12-18T03:40:48.5843387Z We are not in debug mode: linux-focal-rocm6.2-py3.10. Expect the assertion to pass 2024-12-18T03:40:48.5849193Z + cd test 2024-12-18T03:40:48.5850042Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-12-18T03:40:51.6662345Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-12-18T03:40:51.6663205Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-12-18T03:40:51.6672519Z + DYNAMO_BENCHMARK_FLAGS=() 2024-12-18T03:40:51.6673263Z + [[ distributed == *pr_time_benchmarks* ]] 2024-12-18T03:40:51.6673955Z + [[ distributed == *dynamo_eager* ]] 2024-12-18T03:40:51.6674577Z + [[ distributed == *aot_eager* ]] 2024-12-18T03:40:51.6675159Z + [[ distributed == *aot_inductor* ]] 2024-12-18T03:40:51.6675743Z + [[ distributed == *inductor* ]] 2024-12-18T03:40:51.6677750Z + [[ distributed == *dynamic* ]] 2024-12-18T03:40:51.6678310Z + [[ distributed == *cpu* ]] 2024-12-18T03:40:51.6678897Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-12-18T03:40:51.6737415Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-12-18T03:40:51.6738257Z + [[ linux-focal-rocm6.2-py3.10 == *-bazel-* ]] 2024-12-18T03:40:51.6747559Z + cd test 2024-12-18T03:40:51.6748267Z + python -c 'import torch; print(torch.__config__.show())' 2024-12-18T03:40:54.2039839Z PyTorch built with: 2024-12-18T03:40:54.2040212Z - GCC 9.4 2024-12-18T03:40:54.2040452Z - C++ Version: 201703 2024-12-18T03:40:54.2041013Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T03:40:54.2041722Z - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T03:40:54.2042148Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T03:40:54.2042482Z - LAPACK is enabled (usually provided by MKL) 2024-12-18T03:40:54.2042858Z - NNPACK is enabled 2024-12-18T03:40:54.2043114Z - CPU capability usage: AVX2 2024-12-18T03:40:54.2043390Z - HIP Runtime 6.2.41134 2024-12-18T03:40:54.2044245Z - MIOpen 3.2.0 2024-12-18T03:40:54.2044472Z - Magma 2.7.2 2024-12-18T03:40:54.2048679Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=0cdf8b1d09254cfda66191d1bd01e3041c3c76f7, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.6.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, 2024-12-18T03:40:54.2053128Z 2024-12-18T03:40:54.7571216Z + cd test 2024-12-18T03:40:54.7571941Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-12-18T03:40:56.0468673Z ATen/Parallel: 2024-12-18T03:40:56.0469086Z at::get_num_threads() : 128 2024-12-18T03:40:56.0469424Z at::get_num_interop_threads() : 128 2024-12-18T03:40:56.0469743Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-12-18T03:40:56.0470031Z omp_get_max_threads() : 128 2024-12-18T03:40:56.0473290Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-12-18T03:40:56.0473886Z mkl_get_max_threads() : 128 2024-12-18T03:40:56.0474268Z Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af) 2024-12-18T03:40:56.0474746Z std::thread::hardware_concurrency() : 128 2024-12-18T03:40:56.0475057Z Environment variables: 2024-12-18T03:40:56.0475311Z OMP_NUM_THREADS : [not set] 2024-12-18T03:40:56.0475574Z MKL_NUM_THREADS : [not set] 2024-12-18T03:40:56.0475836Z ATen parallel backend: OpenMP 2024-12-18T03:40:56.0476009Z 2024-12-18T03:40:57.8288291Z + [[ distributed == *numpy_2* ]] 2024-12-18T03:40:57.8289102Z + [[ linux-focal-rocm6.2-py3.10 == *aarch64* ]] 2024-12-18T03:40:57.8289779Z + [[ distributed == *backward* ]] 2024-12-18T03:40:57.8290328Z + [[ distributed == *xla* ]] 2024-12-18T03:40:57.8290862Z + [[ distributed == *executorch* ]] 2024-12-18T03:40:57.8291442Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2024-12-18T03:40:57.8292080Z + [[ linux-focal-rocm6.2-py3.10 == *libtorch* ]] 2024-12-18T03:40:57.8292790Z + [[ distributed == distributed ]] 2024-12-18T03:40:57.8293329Z + test_distributed 2024-12-18T03:40:57.8293841Z + echo 'Testing distributed python tests' 2024-12-18T03:40:57.8294497Z Testing distributed python tests 2024-12-18T03:40:57.8295241Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose 2024-12-18T03:40:57.9466208Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T03:40:57.9467786Z import pkg_resources 2024-12-18T03:41:02.4689110Z Excluding distributed/rpc/test_faulty_agent on ROCm 2024-12-18T03:41:02.4690130Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2024-12-18T03:41:02.4690924Z Excluding distributed/rpc/test_share_memory on ROCm 2024-12-18T03:41:02.4691732Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2024-12-18T03:41:02.4692671Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding on ROCm 2024-12-18T03:41:02.4693752Z Excluding distributed/_shard/sharded_tensor/ops/test_embedding_bag on ROCm 2024-12-18T03:41:02.4694770Z Excluding distributed/_shard/sharded_tensor/ops/test_binary_cmp on ROCm 2024-12-18T03:41:02.4699605Z Excluding distributed/_shard/sharded_tensor/ops/test_init on ROCm 2024-12-18T03:41:02.4700584Z Excluding distributed/_shard/sharded_optim/test_sharded_optim on ROCm 2024-12-18T03:41:02.4701461Z Excluding distributed/_tensor/test_attention on ROCm 2024-12-18T03:41:03.4534156Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T03:41:03.7280352Z Ignoring disabled issues: [''] 2024-12-18T03:41:03.7581591Z Found test times from artifacts 2024-12-18T03:41:03.8175165Z Found test times from artifacts 2024-12-18T03:41:03.8189022Z Running all tests 2024-12-18T03:41:03.8272037Z Running parallel tests on 8 processes 2024-12-18T03:41:03.8273001Z Name: tests to run (est. time: 142.47min) 2024-12-18T03:41:03.8273659Z Serial tests (54): 2024-12-18T03:41:03.8274202Z distributed/test_dynamo_distributed 1/1 2024-12-18T03:41:03.8274892Z distributed/_tensor/test_dtensor_ops 1/1 2024-12-18T03:41:03.8275555Z distributed/_tensor/test_xla_integration 1/1 2024-12-18T03:41:03.8276225Z distributed/_tensor/test_convolution_ops 1/1 2024-12-18T03:41:03.8276851Z distributed/_tensor/test_math_ops 1/1 2024-12-18T03:41:03.8277622Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 2024-12-18T03:41:03.8278401Z distributed/_tensor/test_view_ops 1/1 2024-12-18T03:41:03.8279122Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 2024-12-18T03:41:03.8279845Z distributed/fsdp/test_wrap 1/1 2024-12-18T03:41:03.8280400Z distributed/test_c10d_ops_nccl 1/1 2024-12-18T03:41:03.8281014Z distributed/_tensor/debug/test_op_coverage 1/1 2024-12-18T03:41:03.8281674Z distributed/_tools/test_mod_tracker 1/1 2024-12-18T03:41:03.8283104Z distributed/pipelining/test_schedule_multiproc 1/1 2024-12-18T03:41:03.8283927Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 2024-12-18T03:41:03.8284675Z distributed/_tools/test_sac_estimator 1/1 2024-12-18T03:41:03.8285293Z distributed/_tools/test_sac_ilp 1/1 2024-12-18T03:41:03.8285915Z distributed/test_c10d_functional_native 1/1 2024-12-18T03:41:03.8286539Z distributed/test_backends 1/1 2024-12-18T03:41:03.8287100Z distributed/test_symmetric_memory 1/1 2024-12-18T03:41:03.8287712Z distributed/test_distributed_spawn 3/12 2024-12-18T03:41:03.8288357Z distributed/test_distributed_spawn 6/12 2024-12-18T03:41:03.8288960Z distributed/test_distributed_spawn 9/12 2024-12-18T03:41:03.8289572Z distributed/test_distributed_spawn 12/12 2024-12-18T03:41:03.8290325Z distributed/algorithms/quantization/test_quantization 1/1 2024-12-18T03:41:03.8291140Z distributed/tensor/parallel/test_tp_examples 1/1 2024-12-18T03:41:03.8291810Z distributed/test_collective_utils 1/1 2024-12-18T03:41:03.8292410Z distributed/test_c10d_logger 1/1 2024-12-18T03:41:03.8293003Z distributed/fsdp/test_fsdp_state_dict 1/3 2024-12-18T03:41:03.8293630Z distributed/_tensor/test_optimizers 1/1 2024-12-18T03:41:03.8294220Z distributed/test_c10d_nccl 3/3 2024-12-18T03:41:03.8294794Z distributed/pipelining/test_schedule 1/1 2024-12-18T03:41:03.8295439Z distributed/_tensor/test_op_strategy 1/1 2024-12-18T03:41:03.8296094Z distributed/checkpoint/test_fsdp_model_state 1/1 2024-12-18T03:41:03.8296849Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2024-12-18T03:41:03.8297600Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2024-12-18T03:41:03.8298365Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 2024-12-18T03:41:03.8299130Z distributed/fsdp/test_fsdp_freezing_weights 1/1 2024-12-18T03:41:03.8299797Z distributed/fsdp/test_fsdp_grad_acc 1/1 2024-12-18T03:41:03.8300451Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 2024-12-18T03:41:03.8301283Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 2024-12-18T03:41:03.8302189Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 2024-12-18T03:41:03.8303333Z distributed/algorithms/test_join 1/1 2024-12-18T03:41:03.8303942Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2024-12-18T03:41:03.8304542Z distributed/fsdp/test_fsdp_core 2/3 2024-12-18T03:41:03.8305124Z distributed/fsdp/test_fsdp_meta 1/1 2024-12-18T03:41:03.8305709Z distributed/fsdp/test_fsdp_fine_tune 1/1 2024-12-18T03:41:03.8306402Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 2024-12-18T03:41:03.8307139Z distributed/fsdp/test_fsdp_unshard_params 1/1 2024-12-18T03:41:03.8307817Z distributed/fsdp/test_fsdp_mixed_precision 1/2 2024-12-18T03:41:03.8308449Z distributed/test_c10d_gloo 1/2 2024-12-18T03:41:03.8308988Z distributed/test_store 1/1 2024-12-18T03:41:03.8309562Z distributed/elastic/timer/local_timer_test 1/1 2024-12-18T03:41:03.8310235Z distributed/elastic/utils/distributed_test 1/1 2024-12-18T03:41:03.8310878Z distributed/elastic/timer/api_test 1/1 2024-12-18T03:41:03.8311446Z Parallel tests (0): 2024-12-18T03:41:03.8311941Z Name: excluded (est. time: 0.0min) 2024-12-18T03:41:03.8312573Z Serial tests (0): 2024-12-18T03:41:03.8313012Z Parallel tests (0): 2024-12-18T03:41:03.8422252Z Running distributed/test_dynamo_distributed 1/1 ... [2024-12-18 03:41:03.841172] 2024-12-18T03:41:03.8423286Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:41:03.8425314Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:41:03.841513] 2024-12-18T03:46:51.2490493Z 2024-12-18T03:46:51.2492125Z distributed/test_dynamo_distributed 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.1_16ad1c336589fc2c_.log 2024-12-18T03:46:51.2517041Z Running 58 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2024-12-18T03:46:51.2550591Z 2024-12-18T03:46:51.2551068Z Running distributed/_tensor/test_dtensor_ops 1/1 ... [2024-12-18 03:46:51.249305] 2024-12-18T03:46:51.2551928Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:46:51.2553974Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:46:51.249651] 2024-12-18T03:46:57.5968076Z 2024-12-18T03:46:57.5969614Z distributed/_tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_dtensor_ops_1.1_3b635f2a4892396c_.log 2024-12-18T03:46:57.5971209Z Running 0 items in this shard: 2024-12-18T03:46:57.5971585Z 2024-12-18T03:46:57.5976516Z Running distributed/_tensor/test_xla_integration 1/1 ... [2024-12-18 03:46:57.597009] 2024-12-18T03:46:57.5977557Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:46:57.5983163Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:46:57.597605] 2024-12-18T03:47:03.0775024Z 2024-12-18T03:47:03.0776836Z distributed/_tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_xla_integration_1.1_bd008a99bf4445e2_.log 2024-12-18T03:47:03.0784170Z Running 3 items in this shard: test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/_tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d 2024-12-18T03:47:03.0787024Z 2024-12-18T03:47:03.0787525Z Running distributed/_tensor/test_convolution_ops 1/1 ... [2024-12-18 03:47:03.077684] 2024-12-18T03:47:03.0788427Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:47:03.0791242Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:47:03.078210] 2024-12-18T03:48:31.6975831Z 2024-12-18T03:48:31.6977597Z distributed/_tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_convolution_ops_1.1_8e6d133ba3c860ab_.log 2024-12-18T03:48:31.6980592Z Running 2 items in this shard: test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/_tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution 2024-12-18T03:48:31.6982463Z 2024-12-18T03:48:31.6982909Z Running distributed/_tensor/test_math_ops 1/1 ... [2024-12-18 03:48:31.696969] 2024-12-18T03:48:31.6983781Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:48:31.6985762Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:48:31.697557] 2024-12-18T03:50:24.3236214Z 2024-12-18T03:50:24.3241930Z distributed/_tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_math_ops_1.1_291cf4ca2553042a_.log 2024-12-18T03:50:24.3251656Z Running 14 items in this shard: test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/_tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2024-12-18T03:50:24.3260970Z 2024-12-18T03:50:24.3261673Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2024-12-18 03:50:24.323880] 2024-12-18T03:50:24.3262756Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:50:24.3264984Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:50:24.324482] 2024-12-18T03:50:51.7547139Z 2024-12-18T03:50:51.7549041Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_276e86975cdba120_.log 2024-12-18T03:50:51.7553268Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d 2024-12-18T03:50:51.7555260Z 2024-12-18T03:50:51.7555722Z Running distributed/_tensor/test_view_ops 1/1 ... [2024-12-18 03:50:51.754774] 2024-12-18T03:50:51.7556568Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:50:51.7560548Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:50:51.755383] 2024-12-18T03:51:36.6799215Z 2024-12-18T03:51:36.6801141Z distributed/_tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_view_ops_1.1_344f63a62ae481c7_.log 2024-12-18T03:51:36.6804721Z Running 4 items in this shard: test/distributed/_tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/_tensor/test_view_ops.py::TestViewOps::test_view_ops 2024-12-18T03:51:36.6807068Z 2024-12-18T03:51:36.6807733Z Running distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2024-12-18 03:51:36.680102] 2024-12-18T03:51:36.6808819Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:51:36.6812034Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_extensions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:51:36.680640] 2024-12-18T03:52:21.0522490Z 2024-12-18T03:52:21.0524714Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_32b518de7df08f51_.log 2024-12-18T03:52:21.0532234Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiProcess::test_all_gather_extensions_train_parity, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_hsdp_mesh, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_outer_size_stride, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_end_to_end, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_monkey_patch 2024-12-18T03:52:21.0538875Z 2024-12-18T03:52:21.0539301Z Running distributed/fsdp/test_wrap 1/1 ... [2024-12-18 03:52:21.052054] 2024-12-18T03:52:21.0540129Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:52:21.0542014Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:52:21.052589] 2024-12-18T03:56:25.7883302Z 2024-12-18T03:56:25.7884956Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_c9cd5db98dd31260_.log 2024-12-18T03:56:25.7928397Z Running 51 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params 2024-12-18T03:56:25.7969837Z 2024-12-18T03:56:25.7970387Z Running distributed/test_c10d_ops_nccl 1/1 ... [2024-12-18 03:56:25.788766] 2024-12-18T03:56:25.7971407Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:56:25.7973491Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:56:25.789335] 2024-12-18T03:56:39.7884953Z 2024-12-18T03:56:39.7886551Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_eddca391a0566416_.log 2024-12-18T03:56:39.7903779Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T03:56:39.7937046Z Running 25 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook 2024-12-18T03:56:39.7953550Z 2024-12-18T03:56:39.7954111Z Running distributed/_tensor/debug/test_op_coverage 1/1 ... [2024-12-18 03:56:39.788714] 2024-12-18T03:56:39.7955070Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:56:39.7957504Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:56:39.789280] 2024-12-18T03:56:45.8197768Z 2024-12-18T03:56:45.8199628Z distributed/_tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.debug.test_op_coverage_1.1_d796a1d7ee86c8cf_.log 2024-12-18T03:56:45.8202127Z Running 1 items in this shard: test/distributed/_tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp 2024-12-18T03:56:45.8203173Z 2024-12-18T03:56:45.8203852Z Running distributed/_tools/test_mod_tracker 1/1 ... [2024-12-18 03:56:45.819881] 2024-12-18T03:56:45.8204786Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:56:45.8209350Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:56:45.820351] 2024-12-18T03:56:51.5502173Z 2024-12-18T03:56:51.5504350Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_217a28827b71c045_.log 2024-12-18T03:56:51.5508047Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks 2024-12-18T03:56:51.5510447Z 2024-12-18T03:56:51.5511829Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2024-12-18 03:56:51.550391] 2024-12-18T03:56:51.5513047Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T03:56:51.5515173Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 03:56:51.550909] 2024-12-18T04:07:52.0320272Z 2024-12-18T04:07:52.0322248Z PRINTING LOG FILE of distributed/pipelining/test_schedule_multiproc 1/1 (test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_e51da2f02d2e487f_.log) 2024-12-18T04:07:52.0325337Z Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-fb8ca572d9147395.xml 2024-12-18T04:07:52.0327135Z ============================= test session starts ============================== 2024-12-18T04:07:52.0328179Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0329157Z cachedir: .pytest_cache 2024-12-18T04:07:52.0330267Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0331675Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0332323Z configfile: pytest.ini 2024-12-18T04:07:52.0333611Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0334823Z collecting ... collected 27 items 2024-12-18T04:07:52.0335479Z stepcurrent: Cannot find last run test, not skipping 2024-12-18T04:07:52.0361534Z Running 27 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T04:07:52.0386810Z 2024-12-18T04:07:52.0388702Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0 Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-2d75251948b405ed.xml 2024-12-18T04:07:52.0391121Z ============================= test session starts ============================== 2024-12-18T04:07:52.0392123Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0393209Z cachedir: .pytest_cache 2024-12-18T04:07:52.0394320Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0395477Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0396038Z configfile: pytest.ini 2024-12-18T04:07:52.0397140Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0398770Z collecting ... collected 27 items 2024-12-18T04:07:52.0399473Z stepcurrent: skipping 0 already run items. 2024-12-18T04:07:52.0425695Z Running 27 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T04:07:52.0451653Z 2024-12-18T04:07:52.0452790Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2024-12-18T04:07:52.0454467Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2024-12-18T04:07:52.0455576Z PASSED [12.3380s] [ 3%] 2024-12-18T04:07:52.0456830Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False PASSED [12.5780s] [ 3%] 2024-12-18T04:07:52.0458884Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False PASSED [0.0962s] [ 7%] 2024-12-18T04:07:52.0460911Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True PASSED [0.2117s] [ 7%] 2024-12-18T04:07:52.0462926Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True PASSED [0.0453s] [ 11%] 2024-12-18T04:07:52.0464969Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False PASSED [0.1044s] [ 11%] 2024-12-18T04:07:52.0467219Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False PASSED [0.0386s] [ 14%] 2024-12-18T04:07:52.0469229Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True PASSED [0.0384s] [ 14%] 2024-12-18T04:07:52.0471220Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True PASSED [0.0413s] [ 18%] 2024-12-18T04:07:52.0473603Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False PASSED [0.0415s] [ 18%] 2024-12-18T04:07:52.0476277Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False PASSED [0.3394s] [ 22%]PASSED [0.3366s] [ 22%] 2024-12-18T04:07:52.0478847Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True 2024-12-18T04:07:52.0481021Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True PASSED [0.3231s] [ 25%]PASSED [0.3229s] [ 25%] 2024-12-18T04:07:52.0483204Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False 2024-12-18T04:07:52.0485588Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False PASSED [0.3108s] [ 29%] 2024-12-18T04:07:52.0488192Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True PASSED [0.3113s] [ 29%] 2024-12-18T04:07:52.0494433Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True PASSED [0.3278s] [ 33%]PASSED [0.3272s] [ 33%] 2024-12-18T04:07:52.0497070Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False 2024-12-18T04:07:52.0499288Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False PASSED [0.3277s] [ 37%]PASSED [0.3278s] [ 37%] 2024-12-18T04:07:52.0501476Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True 2024-12-18T04:07:52.0503688Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True PASSED [0.3349s] [ 40%]PASSED [0.3347s] [ 40%] 2024-12-18T04:07:52.0505742Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0 2024-12-18T04:07:52.0507680Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0_ModelClass0 PASSED [0.1032s] [ 44%]PASSED [0.1034s] [ 44%] 2024-12-18T04:07:52.0510083Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0 2024-12-18T04:07:52.0511996Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1_ModelClass0 PASSED [0.0982s] [ 48%]PASSED [0.0983s] [ 48%] 2024-12-18T04:07:52.0514089Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0 2024-12-18T04:07:52.0516024Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0 PASSED [0.0755s] [ 51%] 2024-12-18T04:07:52.0517780Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1 PASSED [0.0759s] [ 51%] 2024-12-18T04:07:52.0519621Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1 PASSED [0.0733s] [ 55%] 2024-12-18T04:07:52.0521358Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0 PASSED [0.0732s] [ 55%] 2024-12-18T04:07:52.0523336Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0 PASSED [0.1881s] [ 59%] 2024-12-18T04:07:52.0525312Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1 PASSED [0.1908s] [ 59%] 2024-12-18T04:07:52.0527077Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1 PASSED [0.1821s] [ 62%] 2024-12-18T04:07:52.0528965Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False PASSED [0.1810s] [ 62%] 2024-12-18T04:07:52.0531443Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False PASSED [0.0253s] [ 66%] 2024-12-18T04:07:52.0533557Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True PASSED [0.0284s] [ 66%] 2024-12-18T04:07:52.0535663Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True PASSED [0.0247s] [ 70%] 2024-12-18T04:07:52.0538130Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False PASSED [0.0244s] [ 70%] 2024-12-18T04:07:52.0540340Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False PASSED [0.0322s] [ 74%] 2024-12-18T04:07:52.0542435Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True PASSED [0.0332s] [ 74%] 2024-12-18T04:07:52.0544517Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True PASSED [0.0300s] [ 77%] 2024-12-18T04:07:52.0546607Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False PASSED [0.0296s] [ 77%] 2024-12-18T04:07:52.0548844Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_False PASSED [0.0261s] [ 81%]PASSED [0.0255s] [ 81%] 2024-12-18T04:07:52.0550950Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True 2024-12-18T04:07:52.0553334Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_non_symmetric_stage_ids_schedule_class2_use_new_runtime_True PASSED [0.0267s] [ 85%] 2024-12-18T04:07:52.0555815Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0 PASSED [0.0270s] [ 85%] 2024-12-18T04:07:52.0558063Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0 PASSED [0.3015s] [ 88%]PASSED [0.3028s] [ 88%] 2024-12-18T04:07:52.0560422Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0 2024-12-18T04:07:52.0562446Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass0 PASSED [0.2862s] [ 92%] 2024-12-18T04:07:52.0564739Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1 PASSED [0.3200s] [ 92%] 2024-12-18T04:07:52.0566729Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1 PASSED [0.3648s] [ 96%] 2024-12-18T04:07:52.0568579Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 ('RERUN', {'yellow': True}) [0.5290s] [ 96%] 2024-12-18T04:07:52.0571843Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1 [rank1]:E1218 03:57:18.577000 27329 site-packages/torch/distributed/pipelining/schedules.py:1357] [Rank 1] pipeline schedule ScheduleInterleavedZeroBubble caught the following exception at time_step 3 when running action 3F0 2024-12-18T04:07:52.0575167Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Rank 0 Rank 1 2024-12-18T04:07:52.0576600Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 00: 0F0 2024-12-18T04:07:52.0578027Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 01: 0F1 1F0 2024-12-18T04:07:52.0579424Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 02: 2F0 1F1 2024-12-18T04:07:52.0581249Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 03: 3F0 <-- ERROR HERE 2024-12-18T04:07:52.0582777Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 04: 2F1 3I0 2024-12-18T04:07:52.0584178Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 05: 2I0 3F1 2024-12-18T04:07:52.0585565Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 06: 2W0 3I1 2024-12-18T04:07:52.0586959Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 07: 0F2 3W0 2024-12-18T04:07:52.0588357Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 08: 2I1 1F2 2024-12-18T04:07:52.0589754Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 09: 2W1 1I0 2024-12-18T04:07:52.0591149Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 10: 0F3 3W1 2024-12-18T04:07:52.0592647Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 11: 0I0 1F3 2024-12-18T04:07:52.0594104Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 12: 0W0 1I1 2024-12-18T04:07:52.0595783Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 13: 2F2 1W0 2024-12-18T04:07:52.0597206Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 14: 0I1 3F2 2024-12-18T04:07:52.0598606Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 15: 0W1 3I2 2024-12-18T04:07:52.0600008Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 16: 2F3 1W1 2024-12-18T04:07:52.0601487Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 17: 2I2 3F3 2024-12-18T04:07:52.0603469Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 18: 2W2 3I3 2024-12-18T04:07:52.0605120Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 19: 0F4 3W2 2024-12-18T04:07:52.0606644Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 20: 2I3 1F4 2024-12-18T04:07:52.0608041Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 21: 2W3 1I2 2024-12-18T04:07:52.0609432Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 22: 0F5 3W3 2024-12-18T04:07:52.0610830Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 23: 0I2 1F5 2024-12-18T04:07:52.0612406Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 24: 0W2 1I3 2024-12-18T04:07:52.0614053Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 25: 2F4 1W2 2024-12-18T04:07:52.0615491Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 26: 0I3 3F4 2024-12-18T04:07:52.0616882Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 27: 0W3 3I4 2024-12-18T04:07:52.0618274Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 28: 2F5 1W3 2024-12-18T04:07:52.0619975Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 29: 2I4 3F5 2024-12-18T04:07:52.0621364Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 30: 2W4 3I5 2024-12-18T04:07:52.0622774Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 31: 0F6 3W4 2024-12-18T04:07:52.0624161Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 32: 2I5 1F6 2024-12-18T04:07:52.0625556Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 33: 2W5 1I4 2024-12-18T04:07:52.0626976Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 34: 0F7 3W5 2024-12-18T04:07:52.0628372Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 35: 0I4 1F7 2024-12-18T04:07:52.0629764Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 36: 0W4 1I5 2024-12-18T04:07:52.0631154Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 37: 2F6 1W4 2024-12-18T04:07:52.0632640Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 38: 0I5 3F6 2024-12-18T04:07:52.0634092Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 39: 0W5 3I6 2024-12-18T04:07:52.0635738Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 40: 2F7 1W5 2024-12-18T04:07:52.0637166Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 41: 2I6 3F7 2024-12-18T04:07:52.0638553Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 42: 2W6 3I7 2024-12-18T04:07:52.0639955Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 43: 2I7 3W6 2024-12-18T04:07:52.0641343Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 44: 2W7 1I6 2024-12-18T04:07:52.0643031Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 45: 0I6 3W7 2024-12-18T04:07:52.0644408Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 46: 0W6 1I7 2024-12-18T04:07:52.0645797Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 47: 0I7 1W6 2024-12-18T04:07:52.0647173Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] Step 48: 0W7 1W7 2024-12-18T04:07:52.0648424Z [rank1]:E1218 03:57:18.578000 27329 site-packages/torch/distributed/pipelining/schedules.py:1365] 2024-12-18T04:07:52.0649334Z ('RERUN', {'yellow': True}) [0.1403s] [ 96%] 2024-12-18T04:07:52.0652118Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_native_zero_bubble_ScheduleClass1 [rank0]:[E1218 04:07:18.010519655 ProcessGroupNCCL.cpp:629] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1307, OpType=COALESCED, NumelIn=18446744073709551615, NumelOut=18446744073709551615, Timeout(ms)=600000) ran for 600037 milliseconds before timing out. 2024-12-18T04:07:52.0655537Z [rank0]:[E1218 04:07:18.010809450 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 0] failure detected by watchdog at work sequence id: 1307 PG status: last enqueued work: 22, last completed work: 1306 2024-12-18T04:07:52.0657989Z [rank0]:[E1218 04:07:18.010828937 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value. 2024-12-18T04:07:52.0660859Z [rank0]:[E1218 04:07:18.010842392 ProcessGroupNCCL.cpp:681] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2024-12-18T04:07:52.0662857Z [rank0]:[E1218 04:07:18.010855277 ProcessGroupNCCL.cpp:695] [Rank 0] To avoid data inconsistency, we are taking the entire process down. 2024-12-18T04:07:52.0665704Z [rank0]:[E1218 04:07:18.012699619 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1307, OpType=COALESCED, NumelIn=18446744073709551615, NumelOut=18446744073709551615, Timeout(ms)=600000) ran for 600037 milliseconds before timing out. 2024-12-18T04:07:52.0668752Z Exception raised from checkTimeout at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first): 2024-12-18T04:07:52.0670974Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7ff04365b400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0673719Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x2f8 (0x7ff09c2aeff8 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0676252Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x9e0 (0x7ff09c2b0e20 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0678068Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x15a (0x7ff09c2b1c8a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0679570Z frame #4: + 0xcdaeb (0x7ff041bd5aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0680692Z frame #5: + 0x8609 (0x7ff0bf090609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0681682Z frame #6: clone + 0x43 (0x7ff0bee5b353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0682551Z 2024-12-18T04:07:52.0682923Z terminate called after throwing an instance of 'c10::DistBackendError' 2024-12-18T04:07:52.0685221Z what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1307, OpType=COALESCED, NumelIn=18446744073709551615, NumelOut=18446744073709551615, Timeout(ms)=600000) ran for 600037 milliseconds before timing out. 2024-12-18T04:07:52.0688029Z Exception raised from checkTimeout at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first): 2024-12-18T04:07:52.0690426Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7ff04365b400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0692955Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x2f8 (0x7ff09c2aeff8 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0695135Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x9e0 (0x7ff09c2b0e20 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0696956Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x15a (0x7ff09c2b1c8a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0698447Z frame #4: + 0xcdaeb (0x7ff041bd5aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0699558Z frame #5: + 0x8609 (0x7ff0bf090609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0700813Z frame #6: clone + 0x43 (0x7ff0bee5b353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0701344Z 2024-12-18T04:07:52.0702265Z Exception raised from ncclCommWatchdog at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first): 2024-12-18T04:07:52.0704494Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7ff04365b400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0706476Z frame #1: + 0x6c4104 (0x7ff09a52d104 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0707823Z frame #2: + 0xcdaeb (0x7ff041bd5aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0708911Z frame #3: + 0x8609 (0x7ff0bf090609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0709861Z frame #4: clone + 0x43 (0x7ff0bee5b353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0710358Z 2024-12-18T04:07:52.0710546Z Fatal Python error: Aborted 2024-12-18T04:07:52.0710844Z 2024-12-18T04:07:52.0711088Z Thread 0x00007fe456dff700 (most recent call first): 2024-12-18T04:07:52.0711642Z 2024-12-18T04:07:52.0711879Z 2024-12-18T04:07:52.0712097Z Thread 0x00007fe4a73ff700 (most recent call first): 2024-12-18T04:07:52.0712743Z 2024-12-18T04:07:52.0712978Z 2024-12-18T04:07:52.0713191Z Thread 0x00007fe5a9457700 (most recent call first): 2024-12-18T04:07:52.0714005Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 324 in wait 2024-12-18T04:07:52.0715074Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 607 in wait 2024-12-18T04:07:52.0716189Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run 2024-12-18T04:07:52.0717239Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2024-12-18T04:07:52.0718238Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2024-12-18T04:07:52.0718810Z 2024-12-18T04:07:52.0719356Z Thread 0x00007ff0bed3b180 (most recent call first): 2024-12-18T04:07:52.0720399Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4556 in barrier 2024-12-18T04:07:52.0721825Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81 in wrapper 2024-12-18T04:07:52.0723436Z File "/var/lib/jenkins/pytorch/test/distributed/pipelining/test_schedule_multiproc.py", line 917 in test_schedule_with_weight_update_mlp_e2e 2024-12-18T04:07:52.0725161Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 557 in instantiated_test 2024-12-18T04:07:52.0726695Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T04:07:52.0727980Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549 in _callTestMethod 2024-12-18T04:07:52.0728966Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591 in run 2024-12-18T04:07:52.0730382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3215 in _run_custom 2024-12-18T04:07:52.0731919Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3243 in run 2024-12-18T04:07:52.0733105Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 650 in __call__ 2024-12-18T04:07:52.0734184Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/unittest.py", line 333 in runtest 2024-12-18T04:07:52.0735442Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call 2024-12-18T04:07:52.0736688Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0738203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0739379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0740541Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 262 in 2024-12-18T04:07:52.0741699Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call 2024-12-18T04:07:52.0742914Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook 2024-12-18T04:07:52.0744191Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report 2024-12-18T04:07:52.0745446Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol 2024-12-18T04:07:52.0746831Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pytest_rerunfailures.py", line 549 in pytest_runtest_protocol 2024-12-18T04:07:52.0748171Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0749359Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0750536Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0751747Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 348 in pytest_runtestloop 2024-12-18T04:07:52.0753063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0754351Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0755712Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0756904Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 323 in _main 2024-12-18T04:07:52.0758040Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 269 in wrap_session 2024-12-18T04:07:52.0759564Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main 2024-12-18T04:07:52.0760804Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0761979Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0763132Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0764302Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/config/__init__.py", line 166 in main 2024-12-18T04:07:52.0765669Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests 2024-12-18T04:07:52.0767198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1541 in run_rank 2024-12-18T04:07:52.0768634Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90 in _wrap 2024-12-18T04:07:52.0769918Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2024-12-18T04:07:52.0771165Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2024-12-18T04:07:52.0772248Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2024-12-18T04:07:52.0773328Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2024-12-18T04:07:52.0774152Z File "", line 1 in 2024-12-18T04:07:52.0774493Z 2024-12-18T04:07:52.0783930Z Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, thriftpy2.transport.cybase, thriftpy2.transport.buffered.cybuffered, thriftpy2.transport.framed.cyframed, thriftpy2.transport.memory.cymemory, thriftpy2.transport.sasl.cysasl, thriftpy2.protocol.cybin.cybin, cython.cimports.libc.math, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, scipy._lib._ccallback_c, numba.mviewbuf, psutil._psutil_linux, psutil._psutil_posix (total: 62) 2024-12-18T04:07:52.0795000Z [rank1]:[E1218 04:07:18.168556700 ProcessGroupNCCL.cpp:629] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1315, OpType=RECV, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600065 milliseconds before timing out. 2024-12-18T04:07:52.0797596Z [rank1]:[E1218 04:07:18.168706001 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 1] failure detected by watchdog at work sequence id: 1315 PG status: last enqueued work: 1315, last completed work: 1314 2024-12-18T04:07:52.0800065Z [rank1]:[E1218 04:07:18.168723033 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value. 2024-12-18T04:07:52.0802941Z [rank1]:[E1218 04:07:18.168735597 ProcessGroupNCCL.cpp:681] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. 2024-12-18T04:07:52.0804923Z [rank1]:[E1218 04:07:18.168747088 ProcessGroupNCCL.cpp:695] [Rank 1] To avoid data inconsistency, we are taking the entire process down. 2024-12-18T04:07:52.0807494Z [rank1]:[E1218 04:07:18.170463550 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1315, OpType=RECV, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600065 milliseconds before timing out. 2024-12-18T04:07:52.0810425Z Exception raised from checkTimeout at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first): 2024-12-18T04:07:52.0812711Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7f4b375ca400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0815201Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x2f8 (0x7f4b9021dff8 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0817372Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x9e0 (0x7f4b9021fe20 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0819191Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x15a (0x7f4b90220c8a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0821033Z frame #4: + 0xcdaeb (0x7f4b35b44aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0822166Z frame #5: + 0x8609 (0x7f4bb2fff609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0823130Z frame #6: clone + 0x43 (0x7f4bb2dca353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0823627Z 2024-12-18T04:07:52.0824003Z terminate called after throwing an instance of 'c10::DistBackendError' 2024-12-18T04:07:52.0826009Z what(): [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1315, OpType=RECV, NumelIn=1, NumelOut=1, Timeout(ms)=600000) ran for 600065 milliseconds before timing out. 2024-12-18T04:07:52.0828542Z Exception raised from checkTimeout at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first): 2024-12-18T04:07:52.0830718Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7f4b375ca400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0833328Z frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional > >) + 0x2f8 (0x7f4b9021dff8 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0835854Z frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x9e0 (0x7f4b9021fe20 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0837692Z frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x15a (0x7f4b90220c8a in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0839177Z frame #4: + 0xcdaeb (0x7f4b35b44aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0840291Z frame #5: + 0x8609 (0x7f4bb2fff609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0841227Z frame #6: clone + 0x43 (0x7f4bb2dca353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0842019Z 2024-12-18T04:07:52.0842956Z Exception raised from ncclCommWatchdog at /var/lib/jenkins/workspace/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first): 2024-12-18T04:07:52.0845176Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0xb0 (0x7f4b375ca400 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so) 2024-12-18T04:07:52.0847171Z frame #1: + 0x6c4104 (0x7f4b8e49c104 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so) 2024-12-18T04:07:52.0848530Z frame #2: + 0xcdaeb (0x7f4b35b44aeb in /opt/conda/envs/py_3.10/lib/libstdc++.so.6) 2024-12-18T04:07:52.0849703Z frame #3: + 0x8609 (0x7f4bb2fff609 in /lib/x86_64-linux-gnu/libpthread.so.0) 2024-12-18T04:07:52.0850824Z frame #4: clone + 0x43 (0x7f4bb2dca353 in /lib/x86_64-linux-gnu/libc.so.6) 2024-12-18T04:07:52.0851381Z 2024-12-18T04:07:52.0851634Z Fatal Python error: Aborted 2024-12-18T04:07:52.0851920Z 2024-12-18T04:07:52.0852163Z Thread 0x00007f3f451ff700 (most recent call first): 2024-12-18T04:07:52.0852718Z 2024-12-18T04:07:52.0852950Z 2024-12-18T04:07:52.0853181Z Thread 0x00007f409ca56700 (most recent call first): 2024-12-18T04:07:52.0853725Z 2024-12-18T04:07:52.0853961Z 2024-12-18T04:07:52.0854179Z Thread 0x00007f409d457700 (most recent call first): 2024-12-18T04:07:52.0854939Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 324 in wait 2024-12-18T04:07:52.0855835Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 607 in wait 2024-12-18T04:07:52.0856820Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/tqdm/_monitor.py", line 60 in run 2024-12-18T04:07:52.0858191Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 1016 in _bootstrap_inner 2024-12-18T04:07:52.0859218Z File "/opt/conda/envs/py_3.10/lib/python3.10/threading.py", line 973 in _bootstrap 2024-12-18T04:07:52.0859802Z 2024-12-18T04:07:52.0860032Z Thread 0x00007f4bb2caa180 (most recent call first): 2024-12-18T04:07:52.0861138Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 3362 in recv_object_list 2024-12-18T04:07:52.0862627Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 81 in wrapper 2024-12-18T04:07:52.0864124Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/pipelining/stage.py", line 1345 in _shape_inference 2024-12-18T04:07:52.0865750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/pipelining/stage.py", line 1420 in _prepare_forward_infra 2024-12-18T04:07:52.0867451Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 1135 in _initialize_stages 2024-12-18T04:07:52.0869155Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 1218 in _step_microbatches 2024-12-18T04:07:52.0870752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/pipelining/schedules.py", line 1193 in step 2024-12-18T04:07:52.0872655Z File "/var/lib/jenkins/pytorch/test/distributed/pipelining/test_schedule_multiproc.py", line 585 in test_schedule_with_native_zero_bubble 2024-12-18T04:07:52.0874481Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 557 in instantiated_test 2024-12-18T04:07:52.0876241Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3108 in wrapper 2024-12-18T04:07:52.0877525Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 549 in _callTestMethod 2024-12-18T04:07:52.0878520Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 591 in run 2024-12-18T04:07:52.0880047Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3215 in _run_custom 2024-12-18T04:07:52.0881614Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3243 in run 2024-12-18T04:07:52.0883031Z File "/opt/conda/envs/py_3.10/lib/python3.10/unittest/case.py", line 650 in __call__ 2024-12-18T04:07:52.0884297Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/unittest.py", line 333 in runtest 2024-12-18T04:07:52.0885697Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call 2024-12-18T04:07:52.0886464Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0887133Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0887785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0888436Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 262 in 2024-12-18T04:07:52.0889091Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call 2024-12-18T04:07:52.0889772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook 2024-12-18T04:07:52.0890474Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report 2024-12-18T04:07:52.0891185Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol 2024-12-18T04:07:52.0892053Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pytest_rerunfailures.py", line 549 in pytest_runtest_protocol 2024-12-18T04:07:52.0893144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0893869Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0894447Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0895015Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 348 in pytest_runtestloop 2024-12-18T04:07:52.0895593Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0896142Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0896689Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0897222Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 323 in _main 2024-12-18T04:07:52.0897757Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 269 in wrap_session 2024-12-18T04:07:52.0898335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main 2024-12-18T04:07:52.0898939Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_callers.py", line 103 in _multicall 2024-12-18T04:07:52.0899496Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_manager.py", line 120 in _hookexec 2024-12-18T04:07:52.0900044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/pluggy/_hooks.py", line 513 in __call__ 2024-12-18T04:07:52.0900598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/_pytest/config/__init__.py", line 166 in main 2024-12-18T04:07:52.0901233Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1299 in run_tests 2024-12-18T04:07:52.0901955Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1541 in run_rank 2024-12-18T04:07:52.0902877Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90 in _wrap 2024-12-18T04:07:52.0903433Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 108 in run 2024-12-18T04:07:52.0903944Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 314 in _bootstrap 2024-12-18T04:07:52.0904453Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 129 in _main 2024-12-18T04:07:52.0904955Z File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/spawn.py", line 116 in spawn_main 2024-12-18T04:07:52.0905341Z File "", line 1 in 2024-12-18T04:07:52.0905505Z 2024-12-18T04:07:52.0909780Z Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, thriftpy2.transport.cybase, thriftpy2.transport.buffered.cybuffered, thriftpy2.transport.framed.cyframed, thriftpy2.transport.memory.cymemory, thriftpy2.transport.sasl.cysasl, thriftpy2.protocol.cybin.cybin, cython.cimports.libc.math, Cython.Utils, Cython.Plex.Actions, Cython.Plex.Transitions, Cython.Plex.Machines, Cython.Plex.DFA, Cython.Plex.Scanners, Cython.Compiler.Scanning, Cython.StringIOTree, Cython.Compiler.Code, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, scipy._lib._ccallback_c, numba.mviewbuf, psutil._psutil_linux, psutil._psutil_posix (total: 62) 2024-12-18T04:07:52.0914716Z W1218 04:07:18.970000 27133 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 27329 via signal SIGTERM 2024-12-18T04:07:52.0915363Z Traceback (most recent call last): 2024-12-18T04:07:52.0915972Z File "/var/lib/jenkins/pytorch/test/distributed/pipelining/test_schedule_multiproc.py", line 965, in 2024-12-18T04:07:52.0916530Z torch.multiprocessing.spawn( 2024-12-18T04:07:52.0917050Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn 2024-12-18T04:07:52.0917688Z return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") 2024-12-18T04:07:52.0918354Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes 2024-12-18T04:07:52.0918923Z while not context.join(): 2024-12-18T04:07:52.0919408Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 196, in join 2024-12-18T04:07:52.0919927Z raise ProcessExitedException( 2024-12-18T04:07:52.0920413Z torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGABRT 2024-12-18T04:07:52.0921370Z /opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 2 leaked semaphore objects to clean up at shutdown 2024-12-18T04:07:52.0922180Z warnings.warn('resource_tracker: There appear to be %d ' 2024-12-18T04:07:52.0922513Z Got exit code 1 2024-12-18T04:07:52.0922734Z Retrying single test... 2024-12-18T04:07:52.0923453Z Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-f8c4e99a71d9aac0.xml 2024-12-18T04:07:52.0924420Z ============================= test session starts ============================== 2024-12-18T04:07:52.0924912Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0925361Z cachedir: .pytest_cache 2024-12-18T04:07:52.0925881Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0926451Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0926722Z configfile: pytest.ini 2024-12-18T04:07:52.0927253Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0927911Z collecting ... collected 27 items / 26 deselected / 1 selected 2024-12-18T04:07:52.0928758Z stepcurrent: skipping 26 already run items. Running only test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T04:07:52.0929537Z Running 1 items in this shard 2024-12-18T04:07:52.0929731Z 2024-12-18T04:07:52.0930902Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-bf256e82606edcf2.xml 2024-12-18T04:07:52.0932199Z ============================= test session starts ============================== 2024-12-18T04:07:52.0932615Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0932990Z cachedir: .pytest_cache 2024-12-18T04:07:52.0933433Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0934075Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0934315Z configfile: pytest.ini 2024-12-18T04:07:52.0934767Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0935317Z collecting ... collected 27 items / 26 deselected / 1 selected 2024-12-18T04:07:52.0936027Z stepcurrent: skipping 26 already run items. Running only test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 2024-12-18T04:07:52.0936668Z Running 1 items in this shard 2024-12-18T04:07:52.0936816Z 2024-12-18T04:07:52.0937336Z distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0 libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2024-12-18T04:07:52.0938094Z libibverbs: Warning: couldn't open config directory '/etc/libibverbs.d'. 2024-12-18T04:07:52.0938443Z PASSED [6.8771s] [100%]PASSED [6.8179s] [100%] 2024-12-18T04:07:52.0938613Z 2024-12-18T04:07:52.0939205Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-f8c4e99a71d9aac0.xml - 2024-12-18T04:07:52.0939994Z ======================= 1 passed, 26 deselected in 7.47s ======================= 2024-12-18T04:07:52.0940194Z 2024-12-18T04:07:52.0940208Z 2024-12-18T04:07:52.0940776Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-bf256e82606edcf2.xml - 2024-12-18T04:07:52.0941529Z ======================= 1 passed, 26 deselected in 7.43s ======================= 2024-12-18T04:07:52.0941798Z Got exit code 0 2024-12-18T04:07:52.0942062Z Test succeeeded in new process, continuing with the rest of the tests 2024-12-18T04:07:52.0942781Z Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-2e79c232372605cf.xml 2024-12-18T04:07:52.0943562Z ============================= test session starts ============================== 2024-12-18T04:07:52.0943967Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0944339Z cachedir: .pytest_cache 2024-12-18T04:07:52.0944785Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0945257Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0945486Z configfile: pytest.ini 2024-12-18T04:07:52.0945933Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0946474Z collecting ... collected 27 items / 27 deselected / 0 selected 2024-12-18T04:07:52.0946788Z stepcurrent: skipping 27 already run items. 2024-12-18T04:07:52.0947047Z Running 0 items in this shard 2024-12-18T04:07:52.0947183Z 2024-12-18T04:07:52.0947756Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-2e79c232372605cf.xml - 2024-12-18T04:07:52.0948514Z ============================ 27 deselected in 0.01s ============================ 2024-12-18T04:07:52.0949181Z Test results will be stored in test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-283f336264fd0c2d.xml 2024-12-18T04:07:52.0949845Z ============================= test session starts ============================== 2024-12-18T04:07:52.0950246Z platform linux -- Python 3.10.15, pytest-7.3.2, pluggy-1.5.0 -- /opt/conda/envs/py_3.10/bin/python 2024-12-18T04:07:52.0950623Z cachedir: .pytest_cache 2024-12-18T04:07:52.0951186Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2024-12-18T04:07:52.0951664Z rootdir: /var/lib/jenkins/pytorch 2024-12-18T04:07:52.0951899Z configfile: pytest.ini 2024-12-18T04:07:52.0952400Z plugins: flakefinder-1.1.0, xdoctest-1.1.0, cpp-2.3.0, rerunfailures-14.0, hypothesis-5.35.1, xdist-3.3.1, subtests-0.13.1, typeguard-4.3.0 2024-12-18T04:07:52.0952946Z collecting ... collected 27 items / 27 deselected / 0 selected 2024-12-18T04:07:52.0953256Z stepcurrent: skipping 27 already run items. 2024-12-18T04:07:52.0953516Z Running 0 items in this shard 2024-12-18T04:07:52.0953667Z 2024-12-18T04:07:52.0954243Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.pipelining.test_schedule_multiproc/distributed.pipelining.test_schedule_multiproc-283f336264fd0c2d.xml - 2024-12-18T04:07:52.0954997Z ============================ 27 deselected in 0.01s ============================ 2024-12-18T04:07:52.0955730Z The following tests failed and then succeeded when run in a new process['test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0'] 2024-12-18T04:07:52.0956328Z 2024-12-18T04:07:52.0956810Z FINISHED PRINTING LOG FILE of distributed/pipelining/test_schedule_multiproc 1/1 (test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_e51da2f02d2e487f_.log) 2024-12-18T04:07:52.0957351Z 2024-12-18T04:07:52.0957585Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2024-12-18 04:07:52.033184] 2024-12-18T04:07:52.0957991Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:07:52.0958831Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:07:52.033721] 2024-12-18T04:08:17.3587616Z 2024-12-18T04:08:17.3589550Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_4e0d83cc5c85405f_.log 2024-12-18T04:08:17.3622452Z Running 29 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives 2024-12-18T04:08:17.3652595Z 2024-12-18T04:08:17.3653093Z Running distributed/_tools/test_sac_estimator 1/1 ... [2024-12-18 04:08:17.359105] 2024-12-18T04:08:17.3654483Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:08:17.3656590Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:08:17.359706] 2024-12-18T04:08:30.7569544Z 2024-12-18T04:08:30.7572408Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_5cbdafffc3021308_.log 2024-12-18T04:08:30.7575647Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation 2024-12-18T04:08:30.7577563Z 2024-12-18T04:08:30.7578019Z Running distributed/_tools/test_sac_ilp 1/1 ... [2024-12-18 04:08:30.757266] 2024-12-18T04:08:30.7578889Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:08:30.7582164Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_ilp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:08:30.757813] 2024-12-18T04:08:43.1541737Z 2024-12-18T04:08:43.1543656Z distributed/_tools/test_sac_ilp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_ilp_1.1_3a2ce9cc1e3ed346_.log 2024-12-18T04:08:43.1548377Z Running 4 items in this shard: test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case1, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case2, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case3, test/distributed/_tools/test_sac_ilp.py::TestOptimalCheckpointingPolicy::test_get_optimial_checkpointing_policy_per_module 2024-12-18T04:08:43.1551027Z 2024-12-18T04:08:43.1551527Z Running distributed/test_c10d_functional_native 1/1 ... [2024-12-18 04:08:43.153172] 2024-12-18T04:08:43.1552609Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:08:43.1554567Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:08:43.153700] 2024-12-18T04:11:37.3166156Z 2024-12-18T04:11:37.3167786Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_b73341f986aefbb9_.log 2024-12-18T04:11:37.3193417Z Running 28 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_py_work, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2024-12-18T04:11:37.3226031Z 2024-12-18T04:11:37.3226448Z Running distributed/test_backends 1/1 ... [2024-12-18 04:11:37.316994] 2024-12-18T04:11:37.3227313Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:11:37.3229770Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:11:37.317546] 2024-12-18T04:11:43.4986204Z 2024-12-18T04:11:43.4988056Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_8aeb069ac5d17b29_.log 2024-12-18T04:11:43.4991031Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda 2024-12-18T04:11:43.4992954Z 2024-12-18T04:11:43.4993474Z Running distributed/test_symmetric_memory 1/1 ... [2024-12-18 04:11:43.498765] 2024-12-18T04:11:43.4994556Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:11:43.5001943Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:11:43.499297] 2024-12-18T04:18:54.1822692Z 2024-12-18T04:18:54.1824240Z distributed/test_symmetric_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_symmetric_memory_1.1_d0b62954e6b1d24d_.log 2024-12-18T04:18:54.1923546Z Running 113 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_allow_overlapping_devices, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_barrier_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_signal_pad, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_has_multicast_support, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_large_alloc, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_put_signal_timeout, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_wait_signal_timeout, test/distributed/test_symmetric_memory.py::SubgroupTest::test_subgroup, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemAllReduceTest::test_two_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_memset32, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_stream_write_value32 2024-12-18T04:18:54.2020550Z 2024-12-18T04:18:54.2021037Z Running distributed/test_distributed_spawn 3/12 ... [2024-12-18 04:18:54.182460] 2024-12-18T04:18:54.2021990Z MPI not available -- MPI backend tests will be skipped 2024-12-18T04:18:54.2022886Z Running distributed tests for the test backend with env init_method 2024-12-18T04:18:54.2023674Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:18:54.2025699Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:18:54.187186] 2024-12-18T04:18:59.7066788Z 2024-12-18T04:18:59.7069265Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_f98080a29f095c6c_.log 2024-12-18T04:18:59.7070866Z Running 0 items in this shard: 2024-12-18T04:18:59.7071216Z 2024-12-18T04:18:59.7081561Z Running distributed tests for the test backend with file init_method 2024-12-18T04:18:59.7083559Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:18:59.7090412Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:18:59.708479] 2024-12-18T04:19:05.1988353Z 2024-12-18T04:19:05.1990272Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_498fb0017a764a16_.log 2024-12-18T04:19:05.1992108Z Running 0 items in this shard: 2024-12-18T04:19:05.1992733Z 2024-12-18T04:19:05.2003276Z Running distributed tests for the nccl backend with env init_method 2024-12-18T04:19:05.2004219Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:19:05.2013042Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:19:05.200635] 2024-12-18T04:23:32.0581112Z 2024-12-18T04:23:32.0583292Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_2882ef4803cd4f98_.log 2024-12-18T04:23:32.0601979Z Running 21 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:23:32.0627853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T04:23:32.0630172Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T04:23:32.0632007Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T04:23:32.0633946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T04:23:32.0635931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T04:23:32.0637749Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T04:23:32.0639667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T04:23:32.0641656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T04:23:32.0643931Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T04:23:32.0646075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T04:23:32.0647885Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T04:23:32.0649823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T04:23:32.0652025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T04:23:32.0654016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T04:23:32.0656138Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T04:23:32.0657925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T04:23:32.0659780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T04:23:32.0661608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T04:23:32.0663246Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T04:23:32.0664850Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T04:23:32.0666470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:23:32.0667352Z 2024-12-18T04:23:32.0667747Z Running distributed tests for the nccl backend with file init_method 2024-12-18T04:23:32.0668535Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:23:32.0670555Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:23:32.060312] 2024-12-18T04:27:58.8103141Z 2024-12-18T04:27:58.8105078Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_4e688c38fb4689ef_.log 2024-12-18T04:27:58.8125542Z Running 21 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:27:58.8143352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T04:27:58.8145394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T04:27:58.8147225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T04:27:58.8149074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T04:27:58.8151037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T04:27:58.8152986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T04:27:58.8154702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T04:27:58.8156419Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T04:27:58.8158683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T04:27:58.8161140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T04:27:58.8163355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T04:27:58.8165161Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T04:27:58.8167241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T04:27:58.8169258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T04:27:58.8171009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T04:27:58.8172817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T04:27:58.8174686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T04:27:58.8176516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T04:27:58.8178190Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T04:27:58.8179777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T04:27:58.8181393Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:27:58.8182272Z 2024-12-18T04:27:58.8182661Z Running distributed tests for the gloo backend with env init_method 2024-12-18T04:27:58.8183446Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:27:58.8185456Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:27:58.812476] 2024-12-18T04:32:34.2950600Z 2024-12-18T04:32:34.2952563Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_b7ae1d2cebfe89a4_.log 2024-12-18T04:32:34.2971488Z Running 21 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:32:34.2988985Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T04:32:34.2991014Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T04:32:34.2993002Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T04:32:34.2994858Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T04:32:34.2996802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T04:32:34.2998629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T04:32:34.3000403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T04:32:34.3002384Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T04:32:34.3004657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T04:32:34.3006760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T04:32:34.3009076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T04:32:34.3010871Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T04:32:34.3012976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T04:32:34.3014967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T04:32:34.3016739Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T04:32:34.3018530Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T04:32:34.3020392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T04:32:34.3022228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T04:32:34.3023879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T04:32:34.3025464Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T04:32:34.3027074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:32:34.3027928Z 2024-12-18T04:32:34.3028689Z Running distributed tests for the gloo backend with file init_method 2024-12-18T04:32:34.3029483Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:32:34.3031763Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:32:34.297286] 2024-12-18T04:37:10.3076211Z 2024-12-18T04:37:10.3078208Z distributed/test_distributed_spawn 3/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.12_93fb96f402b7c5cf_.log 2024-12-18T04:37:10.3100386Z Running 21 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:37:10.3118684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2024-12-18T04:37:10.3120741Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2024-12-18T04:37:10.3122550Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2024-12-18T04:37:10.3124414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group 2024-12-18T04:37:10.3126675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2024-12-18T04:37:10.3128785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2024-12-18T04:37:10.3130923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2024-12-18T04:37:10.3132647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group 2024-12-18T04:37:10.3134567Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false 2024-12-18T04:37:10.3136653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group 2024-12-18T04:37:10.3138477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu 2024-12-18T04:37:10.3140270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2024-12-18T04:37:10.3142373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2024-12-18T04:37:10.3144381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd 2024-12-18T04:37:10.3146156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks 2024-12-18T04:37:10.3147947Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather 2024-12-18T04:37:10.3149793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2024-12-18T04:37:10.3151623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda 2024-12-18T04:37:10.3153412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2024-12-18T04:37:10.3155212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list 2024-12-18T04:37:10.3157413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2024-12-18T04:37:10.3158286Z 2024-12-18T04:37:10.3158725Z Running distributed/test_distributed_spawn 6/12 ... [2024-12-18 04:37:10.308835] 2024-12-18T04:37:10.3159693Z MPI not available -- MPI backend tests will be skipped 2024-12-18T04:37:10.3160579Z Running distributed tests for the test backend with env init_method 2024-12-18T04:37:10.3161367Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:37:10.3163405Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:37:10.312993] 2024-12-18T04:37:15.7859313Z 2024-12-18T04:37:15.7860963Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_471d536ef15f690d_.log 2024-12-18T04:37:15.7862616Z Running 0 items in this shard: 2024-12-18T04:37:15.7862978Z 2024-12-18T04:37:15.7876121Z Running distributed tests for the test backend with file init_method 2024-12-18T04:37:15.7881067Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:37:15.7883203Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:37:15.787737] 2024-12-18T04:37:21.2518010Z 2024-12-18T04:37:21.2520659Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_774e0ff2f4c3f82b_.log 2024-12-18T04:37:21.2522295Z Running 0 items in this shard: 2024-12-18T04:37:21.2522665Z 2024-12-18T04:37:21.2530642Z Running distributed tests for the nccl backend with env init_method 2024-12-18T04:37:21.2533120Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:37:21.2540035Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:37:21.253352] 2024-12-18T04:41:57.2563388Z 2024-12-18T04:41:57.2569267Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_3ab0d8af6952ef4d_.log 2024-12-18T04:41:57.2585667Z Running 20 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:41:57.2602900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T04:41:57.2605200Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T04:41:57.2607199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T04:41:57.2608977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T04:41:57.2610694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T04:41:57.2613123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T04:41:57.2615288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T04:41:57.2617043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T04:41:57.2618818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T04:41:57.2620618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T04:41:57.2622389Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T04:41:57.2624225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T04:41:57.2626046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T04:41:57.2627925Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T04:41:57.2629707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T04:41:57.2631434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T04:41:57.2633192Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T04:41:57.2634810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T04:41:57.2636785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T04:41:57.2638632Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:41:57.2639968Z 2024-12-18T04:41:57.2640383Z Running distributed tests for the nccl backend with file init_method 2024-12-18T04:41:57.2641167Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:41:57.2643177Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:41:57.258567] 2024-12-18T04:46:30.7018927Z 2024-12-18T04:46:30.7020643Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_3ff8e3996d448505_.log 2024-12-18T04:46:30.7037723Z Running 20 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:46:30.7053908Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T04:46:30.7056132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T04:46:30.7058072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T04:46:30.7059832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T04:46:30.7061864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T04:46:30.7064042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T04:46:30.7066625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T04:46:30.7068375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T04:46:30.7070132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T04:46:30.7071968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T04:46:30.7073894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T04:46:30.7076068Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T04:46:30.7077869Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T04:46:30.7079720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T04:46:30.7081496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T04:46:30.7083203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T04:46:30.7084830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T04:46:30.7086417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T04:46:30.7088437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T04:46:30.7090191Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:46:30.7091326Z 2024-12-18T04:46:30.7091786Z Running distributed tests for the gloo backend with env init_method 2024-12-18T04:46:30.7092595Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:46:30.7094719Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:46:30.704080] 2024-12-18T04:51:36.2520721Z 2024-12-18T04:51:36.2522538Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_5eb9e6c308aefe1a_.log 2024-12-18T04:51:36.2541132Z Running 20 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:51:36.2558612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T04:51:36.2560839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T04:51:36.2562795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T04:51:36.2564893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T04:51:36.2566942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T04:51:36.2569492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T04:51:36.2571600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T04:51:36.2573324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T04:51:36.2575095Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T04:51:36.2576900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T04:51:36.2578655Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T04:51:36.2580482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T04:51:36.2582295Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T04:51:36.2584142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T04:51:36.2585923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T04:51:36.2587600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T04:51:36.2589228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T04:51:36.2590811Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T04:51:36.2592556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T04:51:36.2595025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:51:36.2596093Z 2024-12-18T04:51:36.2596491Z Running distributed tests for the gloo backend with file init_method 2024-12-18T04:51:36.2597273Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:51:36.2599279Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:51:36.254214] 2024-12-18T04:56:41.9849387Z 2024-12-18T04:56:41.9851218Z distributed/test_distributed_spawn 6/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.12_7da9f2667390a8b1_.log 2024-12-18T04:56:41.9870567Z Running 20 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:56:41.9888122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient 2024-12-18T04:56:41.9890576Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream 2024-12-18T04:56:41.9892873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum 2024-12-18T04:56:41.9894687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda 2024-12-18T04:56:41.9896699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo 2024-12-18T04:56:41.9898656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger 2024-12-18T04:56:41.9901305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks 2024-12-18T04:56:41.9903063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device 2024-12-18T04:56:41.9904801Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error 2024-12-18T04:56:41.9906586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple 2024-12-18T04:56:41.9908376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph 2024-12-18T04:56:41.9910355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks 2024-12-18T04:56:41.9912408Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad 2024-12-18T04:56:41.9914335Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity 2024-12-18T04:56:41.9916349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min 2024-12-18T04:56:41.9918040Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum 2024-12-18T04:56:41.9919654Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min 2024-12-18T04:56:41.9921778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex 2024-12-18T04:56:41.9923734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex 2024-12-18T04:56:41.9925719Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda 2024-12-18T04:56:41.9926697Z 2024-12-18T04:56:41.9927142Z Running distributed/test_distributed_spawn 9/12 ... [2024-12-18 04:56:41.985865] 2024-12-18T04:56:41.9928085Z MPI not available -- MPI backend tests will be skipped 2024-12-18T04:56:41.9928948Z Running distributed tests for the test backend with env init_method 2024-12-18T04:56:41.9929728Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:56:41.9931756Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:56:41.989715] 2024-12-18T04:56:47.5005620Z 2024-12-18T04:56:47.5007481Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_825cfef854325273_.log 2024-12-18T04:56:47.5009228Z Running 0 items in this shard: 2024-12-18T04:56:47.5009601Z 2024-12-18T04:56:47.5019637Z Running distributed tests for the test backend with file init_method 2024-12-18T04:56:47.5022871Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:56:47.5029617Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:56:47.502298] 2024-12-18T04:56:53.0406724Z 2024-12-18T04:56:53.0408767Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_4f91b4ca270ae8fc_.log 2024-12-18T04:56:53.0411473Z Running 0 items in this shard: 2024-12-18T04:56:53.0411830Z 2024-12-18T04:56:53.0420319Z Running distributed tests for the nccl backend with env init_method 2024-12-18T04:56:53.0422658Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T04:56:53.0430339Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 04:56:53.042383] 2024-12-18T05:02:19.6207058Z 2024-12-18T05:02:19.6208702Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_c391e52e7f8b7fb4_.log 2024-12-18T05:02:19.6232189Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:02:19.6254034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T05:02:19.6255899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T05:02:19.6258038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T05:02:19.6259839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T05:02:19.6261696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T05:02:19.6263531Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T05:02:19.6265293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T05:02:19.6267054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T05:02:19.6268906Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T05:02:19.6270803Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T05:02:19.6273089Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T05:02:19.6274972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T05:02:19.6276758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T05:02:19.6278893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T05:02:19.6281271Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T05:02:19.6283442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T05:02:19.6285042Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T05:02:19.6286766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T05:02:19.6288712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T05:02:19.6291045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T05:02:19.6293087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T05:02:19.6295077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T05:02:19.6296851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T05:02:19.6298560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T05:02:19.6300409Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T05:02:19.6302365Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:02:19.6303423Z 2024-12-18T05:02:19.6304153Z Running distributed tests for the nccl backend with file init_method 2024-12-18T05:02:19.6304953Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:02:19.6306967Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:02:19.623042] 2024-12-18T05:07:45.8393009Z 2024-12-18T05:07:45.8394631Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_57648f8c3b1c4072_.log 2024-12-18T05:07:45.8418069Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:07:45.8439167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T05:07:45.8441025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T05:07:45.8442793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T05:07:45.8454104Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T05:07:45.8455967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T05:07:45.8457795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T05:07:45.8459558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T05:07:45.8461317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T05:07:45.8463169Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T05:07:45.8465054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T05:07:45.8466899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T05:07:45.8468775Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T05:07:45.8470570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T05:07:45.8472684Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T05:07:45.8475554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T05:07:45.8477735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T05:07:45.8479337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T05:07:45.8481062Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T05:07:45.8483065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T05:07:45.8485473Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T05:07:45.8487883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T05:07:45.8489933Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T05:07:45.8491740Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T05:07:45.8493487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T05:07:45.8495720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T05:07:45.8497732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:07:45.8498778Z 2024-12-18T05:07:45.8499175Z Running distributed tests for the gloo backend with env init_method 2024-12-18T05:07:45.8499955Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:07:45.8501962Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:07:45.841738] 2024-12-18T05:13:32.6029283Z 2024-12-18T05:13:32.6035395Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_2e77be0ef2e34d7c_.log 2024-12-18T05:13:32.6058813Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:13:32.6080918Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T05:13:32.6082783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T05:13:32.6084456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T05:13:32.6086245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T05:13:32.6088526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T05:13:32.6090355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T05:13:32.6092108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T05:13:32.6093856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T05:13:32.6095679Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T05:13:32.6097636Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T05:13:32.6099487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T05:13:32.6101368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T05:13:32.6103156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T05:13:32.6104937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T05:13:32.6106994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T05:13:32.6109304Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T05:13:32.6110901Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T05:13:32.6112928Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T05:13:32.6114896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T05:13:32.6116974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T05:13:32.6119000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T05:13:32.6121000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T05:13:32.6122797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T05:13:32.6124516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T05:13:32.6126427Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T05:13:32.6128762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:13:32.6129846Z 2024-12-18T05:13:32.6130252Z Running distributed tests for the gloo backend with file init_method 2024-12-18T05:13:32.6131035Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:13:32.6133051Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=9', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:13:32.605232] 2024-12-18T05:19:20.2688510Z 2024-12-18T05:19:20.2691678Z distributed/test_distributed_spawn 9/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_9.12_bf0507b9acece932_.log 2024-12-18T05:19:20.2714257Z Running 26 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:19:20.2736422Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view 2024-12-18T05:19:20.2738326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather 2024-12-18T05:19:20.2740026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2024-12-18T05:19:20.2741835Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2024-12-18T05:19:20.2743702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda 2024-12-18T05:19:20.2745978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product 2024-12-18T05:19:20.2747770Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2024-12-18T05:19:20.2749569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex 2024-12-18T05:19:20.2751423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2024-12-18T05:19:20.2753439Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2024-12-18T05:19:20.2755298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl 2024-12-18T05:19:20.2757179Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook 2024-12-18T05:19:20.2759057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2024-12-18T05:19:20.2760905Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD 2024-12-18T05:19:20.2763021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2024-12-18T05:19:20.2765003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks 2024-12-18T05:19:20.2767171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object 2024-12-18T05:19:20.2769160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2024-12-18T05:19:20.2771146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank 2024-12-18T05:19:20.2773229Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size 2024-12-18T05:19:20.2775250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2024-12-18T05:19:20.2777245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler 2024-12-18T05:19:20.2779052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag 2024-12-18T05:19:20.2780763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2024-12-18T05:19:20.2782640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters 2024-12-18T05:19:20.2784608Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2024-12-18T05:19:20.2785663Z 2024-12-18T05:19:20.2786123Z Running distributed/test_distributed_spawn 12/12 ... [2024-12-18 05:19:20.270725] 2024-12-18T05:19:20.2787069Z MPI not available -- MPI backend tests will be skipped 2024-12-18T05:19:20.2787957Z Running distributed tests for the test backend with env init_method 2024-12-18T05:19:20.2788749Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:19:20.2790819Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:20.275063] 2024-12-18T05:19:25.8680513Z 2024-12-18T05:19:25.8682576Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_16372eabfd37620f_.log 2024-12-18T05:19:25.8684294Z Running 0 items in this shard: 2024-12-18T05:19:25.8684663Z 2024-12-18T05:19:25.8693299Z Running distributed tests for the test backend with file init_method 2024-12-18T05:19:25.8696249Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:19:25.8703185Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:25.869644] 2024-12-18T05:19:31.4086586Z 2024-12-18T05:19:31.4087505Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_5e61bf474ead82f1_.log 2024-12-18T05:19:31.4088351Z Running 0 items in this shard: 2024-12-18T05:19:31.4088544Z 2024-12-18T05:19:31.4099787Z Running distributed tests for the nccl backend with env init_method 2024-12-18T05:19:31.4105395Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:19:31.4111044Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:19:31.410411] 2024-12-18T05:23:04.6517846Z 2024-12-18T05:23:04.6520376Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_89b262e495d80c88_.log 2024-12-18T05:23:04.6539049Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:23:04.6556870Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T05:23:04.6558571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T05:23:04.6560297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T05:23:04.6562130Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T05:23:04.6564140Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T05:23:04.6566114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T05:23:04.6568142Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T05:23:04.6570016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T05:23:04.6571899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T05:23:04.6573866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T05:23:04.6576158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T05:23:04.6577864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T05:23:04.6579460Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T05:23:04.6581235Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T05:23:04.6583035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T05:23:04.6584776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T05:23:04.6586830Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T05:23:04.6588836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T05:23:04.6590499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T05:23:04.6592162Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T05:23:04.6594161Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T05:23:04.6596148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:23:04.6597176Z 2024-12-18T05:23:04.6597584Z Running distributed tests for the nccl backend with file init_method 2024-12-18T05:23:04.6598384Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:23:04.6600416Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:23:04.653793] 2024-12-18T05:26:39.0568607Z 2024-12-18T05:26:39.0570182Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_8f28c6725dedfc9c_.log 2024-12-18T05:26:39.0588130Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:26:39.0605337Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T05:26:39.0607012Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T05:26:39.0608759Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T05:26:39.0610579Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T05:26:39.0612328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T05:26:39.0613961Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T05:26:39.0615687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T05:26:39.0617520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T05:26:39.0619917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T05:26:39.0621886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T05:26:39.0623746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T05:26:39.0625404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T05:26:39.0626964Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T05:26:39.0628724Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T05:26:39.0630502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T05:26:39.0632559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T05:26:39.0634633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T05:26:39.0636633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T05:26:39.0638623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T05:26:39.0640766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T05:26:39.0642399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T05:26:39.0644175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:26:39.0645412Z 2024-12-18T05:26:39.0645869Z Running distributed tests for the gloo backend with env init_method 2024-12-18T05:26:39.0646803Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:26:39.0649185Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:26:39.059260] 2024-12-18T05:30:41.7741325Z 2024-12-18T05:30:41.7743291Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_f6f55994a30b1dba_.log 2024-12-18T05:30:41.7762764Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:30:41.7789491Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T05:30:41.7791372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T05:30:41.7793510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T05:30:41.7795543Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T05:30:41.7797866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T05:30:41.7799541Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T05:30:41.7801281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T05:30:41.7803413Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T05:30:41.7805627Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T05:30:41.7807711Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T05:30:41.7809617Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T05:30:41.7811305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T05:30:41.7812909Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T05:30:41.7814700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T05:30:41.7816520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T05:30:41.7818280Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T05:30:41.7820326Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T05:30:41.7822317Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T05:30:41.7824324Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T05:30:41.7825982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T05:30:41.7827601Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T05:30:41.7829380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:30:41.7830418Z 2024-12-18T05:30:41.7830806Z Running distributed tests for the gloo backend with file init_method 2024-12-18T05:30:41.7831597Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:30:41.7833934Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=12', '--num-shards=12', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:30:41.776323] 2024-12-18T05:34:44.3948986Z 2024-12-18T05:34:44.3951158Z distributed/test_distributed_spawn 12/12 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_12.12_2e1379a3c32c97e3_.log 2024-12-18T05:34:44.3969523Z Running 22 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:34:44.3986397Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2024-12-18T05:34:44.3988126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group 2024-12-18T05:34:44.3990318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum 2024-12-18T05:34:44.3992132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product 2024-12-18T05:34:44.3994033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2024-12-18T05:34:44.3995855Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2024-12-18T05:34:44.3997916Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl 2024-12-18T05:34:44.3999795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl 2024-12-18T05:34:44.4001699Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2024-12-18T05:34:44.4003660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks 2024-12-18T05:34:44.4005538Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types 2024-12-18T05:34:44.4007211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2024-12-18T05:34:44.4008816Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group 2024-12-18T05:34:44.4010946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2024-12-18T05:34:44.4012776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce 2024-12-18T05:34:44.4014535Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream 2024-12-18T05:34:44.4016575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2024-12-18T05:34:44.4018569Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max 2024-12-18T05:34:44.4020242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice 2024-12-18T05:34:44.4021888Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2024-12-18T05:34:44.4023513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group 2024-12-18T05:34:44.4025290Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler 2024-12-18T05:34:44.4026330Z 2024-12-18T05:34:44.4026964Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2024-12-18 05:34:44.396393] 2024-12-18T05:34:44.4028089Z MPI not available -- MPI backend tests will be skipped 2024-12-18T05:34:44.4028969Z Running distributed tests for the test backend with env init_method 2024-12-18T05:34:44.4029760Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:34:44.4031988Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:34:44.400671] 2024-12-18T05:34:49.7452453Z 2024-12-18T05:34:49.7454545Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_47099b443fc55f91_.log 2024-12-18T05:34:49.7455558Z Running 0 items in this shard: 2024-12-18T05:34:49.7455750Z 2024-12-18T05:34:49.7465348Z Running distributed tests for the test backend with file init_method 2024-12-18T05:34:49.7467818Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:34:49.7474985Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:34:49.746852] 2024-12-18T05:34:55.1167845Z 2024-12-18T05:34:55.1170305Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_03fc4e4a95946e85_.log 2024-12-18T05:34:55.1172673Z Running 0 items in this shard: 2024-12-18T05:34:55.1173128Z 2024-12-18T05:34:55.1181978Z Running distributed tests for the nccl backend with env init_method 2024-12-18T05:34:55.1184824Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:34:55.1192501Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:34:55.118537] 2024-12-18T05:35:58.2216134Z 2024-12-18T05:35:58.2218721Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_f79748d3b23fa0ee_.log 2024-12-18T05:35:58.2226167Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:35:58.2232719Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T05:35:58.2234915Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T05:35:58.2236891Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T05:35:58.2238849Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T05:35:58.2240823Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T05:35:58.2242864Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:35:58.2244158Z 2024-12-18T05:35:58.2244645Z Running distributed tests for the nccl backend with file init_method 2024-12-18T05:35:58.2245535Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:35:58.2247755Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:35:58.223519] 2024-12-18T05:37:01.2480007Z 2024-12-18T05:37:01.2485724Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_55aee1f52501ece7_.log 2024-12-18T05:37:01.2492993Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:37:01.2499392Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T05:37:01.2501377Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T05:37:01.2503300Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T05:37:01.2505219Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T05:37:01.2507826Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T05:37:01.2509944Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:37:01.2511072Z 2024-12-18T05:37:01.2511483Z Running distributed tests for the gloo backend with env init_method 2024-12-18T05:37:01.2512649Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:37:01.2514902Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:37:01.249768] 2024-12-18T05:37:49.7728734Z 2024-12-18T05:37:49.7730839Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_d7de43dbeb41849b_.log 2024-12-18T05:37:49.7738729Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:37:49.7744829Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T05:37:49.7746853Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T05:37:49.7749737Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T05:37:49.7751862Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T05:37:49.7753990Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T05:37:49.7755998Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:37:49.7757086Z 2024-12-18T05:37:49.7757485Z Running distributed tests for the gloo backend with file init_method 2024-12-18T05:37:49.7758311Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:37:49.7760483Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:37:49.774913] 2024-12-18T05:38:38.1283825Z 2024-12-18T05:38:38.1289881Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_a0686c84fb1180ef_.log 2024-12-18T05:38:38.1297192Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:38:38.1303438Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16 2024-12-18T05:38:38.1305623Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16 2024-12-18T05:38:38.1307561Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16 2024-12-18T05:38:38.1309496Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16 2024-12-18T05:38:38.1311473Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16 2024-12-18T05:38:38.1313946Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16 2024-12-18T05:38:38.1315064Z 2024-12-18T05:38:38.1315611Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2024-12-18 05:38:38.129731] 2024-12-18T05:38:38.1316546Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:38:38.1318546Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:38:38.130278] 2024-12-18T05:41:09.6956796Z 2024-12-18T05:41:09.6959364Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_d8cab12536a567e0_.log 2024-12-18T05:41:09.6979525Z Running 16 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_float64_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_0_attention_wv__layers_0_feed_forward_w1__layers_1_feed_forward_w2__layers_1_ffn_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_1_ffn_norm__norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying 2024-12-18T05:41:09.6997841Z 2024-12-18T05:41:09.6998331Z Running distributed/test_collective_utils 1/1 ... [2024-12-18 05:41:09.695839] 2024-12-18T05:41:09.6999204Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:41:09.7001127Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:41:09.696387] 2024-12-18T05:41:14.7685033Z 2024-12-18T05:41:14.7686875Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_d7bea068afe5c412_.log 2024-12-18T05:41:14.7688330Z 2024-12-18T05:41:14.7690894Z Running distributed/test_c10d_logger 1/1 ... [2024-12-18 05:41:14.768653] 2024-12-18T05:41:14.7691726Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:41:14.7698589Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:41:14.769199] 2024-12-18T05:41:32.6262911Z 2024-12-18T05:41:32.6265462Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_540f669cd530fe99_.log 2024-12-18T05:41:32.6268332Z Running 2 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger 2024-12-18T05:41:32.6270925Z 2024-12-18T05:41:32.6271464Z Running distributed/fsdp/test_fsdp_state_dict 1/3 ... [2024-12-18 05:41:32.625917] 2024-12-18T05:41:32.6272582Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:41:32.6274707Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:41:32.626453] 2024-12-18T05:50:28.1582066Z 2024-12-18T05:50:28.1583582Z distributed/fsdp/test_fsdp_state_dict 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.3_eb9cb9fa0649049c_.log 2024-12-18T05:50:28.1671032Z Running 62 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard 2024-12-18T05:50:28.1754650Z 2024-12-18T05:50:28.1755119Z Running distributed/_tensor/test_optimizers 1/1 ... [2024-12-18 05:50:28.160096] 2024-12-18T05:50:28.1755988Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:50:28.1757968Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:50:28.160722] 2024-12-18T05:52:53.6607672Z 2024-12-18T05:52:53.6613313Z distributed/_tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_optimizers_1.1_518e822da677e1d7_.log 2024-12-18T05:52:53.6622821Z Running 11 items in this shard: test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/_tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2024-12-18T05:52:53.6630689Z 2024-12-18T05:52:53.6631109Z Running distributed/test_c10d_nccl 3/3 ... [2024-12-18 05:52:53.660837] 2024-12-18T05:52:53.6632057Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T05:52:53.6634315Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 05:52:53.661246] 2024-12-18T06:07:07.9981268Z 2024-12-18T06:07:07.9987011Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_0ca76520b095e295_.log 2024-12-18T06:07:08.0021523Z Running 64 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout, test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T06:07:08.0054779Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str 2024-12-18T06:07:08.0055832Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2024-12-18T06:07:08.0056877Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup 2024-12-18T06:07:08.0057860Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context 2024-12-18T06:07:08.0058776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2024-12-18T06:07:08.0059677Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2024-12-18T06:07:08.0060637Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2024-12-18T06:07:08.0061657Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2024-12-18T06:07:08.0062626Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg 2024-12-18T06:07:08.0063597Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2024-12-18T06:07:08.0064655Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False 2024-12-18T06:07:08.0065762Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2024-12-18T06:07:08.0066888Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2024-12-18T06:07:08.0067896Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2024-12-18T06:07:08.0069120Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2024-12-18T06:07:08.0070307Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T06:07:08.0072106Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2024-12-18T06:07:08.0074010Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T06:07:08.0075597Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T06:07:08.0077083Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2024-12-18T06:07:08.0078616Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T06:07:08.0080286Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T06:07:08.0082075Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2024-12-18T06:07:08.0083454Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2024-12-18T06:07:08.0084411Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2024-12-18T06:07:08.0085702Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2024-12-18T06:07:08.0086872Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2024-12-18T06:07:08.0088006Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2024-12-18T06:07:08.0089190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2024-12-18T06:07:08.0090351Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T06:07:08.0091924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2024-12-18T06:07:08.0093277Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2024-12-18T06:07:08.0094540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking_abort 2024-12-18T06:07:08.0095713Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_timeout 2024-12-18T06:07:08.0096974Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclUserBufferRegistrationTest::test_nccl_user_buffer_registration 2024-12-18T06:07:08.0098241Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2024-12-18T06:07:08.0099413Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids_function_argument 2024-12-18T06:07:08.0100624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2024-12-18T06:07:08.0101787Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2024-12-18T06:07:08.0103275Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced_float8_errors 2024-12-18T06:07:08.0104489Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2024-12-18T06:07:08.0105587Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2024-12-18T06:07:08.0106648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2024-12-18T06:07:08.0107935Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2024-12-18T06:07:08.0109531Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2024-12-18T06:07:08.0111142Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2024-12-18T06:07:08.0112852Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T06:07:08.0114450Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False 2024-12-18T06:07:08.0115854Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True 2024-12-18T06:07:08.0117101Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2024-12-18T06:07:08.0118618Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2024-12-18T06:07:08.0119826Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True 2024-12-18T06:07:08.0121224Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False 2024-12-18T06:07:08.0122759Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True 2024-12-18T06:07:08.0123865Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling 2024-12-18T06:07:08.0124853Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2024-12-18T06:07:08.0125985Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2024-12-18T06:07:08.0127072Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2024-12-18T06:07:08.0128154Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2024-12-18T06:07:08.0129282Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2024-12-18T06:07:08.0130478Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2024-12-18T06:07:08.0132159Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2024-12-18T06:07:08.0133519Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2024-12-18T06:07:08.0134776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2024-12-18T06:07:08.0135478Z 2024-12-18T06:07:08.0135863Z Running distributed/pipelining/test_schedule 1/1 ... [2024-12-18 06:07:07.998667] 2024-12-18T06:07:08.0136803Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:07:08.0138339Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:07:07.999106] 2024-12-18T06:07:51.0141713Z 2024-12-18T06:07:51.0143511Z distributed/pipelining/test_schedule 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_1.1_7b6f0b928d7e5a04_.log 2024-12-18T06:07:51.0167841Z Running 28 items in this shard: test/distributed/pipelining/test_schedule.py::ScheduleTest::test_get_schedule_class, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass2, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass3, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass4, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref2, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref3, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref4, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref5, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref6, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref7, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_csv_csv_name_zb1p_2rank_2stagep, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_split_b_w, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_v_schedule, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_merge_bw_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_unshard_reshard_test_info0, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_action, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_rank, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_valid_schedule 2024-12-18T06:07:51.0189059Z 2024-12-18T06:07:51.0189524Z Running distributed/_tensor/test_op_strategy 1/1 ... [2024-12-18 06:07:51.014489] 2024-12-18T06:07:51.0190433Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:07:51.0192509Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tensor/test_op_strategy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:07:51.015051] 2024-12-18T06:07:57.0458499Z 2024-12-18T06:07:57.0460153Z distributed/_tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tensor.test_op_strategy_1.1_f7a637a035e1472d_.log 2024-12-18T06:07:57.0470246Z Running 15 items in this shard: test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/_tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_latency, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/_tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d 2024-12-18T06:07:57.0479779Z 2024-12-18T06:07:57.0480422Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2024-12-18 06:07:57.045599] 2024-12-18T06:07:57.0481514Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:07:57.0484398Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:07:57.046140] 2024-12-18T06:08:20.9185855Z 2024-12-18T06:08:20.9187742Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_f6fb685a9b058114_.log 2024-12-18T06:08:20.9191455Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding 2024-12-18T06:08:20.9193672Z 2024-12-18T06:08:20.9194243Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2024-12-18 06:08:20.918886] 2024-12-18T06:08:20.9195232Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:08:20.9200949Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:08:20.919491] 2024-12-18T06:11:53.7059246Z 2024-12-18T06:11:53.7061113Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_ba89cd8ec8b1b6c2_.log 2024-12-18T06:11:53.7075957Z Running 15 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init 2024-12-18T06:11:53.7090115Z 2024-12-18T06:11:53.7090666Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2024-12-18 06:11:53.707579] 2024-12-18T06:11:53.7091592Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:11:53.7094009Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:11:53.708148] 2024-12-18T06:15:14.4900499Z 2024-12-18T06:15:14.4907287Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_c7549696edecccb6_.log 2024-12-18T06:15:14.4927929Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_raises_warning_or_errors_cuda 2024-12-18T06:15:14.4948542Z 2024-12-18T06:15:14.4949251Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2024-12-18 06:15:14.490017] 2024-12-18T06:15:14.4950400Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:15:14.4952603Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:15:14.490558] 2024-12-18T06:16:04.8211932Z 2024-12-18T06:16:04.8213877Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_e72928d74a140562_.log 2024-12-18T06:16:04.8222100Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group 2024-12-18T06:16:04.8225248Z 2024-12-18T06:16:04.8225772Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2024-12-18 06:16:04.821374] 2024-12-18T06:16:04.8226708Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:16:04.8228693Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:16:04.821950] 2024-12-18T06:23:16.4858640Z 2024-12-18T06:23:16.4860534Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_4bd3f8900effcaea_.log 2024-12-18T06:23:16.4916778Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True 2024-12-18T06:23:16.4968967Z 2024-12-18T06:23:16.4969434Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2024-12-18 06:23:16.486406] 2024-12-18T06:23:16.4970288Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:23:16.4972237Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:23:16.486967] 2024-12-18T06:24:41.3964505Z 2024-12-18T06:24:41.3971229Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_ec0ac7d4ca62c56a_.log 2024-12-18T06:24:41.3977040Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2024-12-18T06:24:41.3981435Z 2024-12-18T06:24:41.3981988Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2024-12-18 06:24:41.396471] 2024-12-18T06:24:41.3983696Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:24:41.3985685Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:24:41.397047] 2024-12-18T06:25:00.9079339Z 2024-12-18T06:25:00.9081121Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_29964f266c838f48_.log 2024-12-18T06:25:00.9083636Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrappingCUDA::test_multiple_wrapping_cuda 2024-12-18T06:25:00.9084729Z 2024-12-18T06:25:00.9085428Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2024-12-18 06:25:00.907627] 2024-12-18T06:25:00.9086476Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:25:00.9088653Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:25:00.908197] 2024-12-18T06:26:21.6607598Z 2024-12-18T06:26:21.6613121Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_69f36c5f89c6dd54_.log 2024-12-18T06:26:21.6622705Z Running 7 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_grad_acc_with_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs 2024-12-18T06:26:21.6629705Z 2024-12-18T06:26:21.6630342Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2024-12-18 06:26:21.661208] 2024-12-18T06:26:21.6631350Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:26:21.6633588Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:26:21.661805] 2024-12-18T06:27:17.3089160Z 2024-12-18T06:27:17.3090953Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_b90b0c4141655a79_.log 2024-12-18T06:27:17.3099384Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_2d_state_dict_correctness, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_cpu_offload, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_hsdp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiThread::test_rank0_offload_full_state_dict 2024-12-18T06:27:17.3106040Z 2024-12-18T06:27:17.3106498Z Running distributed/algorithms/test_join 1/1 ... [2024-12-18 06:27:17.309168] 2024-12-18T06:27:17.3107365Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:27:17.3109294Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:27:17.309756] 2024-12-18T06:28:24.2829488Z 2024-12-18T06:28:24.2832092Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_0867296d306e5445_.log 2024-12-18T06:28:24.2838756Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw 2024-12-18T06:28:24.2844050Z 2024-12-18T06:28:24.2852870Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2024-12-18 06:28:24.283095] 2024-12-18T06:28:24.2853826Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:28:24.2855973Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:28:24.283665] 2024-12-18T06:33:09.7189626Z 2024-12-18T06:33:09.7191370Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_e210aabc9f784a01_.log 2024-12-18T06:33:09.7219165Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2024-12-18T06:33:09.7245460Z 2024-12-18T06:33:09.7245912Z Running distributed/fsdp/test_fsdp_core 2/3 ... [2024-12-18 06:33:09.719071] 2024-12-18T06:33:09.7246746Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:33:09.7248649Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:33:09.719686] 2024-12-18T06:41:56.5475504Z 2024-12-18T06:41:56.5477225Z distributed/fsdp/test_fsdp_core 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_2.3_6912787bceb2f2d5_.log 2024-12-18T06:41:56.5492765Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda 2024-12-18T06:41:56.5503865Z 2024-12-18T06:41:56.5504132Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2024-12-18 06:41:56.548064] 2024-12-18T06:41:56.5504645Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:41:56.5506048Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:41:56.548656] 2024-12-18T06:43:32.3368483Z 2024-12-18T06:43:32.3370054Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_9e812b9b1c65c70f_.log 2024-12-18T06:43:32.3386872Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2024-12-18T06:43:32.3399644Z 2024-12-18T06:43:32.3400119Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2024-12-18 06:43:32.337051] 2024-12-18T06:43:32.3415720Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:43:32.3417698Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:43:32.337646] 2024-12-18T06:44:41.9588992Z 2024-12-18T06:44:41.9590690Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_c097b42d5ef4bb96_.log 2024-12-18T06:44:41.9596661Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_backward_reshard_hooks_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_hooks_multi_traversal_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_ddp_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_non_frozen_fsdp_cuda 2024-12-18T06:44:41.9599846Z 2024-12-18T06:44:41.9600433Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2024-12-18 06:44:41.959216] 2024-12-18T06:44:41.9601417Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:44:41.9603528Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:44:41.959818] 2024-12-18T06:45:04.5785905Z 2024-12-18T06:45:04.5793124Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_768b6e2ce47e1439_.log 2024-12-18T06:45:04.5803357Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor 2024-12-18T06:45:04.5812480Z 2024-12-18T06:45:04.5812974Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2024-12-18 06:45:04.578668] 2024-12-18T06:45:04.5813872Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:45:04.5815835Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:45:04.579218] 2024-12-18T06:47:13.0465613Z 2024-12-18T06:47:13.0467464Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_fbd82f94ab2f7054_.log 2024-12-18T06:47:13.0480916Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_submodule, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises 2024-12-18T06:47:13.0496875Z 2024-12-18T06:47:13.0497434Z Running distributed/fsdp/test_fsdp_mixed_precision 1/2 ... [2024-12-18 06:47:13.046738] 2024-12-18T06:47:13.0498425Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:47:13.0500934Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:47:13.047317] 2024-12-18T06:55:25.4635060Z 2024-12-18T06:55:25.4642035Z distributed/fsdp/test_fsdp_mixed_precision 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.2_db7669219cd5d95c_.log 2024-12-18T06:55:25.4685591Z Running 39 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow 2024-12-18T06:55:25.4727091Z 2024-12-18T06:55:25.4727512Z Running distributed/test_c10d_gloo 1/2 ... [2024-12-18 06:55:25.464171] 2024-12-18T06:55:25.4728305Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T06:55:25.4730249Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 06:55:25.464775] 2024-12-18T07:08:33.4143072Z 2024-12-18T07:08:33.4148590Z distributed/test_c10d_gloo 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.2_ef2df86330e37c85_.log 2024-12-18T07:08:33.4191727Z Running 64 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-12-18T07:08:33.4236789Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2024-12-18T07:08:33.4238342Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2024-12-18T07:08:33.4239917Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2024-12-18T07:08:33.4241525Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2024-12-18T07:08:33.4243057Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2024-12-18T07:08:33.4244514Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2024-12-18T07:08:33.4245961Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2024-12-18T07:08:33.4247413Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2024-12-18T07:08:33.4248850Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2024-12-18T07:08:33.4250788Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2024-12-18T07:08:33.4252385Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2024-12-18T07:08:33.4253924Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2024-12-18T07:08:33.4255364Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2024-12-18T07:08:33.4256803Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2024-12-18T07:08:33.4258276Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2024-12-18T07:08:33.4259723Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2024-12-18T07:08:33.4261187Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2024-12-18T07:08:33.4262628Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2024-12-18T07:08:33.4264051Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2024-12-18T07:08:33.4265533Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2024-12-18T07:08:33.4266985Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2024-12-18T07:08:33.4268372Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2024-12-18T07:08:33.4270119Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2024-12-18T07:08:33.4271579Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2024-12-18T07:08:33.4273116Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2024-12-18T07:08:33.4274536Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2024-12-18T07:08:33.4275978Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2024-12-18T07:08:33.4277443Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2024-12-18T07:08:33.4278937Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2024-12-18T07:08:33.4280512Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2024-12-18T07:08:33.4282096Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2024-12-18T07:08:33.4283753Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2024-12-18T07:08:33.4285674Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2024-12-18T07:08:33.4287640Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2024-12-18T07:08:33.4289619Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2024-12-18T07:08:33.4291631Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2024-12-18T07:08:33.4294022Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2024-12-18T07:08:33.4295972Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2024-12-18T07:08:33.4297933Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2024-12-18T07:08:33.4299964Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2024-12-18T07:08:33.4301867Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2024-12-18T07:08:33.4303701Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2024-12-18T07:08:33.4305526Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2024-12-18T07:08:33.4307279Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2024-12-18T07:08:33.4309154Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2024-12-18T07:08:33.4310971Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2024-12-18T07:08:33.4313070Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2024-12-18T07:08:33.4315088Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2024-12-18T07:08:33.4316694Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2024-12-18T07:08:33.4318280Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2024-12-18T07:08:33.4319907Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2024-12-18T07:08:33.4321544Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2024-12-18T07:08:33.4322873Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2024-12-18T07:08:33.4324080Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2024-12-18T07:08:33.4325335Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2024-12-18T07:08:33.4326587Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2024-12-18T07:08:33.4327832Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2024-12-18T07:08:33.4329056Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2024-12-18T07:08:33.4330328Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2024-12-18T07:08:33.4331661Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2024-12-18T07:08:33.4332884Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2024-12-18T07:08:33.4334344Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2024-12-18T07:08:33.4336194Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2024-12-18T07:08:33.4338213Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2024-12-18T07:08:33.4338983Z 2024-12-18T07:08:33.4339306Z Running distributed/test_store 1/1 ... [2024-12-18 07:08:33.416481] 2024-12-18T07:08:33.4339968Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:08:33.4341660Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 07:08:33.417102] 2024-12-18T07:17:31.8809051Z 2024-12-18T07:17:31.8811054Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_35be0f8c48b95906_.log 2024-12-18T07:17:31.8867159Z Running 88 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_repr, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_repr, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param, test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-12-18T07:17:31.8911134Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append 2024-12-18T07:17:31.8912660Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set 2024-12-18T07:17:31.8914113Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file 2024-12-18T07:17:31.8915495Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get 2024-12-18T07:17:31.8916712Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set 2024-12-18T07:17:31.8917924Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount 2024-12-18T07:17:31.8919607Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check 2024-12-18T07:17:31.8920861Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait 2024-12-18T07:17:31.8922060Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append 2024-12-18T07:17:31.8923255Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set 2024-12-18T07:17:31.8924470Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get 2024-12-18T07:17:31.8925652Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set 2024-12-18T07:17:31.8926871Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check 2024-12-18T07:17:31.8928124Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait 2024-12-18T07:17:31.8929446Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store 2024-12-18T07:17:31.8930785Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append 2024-12-18T07:17:31.8932086Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set 2024-12-18T07:17:31.8933409Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get 2024-12-18T07:17:31.8934789Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set 2024-12-18T07:17:31.8936147Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check 2024-12-18T07:17:31.8937520Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait 2024-12-18T07:17:31.8939205Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use 2024-12-18T07:17:31.8940476Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append 2024-12-18T07:17:31.8941690Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set 2024-12-18T07:17:31.8943062Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-12-18T07:17:31.8944416Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get 2024-12-18T07:17:31.8945605Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set 2024-12-18T07:17:31.8946952Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size 2024-12-18T07:17:31.8948475Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-12-18T07:17:31.8949877Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy 2024-12-18T07:17:31.8951160Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys 2024-12-18T07:17:31.8952475Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_repr 2024-12-18T07:17:31.8953672Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check 2024-12-18T07:17:31.8954920Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait 2024-12-18T07:17:31.8956262Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients 2024-12-18T07:17:31.8957681Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket 2024-12-18T07:17:31.8959055Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises 2024-12-18T07:17:31.8960464Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use 2024-12-18T07:17:31.8962162Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append 2024-12-18T07:17:31.8963439Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set 2024-12-18T07:17:31.8964889Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket 2024-12-18T07:17:31.8966325Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get 2024-12-18T07:17:31.8967587Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set 2024-12-18T07:17:31.8969014Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size 2024-12-18T07:17:31.8970641Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size 2024-12-18T07:17:31.8972121Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy 2024-12-18T07:17:31.8973470Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys 2024-12-18T07:17:31.8974753Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_repr 2024-12-18T07:17:31.8976024Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check 2024-12-18T07:17:31.8977349Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait 2024-12-18T07:17:31.8978780Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients 2024-12-18T07:17:31.8980288Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket 2024-12-18T07:17:31.8982031Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises 2024-12-18T07:17:31.8983390Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append 2024-12-18T07:17:31.8984675Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set 2024-12-18T07:17:31.8985993Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get 2024-12-18T07:17:31.8987266Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set 2024-12-18T07:17:31.8988568Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check 2024-12-18T07:17:31.8989894Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait 2024-12-18T07:17:31.8991321Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store 2024-12-18T07:17:31.8993011Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get 2024-12-18T07:17:31.8994273Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler 2024-12-18T07:17:31.8995609Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params 2024-12-18T07:17:31.8996907Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal 2024-12-18T07:17:31.8998192Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors 2024-12-18T07:17:31.8999479Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal 2024-12-18T07:17:31.9000765Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors 2024-12-18T07:17:31.9002073Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout 2024-12-18T07:17:31.9003325Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal 2024-12-18T07:17:31.9004954Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client 2024-12-18T07:17:31.9006297Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set 2024-12-18T07:17:31.9007566Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv 2024-12-18T07:17:31.9008787Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip 2024-12-18T07:17:31.9009999Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks 2024-12-18T07:17:31.9011278Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough 2024-12-18T07:17:31.9012560Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip 2024-12-18T07:17:31.9013794Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip 2024-12-18T07:17:31.9014996Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip 2024-12-18T07:17:31.9016186Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail 2024-12-18T07:17:31.9017412Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store 2024-12-18T07:17:31.9018626Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store 2024-12-18T07:17:31.9019878Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store 2024-12-18T07:17:31.9021427Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store 2024-12-18T07:17:31.9022704Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv 2024-12-18T07:17:31.9024173Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait 2024-12-18T07:17:31.9025379Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var 2024-12-18T07:17:31.9026574Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param 2024-12-18T07:17:31.9027762Z Running 1 items in this shard: test/distributed/test_store.py::TestClientProtocol::test_client_connect 2024-12-18T07:17:31.9028440Z 2024-12-18T07:17:31.9028893Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2024-12-18 07:17:31.882797] 2024-12-18T07:17:31.9029696Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:17:31.9031454Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 07:17:31.883337] 2024-12-18T07:17:44.3799783Z 2024-12-18T07:17:44.3801997Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_e1b41f5eb26a0315_.log 2024-12-18T07:17:44.3814126Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue 2024-12-18T07:17:44.3825022Z 2024-12-18T07:17:44.3825569Z Running distributed/elastic/utils/distributed_test 1/1 ... [2024-12-18 07:17:44.379819] 2024-12-18T07:17:44.3826520Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:17:44.3828739Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 07:17:44.380410] 2024-12-18T07:17:55.6236456Z 2024-12-18T07:17:55.6238346Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_8ed07c6015a18c86_.log 2024-12-18T07:17:55.6247964Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker 2024-12-18T07:17:55.6254428Z 2024-12-18T07:17:55.6254905Z Running distributed/elastic/timer/api_test 1/1 ... [2024-12-18 07:17:55.624047] 2024-12-18T07:17:55.6255784Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:17:55.6257754Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 07:17:55.624609] 2024-12-18T07:17:58.5434212Z 2024-12-18T07:17:58.5436066Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_f4ff2c453abceeaa_.log 2024-12-18T07:17:58.5437567Z 2024-12-18T07:17:59.7203377Z Running test batch 'tests to run' cost 13015.89 seconds 2024-12-18T07:18:00.5627031Z 2024-12-18T07:18:00.5627544Z real 217m2.734s 2024-12-18T07:18:00.5628062Z user 333m18.670s 2024-12-18T07:18:00.5628527Z sys 385m14.679s 2024-12-18T07:18:00.5628958Z + assert_git_not_dirty 2024-12-18T07:18:00.5629532Z + [[ linux-focal-rocm6.2-py3.10 != *rocm* ]] 2024-12-18T07:18:00.5630203Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T07:18:00.5630854Z + [[ linux-focal-rocm6.2-py3.10 == *rocm* ]] 2024-12-18T07:18:00.5631393Z + [[ 1 == 1 ]] 2024-12-18T07:18:00.5631844Z + echo 'Testing distributed C++ tests' 2024-12-18T07:18:00.5632552Z Testing distributed C++ tests 2024-12-18T07:18:00.5636777Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:18:00.5674499Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:18:00.5708164Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:18:00.5709434Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:18:00.5710498Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest 2024-12-18T07:18:00.6846369Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T07:18:00.6848094Z import pkg_resources 2024-12-18T07:18:06.0154336Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T07:18:06.3451498Z Ignoring disabled issues: [''] 2024-12-18T07:18:06.3748062Z Found test times from artifacts 2024-12-18T07:18:06.4376036Z Found test times from artifacts 2024-12-18T07:18:06.4392170Z Running all tests 2024-12-18T07:18:06.4393033Z Running parallel tests on 8 processes 2024-12-18T07:18:06.4393742Z Name: tests to run (est. time: 0.0min) 2024-12-18T07:18:06.4394556Z Serial tests (1): 2024-12-18T07:18:06.4396834Z cpp/FileStoreTest 1/1 2024-12-18T07:18:06.4398023Z Parallel tests (0): 2024-12-18T07:18:06.4398548Z Name: excluded (est. time: 0.0min) 2024-12-18T07:18:06.4399079Z Serial tests (0): 2024-12-18T07:18:06.4399538Z Parallel tests (0): 2024-12-18T07:18:06.4542924Z Running cpp/FileStoreTest 1/1 ... [2024-12-18 07:18:06.453204] 2024-12-18T07:18:06.4543737Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:18:06.4545938Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7a59bec5eaf2699b.xml', '-x', '--reruns=2'] ... [2024-12-18 07:18:06.453774] 2024-12-18T07:18:09.3771788Z 2024-12-18T07:18:09.3772919Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_0d51a3b900acacec_.log 2024-12-18T07:18:09.3773636Z 2024-12-18T07:18:12.6186652Z Running test batch 'tests to run' cost 6.18 seconds 2024-12-18T07:18:13.4360368Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest 2024-12-18T07:18:13.5503337Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T07:18:13.5504987Z import pkg_resources 2024-12-18T07:18:19.1087223Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T07:18:19.1225484Z Found test times from artifacts 2024-12-18T07:18:19.1726743Z Found test times from artifacts 2024-12-18T07:18:19.1742437Z Running all tests 2024-12-18T07:18:19.1746779Z Running parallel tests on 8 processes 2024-12-18T07:18:19.1747558Z Name: tests to run (est. time: 0.0min) 2024-12-18T07:18:19.1748120Z Serial tests (1): 2024-12-18T07:18:19.1748581Z cpp/HashStoreTest 1/1 2024-12-18T07:18:19.1749074Z Parallel tests (0): 2024-12-18T07:18:19.1749551Z Name: excluded (est. time: 0.0min) 2024-12-18T07:18:19.1750126Z Serial tests (0): 2024-12-18T07:18:19.1750570Z Parallel tests (0): 2024-12-18T07:18:19.1884254Z Running cpp/HashStoreTest 1/1 ... [2024-12-18 07:18:19.187567] 2024-12-18T07:18:19.1885067Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:18:19.1888021Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5a62ebbd0ba2837c.xml', '-x', '--reruns=2'] ... [2024-12-18 07:18:19.188141] 2024-12-18T07:18:21.9623383Z 2024-12-18T07:18:21.9624805Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_f12b20f88d15181c_.log 2024-12-18T07:18:21.9625926Z 2024-12-18T07:18:25.5613024Z Running test batch 'tests to run' cost 6.39 seconds 2024-12-18T07:18:26.5609403Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest 2024-12-18T07:18:26.6745232Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T07:18:26.6747069Z import pkg_resources 2024-12-18T07:18:32.0235586Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T07:18:32.0356251Z Found test times from artifacts 2024-12-18T07:18:32.0863486Z Found test times from artifacts 2024-12-18T07:18:32.0877424Z Running all tests 2024-12-18T07:18:32.0882553Z Running parallel tests on 8 processes 2024-12-18T07:18:32.0883258Z Name: tests to run (est. time: 0.0min) 2024-12-18T07:18:32.0883818Z Serial tests (1): 2024-12-18T07:18:32.0884288Z cpp/TCPStoreTest 1/1 2024-12-18T07:18:32.0884784Z Parallel tests (0): 2024-12-18T07:18:32.0885282Z Name: excluded (est. time: 0.0min) 2024-12-18T07:18:32.0885817Z Serial tests (0): 2024-12-18T07:18:32.0887069Z Parallel tests (0): 2024-12-18T07:18:32.1018725Z Running cpp/TCPStoreTest 1/1 ... [2024-12-18 07:18:32.100621] 2024-12-18T07:18:32.1019991Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:18:32.1022291Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-5a3ffa2dc5b0ee1e.xml', '-x', '--reruns=2'] ... [2024-12-18 07:18:32.101118] 2024-12-18T07:18:44.7557917Z 2024-12-18T07:18:44.7559398Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_3cb1010cebed70c4_.log 2024-12-18T07:18:44.7560610Z 2024-12-18T07:18:45.8764933Z Running test batch 'tests to run' cost 13.79 seconds 2024-12-18T07:18:46.6981958Z + echo 'Testing multi-GPU linalg tests' 2024-12-18T07:18:46.6982694Z Testing multi-GPU linalg tests 2024-12-18T07:18:46.6983610Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose 2024-12-18T07:18:46.8113475Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T07:18:46.8115040Z import pkg_resources 2024-12-18T07:18:52.1794971Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T07:18:52.1916565Z Found test times from artifacts 2024-12-18T07:18:52.2426788Z Found test times from artifacts 2024-12-18T07:18:52.2440410Z Running all tests 2024-12-18T07:18:52.2443672Z Running parallel tests on 8 processes 2024-12-18T07:18:52.2444361Z Name: tests to run (est. time: 0.0min) 2024-12-18T07:18:52.2444917Z Serial tests (1): 2024-12-18T07:18:52.2445354Z test_linalg 1/1 2024-12-18T07:18:52.2445803Z Parallel tests (0): 2024-12-18T07:18:52.2446289Z Name: excluded (est. time: 0.0min) 2024-12-18T07:18:52.2446837Z Serial tests (0): 2024-12-18T07:18:52.2447268Z Parallel tests (0): 2024-12-18T07:18:52.2580372Z Running test_linalg 1/1 ... [2024-12-18 07:18:52.257304] 2024-12-18T07:18:52.2582121Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:18:52.2584145Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2024-12-18 07:18:52.257703] 2024-12-18T07:18:58.7887763Z 2024-12-18T07:18:58.7889132Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_0d24e47dc3198b91_.log 2024-12-18T07:18:58.7891021Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32 2024-12-18T07:18:58.7891942Z 2024-12-18T07:18:59.8327174Z Running test batch 'tests to run' cost 7.59 seconds 2024-12-18T07:19:00.6767371Z + [[ linux-focal-rocm6.2-py3.10 == *cuda* ]] 2024-12-18T07:19:00.6768045Z + [[ 1 == 1 ]] 2024-12-18T07:19:00.6768544Z + test_rpc 2024-12-18T07:19:00.6768972Z + echo 'Testing RPC C++ tests' 2024-12-18T07:19:00.6769520Z Testing RPC C++ tests 2024-12-18T07:19:00.6773723Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:19:00.6817700Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:19:00.6855433Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-12-18T07:19:00.6856625Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc 2024-12-18T07:19:00.7993739Z /var/lib/jenkins/pytorch/test/run_test.py:22: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html 2024-12-18T07:19:00.7995282Z import pkg_resources 2024-12-18T07:19:06.1464390Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json?versionId=PhiMB7EP3187qvpKvnORewoK3InOIvX5 to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2024-12-18T07:19:06.1584091Z Found test times from artifacts 2024-12-18T07:19:06.2091413Z Found test times from artifacts 2024-12-18T07:19:06.2105487Z Running all tests 2024-12-18T07:19:06.2109975Z Running parallel tests on 8 processes 2024-12-18T07:19:06.2110647Z Name: tests to run (est. time: 0.0min) 2024-12-18T07:19:06.2111217Z Serial tests (1): 2024-12-18T07:19:06.2111681Z cpp/test_cpp_rpc 1/1 2024-12-18T07:19:06.2112231Z Parallel tests (0): 2024-12-18T07:19:06.2112901Z Name: excluded (est. time: 0.0min) 2024-12-18T07:19:06.2113434Z Serial tests (0): 2024-12-18T07:19:06.2113897Z Parallel tests (0): 2024-12-18T07:19:06.2245919Z Running cpp/test_cpp_rpc 1/1 ... [2024-12-18 07:19:06.223836] 2024-12-18T07:19:06.2246701Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2024-12-18T07:19:06.2249567Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-7afef4a8357f48c9.xml', '-x', '--reruns=2'] ... [2024-12-18 07:19:06.224446] 2024-12-18T07:19:13.9125357Z 2024-12-18T07:19:13.9127003Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_428378948de0d9aa_.log 2024-12-18T07:19:13.9128203Z 2024-12-18T07:19:15.0398049Z Running test batch 'tests to run' cost 8.83 seconds 2024-12-18T07:19:15.9191071Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-12-18T07:19:15.9193022Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2024-12-18T07:19:15.9194862Z docker exec -t "32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2024-12-18T07:19:15.9267410Z shell: /usr/bin/bash -e {0} 2024-12-18T07:19:15.9267941Z env: 2024-12-18T07:19:15.9268351Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:15.9268959Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:15.9270001Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:15.9270986Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:15.9271520Z AWS_REGION: us-east-1 2024-12-18T07:19:15.9272117Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:15.9273048Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:15.9284300Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:15.9285253Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:15.9286168Z ##[endgroup] 2024-12-18T07:19:16.0383152Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-12-18T07:19:16.1273601Z ##[group]Run cat test/**/*_toprint.log || true 2024-12-18T07:19:16.1274485Z cat test/**/*_toprint.log || true 2024-12-18T07:19:16.1347243Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T07:19:16.1347962Z env: 2024-12-18T07:19:16.1348387Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:16.1348993Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:16.1350082Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:16.1351033Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:16.1351561Z AWS_REGION: us-east-1 2024-12-18T07:19:16.1352156Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:16.1352995Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:16.1363661Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:16.1364472Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:16.1365338Z ##[endgroup] 2024-12-18T07:19:16.1628020Z cat: 'test/**/*_toprint.log': No such file or directory 2024-12-18T07:19:16.1867964Z Prepare all required actions 2024-12-18T07:19:16.1868716Z Getting action download info 2024-12-18T07:19:16.9726051Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-12-18T07:19:17.6486695Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-12-18T07:19:17.6486974Z with: 2024-12-18T07:19:17.6487146Z use-gha: true 2024-12-18T07:19:17.6487404Z file-suffix: test-distributed-1-3-linux.rocm.gpu_34566686857 2024-12-18T07:19:17.6487709Z s3-bucket: gha-artifacts 2024-12-18T07:19:17.6487909Z env: 2024-12-18T07:19:17.6488081Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:17.6488360Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:17.6488779Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:17.6489165Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:17.6489381Z AWS_REGION: us-east-1 2024-12-18T07:19:17.6489688Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:17.6489973Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:17.6494185Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:17.6494524Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:17.6494873Z ##[endgroup] 2024-12-18T07:19:17.6581792Z ##[group]Run actions/upload-artifact@v4 2024-12-18T07:19:17.6582196Z with: 2024-12-18T07:19:17.6582621Z name: test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip 2024-12-18T07:19:17.6583130Z retention-days: 14 2024-12-18T07:19:17.6583456Z if-no-files-found: warn 2024-12-18T07:19:17.6583753Z path: test/**/*.json 2024-12-18T07:19:17.6584392Z compression-level: 6 2024-12-18T07:19:17.6584673Z overwrite: false 2024-12-18T07:19:17.6584944Z include-hidden-files: false 2024-12-18T07:19:17.6585428Z env: 2024-12-18T07:19:17.6585685Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:17.6586058Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:17.6586610Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:17.6587086Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:17.6587469Z AWS_REGION: us-east-1 2024-12-18T07:19:17.6587832Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:17.6588274Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:17.6592758Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:17.6593176Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:17.6593813Z ##[endgroup] 2024-12-18T07:19:18.3606143Z With the provided path, there will be 11 files uploaded 2024-12-18T07:19:18.3612363Z Artifact name is valid! 2024-12-18T07:19:18.3613031Z Root directory input is valid! 2024-12-18T07:19:18.5402747Z Beginning upload of artifact content to blob storage 2024-12-18T07:19:18.7875984Z Uploaded bytes 45938 2024-12-18T07:19:18.8337217Z Finished uploading artifact content to blob storage! 2024-12-18T07:19:18.8342910Z SHA256 hash of uploaded artifact zip is 4c15c035b456f0015b8b955982b11fb9a240eeca51baea1fa95762a8a9135176 2024-12-18T07:19:18.8345637Z Finalizing artifact upload 2024-12-18T07:19:18.9294895Z Artifact test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip.zip successfully finalized. Artifact ID 2336126153 2024-12-18T07:19:18.9297230Z Artifact test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip has been successfully uploaded! Final size is 45938 bytes. Artifact ID is 2336126153 2024-12-18T07:19:18.9313223Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336126153 2024-12-18T07:19:18.9703637Z ##[group]Run actions/upload-artifact@v4 2024-12-18T07:19:18.9704206Z with: 2024-12-18T07:19:18.9704903Z name: test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip 2024-12-18T07:19:18.9705722Z retention-days: 14 2024-12-18T07:19:18.9706142Z if-no-files-found: ignore 2024-12-18T07:19:18.9706603Z path: test/**/*.xml test/**/*.csv 2024-12-18T07:19:18.9707555Z compression-level: 6 2024-12-18T07:19:18.9708002Z overwrite: false 2024-12-18T07:19:18.9708427Z include-hidden-files: false 2024-12-18T07:19:18.9708882Z env: 2024-12-18T07:19:18.9709233Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:18.9709748Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:18.9710768Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:18.9711600Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:18.9712048Z AWS_REGION: us-east-1 2024-12-18T07:19:18.9712665Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:18.9713280Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:18.9722794Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:18.9723492Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:18.9724232Z ##[endgroup] 2024-12-18T07:19:19.7519627Z With the provided path, there will be 651 files uploaded 2024-12-18T07:19:19.7524370Z Artifact name is valid! 2024-12-18T07:19:19.7525176Z Root directory input is valid! 2024-12-18T07:19:19.9049535Z Beginning upload of artifact content to blob storage 2024-12-18T07:19:20.6026652Z Uploaded bytes 431178 2024-12-18T07:19:20.6466223Z Finished uploading artifact content to blob storage! 2024-12-18T07:19:20.6471980Z SHA256 hash of uploaded artifact zip is 4c3aeb2b4f3da3f2755a449686d1795c172e64ee1af1999740dd60471f800ab8 2024-12-18T07:19:20.6475228Z Finalizing artifact upload 2024-12-18T07:19:20.7393226Z Artifact test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip.zip successfully finalized. Artifact ID 2336126236 2024-12-18T07:19:20.7395667Z Artifact test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip has been successfully uploaded! Final size is 431178 bytes. Artifact ID is 2336126236 2024-12-18T07:19:20.7406928Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336126236 2024-12-18T07:19:20.7851665Z ##[group]Run actions/upload-artifact@v4 2024-12-18T07:19:20.7852327Z with: 2024-12-18T07:19:20.7853085Z name: logs-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip 2024-12-18T07:19:20.7854115Z retention-days: 14 2024-12-18T07:19:20.7854712Z if-no-files-found: ignore 2024-12-18T07:19:20.7855320Z path: usage_log.txt test/**/*.log 2024-12-18T07:19:20.7855913Z compression-level: 6 2024-12-18T07:19:20.7856408Z overwrite: false 2024-12-18T07:19:20.7856928Z include-hidden-files: false 2024-12-18T07:19:20.7857459Z env: 2024-12-18T07:19:20.7857894Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:20.7858508Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:20.7859579Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:20.7860612Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:20.7861169Z AWS_REGION: us-east-1 2024-12-18T07:19:20.7861796Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:20.7862593Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:20.7873927Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:20.7874798Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:20.7875720Z ##[endgroup] 2024-12-18T07:19:21.5219749Z Multiple search paths detected. Calculating the least common ancestor of all paths 2024-12-18T07:19:21.5222104Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2024-12-18T07:19:21.5223573Z With the provided path, there will be 84 files uploaded 2024-12-18T07:19:21.5228009Z Artifact name is valid! 2024-12-18T07:19:21.5228639Z Root directory input is valid! 2024-12-18T07:19:21.6803116Z Beginning upload of artifact content to blob storage 2024-12-18T07:19:22.0516616Z Uploaded bytes 171789 2024-12-18T07:19:22.0994130Z Finished uploading artifact content to blob storage! 2024-12-18T07:19:22.1000065Z SHA256 hash of uploaded artifact zip is bfe38fa613ab7f5c7fa966e84d7a08ba560485e337498ad21188dc57e0cb6319 2024-12-18T07:19:22.1003675Z Finalizing artifact upload 2024-12-18T07:19:22.1926078Z Artifact logs-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip.zip successfully finalized. Artifact ID 2336126315 2024-12-18T07:19:22.1928222Z Artifact logs-runattempt1-test-distributed-1-3-linux.rocm.gpu_34566686857.zip has been successfully uploaded! Final size is 171789 bytes. Artifact ID is 2336126315 2024-12-18T07:19:22.1940445Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/12383255654/artifacts/2336126315 2024-12-18T07:19:22.2320473Z ##[group]Run # shellcheck disable=SC2156 2024-12-18T07:19:22.2321241Z # shellcheck disable=SC2156 2024-12-18T07:19:22.2322389Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-12-18T07:19:22.2397990Z shell: /usr/bin/bash -e {0} 2024-12-18T07:19:22.2398552Z env: 2024-12-18T07:19:22.2398992Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:22.2399628Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:22.2400729Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:22.2401731Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:22.2402302Z AWS_REGION: us-east-1 2024-12-18T07:19:22.2402924Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:22.2403658Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:22.2414627Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:22.2415540Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:22.2416452Z ##[endgroup] 2024-12-18T07:19:22.5635353Z Prepare all required actions 2024-12-18T07:19:22.5636069Z Getting action download info 2024-12-18T07:19:22.5685420Z ##[group]Run ./.github/actions/teardown-rocm 2024-12-18T07:19:22.5685938Z env: 2024-12-18T07:19:22.5686301Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:22.5686820Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:22.5687729Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:22.5688558Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:22.5689038Z AWS_REGION: us-east-1 2024-12-18T07:19:22.5689632Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:22.5690339Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:22.5699790Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:22.5700495Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:22.5701255Z ##[endgroup] 2024-12-18T07:19:22.5728658Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T07:19:22.5729606Z # ignore expansion of "docker ps -q" since it could be empty 2024-12-18T07:19:22.5730425Z # shellcheck disable=SC2046 2024-12-18T07:19:22.5731077Z docker stop $(docker ps -q) || true 2024-12-18T07:19:22.5731744Z # Prune all stopped containers. 2024-12-18T07:19:22.5732369Z docker container prune -f 2024-12-18T07:19:22.5786858Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T07:19:22.5787486Z env: 2024-12-18T07:19:22.5787859Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:22.5788391Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:22.5789321Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:22.5790165Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:22.5790628Z AWS_REGION: us-east-1 2024-12-18T07:19:22.5791116Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:22.5791724Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:22.5800926Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:22.5801627Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:22.5802392Z ##[endgroup] 2024-12-18T07:19:23.4068867Z 32ffe0408e4f 2024-12-18T07:19:29.2653690Z Deleted Containers: 2024-12-18T07:19:29.2654665Z 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:29.2655394Z 2024-12-18T07:19:29.2655651Z Total reclaimed space: 7.135GB 2024-12-18T07:19:29.2744672Z Prepare all required actions 2024-12-18T07:19:29.2801853Z ##[group]Run ./.github/actions/diskspace-cleanup 2024-12-18T07:19:29.2802457Z with: 2024-12-18T07:19:29.2802880Z diskspace-cutoff: 70 2024-12-18T07:19:29.2803355Z env: 2024-12-18T07:19:29.2803820Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:29.2804504Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:29.2805703Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:29.2806807Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:29.2807321Z AWS_REGION: us-east-1 2024-12-18T07:19:29.2807914Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:29.2808586Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:29.2819402Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:29.2820186Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:29.2821028Z ##[endgroup] 2024-12-18T07:19:29.2851905Z ##[group]Run set -ex 2024-12-18T07:19:29.2852525Z set -ex 2024-12-18T07:19:29.2852973Z diskspace_cutoff=70 2024-12-18T07:19:29.2853648Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2024-12-18T07:19:29.2854717Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T07:19:29.2856334Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2024-12-18T07:19:29.2858289Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2024-12-18T07:19:29.2859145Z  docker system prune -af 2024-12-18T07:19:29.2860681Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2024-12-18T07:19:29.2861694Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2024-12-18T07:19:29.2862760Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2024-12-18T07:19:29.2863706Z  echo "$msg" 2024-12-18T07:19:29.2864189Z  exit 1 2024-12-18T07:19:29.2864636Z  else 2024-12-18T07:19:29.2865147Z  difference=$((diskspace - diskspace_new)) 2024-12-18T07:19:29.2865887Z  echo "Diskspace saved: $difference percent" 2024-12-18T07:19:29.2866504Z  fi 2024-12-18T07:19:29.2866911Z fi 2024-12-18T07:19:29.2939065Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-12-18T07:19:29.2939765Z env: 2024-12-18T07:19:29.2940178Z GIT_DEFAULT_BRANCH: main 2024-12-18T07:19:29.2940788Z DOCKER_HOST: unix:///run/user/1307800118/docker.sock 2024-12-18T07:19:29.2941812Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device=/dev/dri --group-add video --group-add daemon 2024-12-18T07:19:29.2942741Z AWS_DEFAULT_REGION: us-east-1 2024-12-18T07:19:29.2943267Z AWS_REGION: us-east-1 2024-12-18T07:19:29.2943922Z AWS_ACCESS_KEY_ID: *** 2024-12-18T07:19:29.2944732Z AWS_SECRET_ACCESS_KEY: *** 2024-12-18T07:19:29.2955426Z AWS_SESSION_TOKEN: *** 2024-12-18T07:19:29.2956252Z CONTAINER_NAME: 32ffe0408e4f6f100b98c163995f2e2d69ee2eccbdc33fee5bf21e67ae11b957 2024-12-18T07:19:29.2957099Z ##[endgroup] 2024-12-18T07:19:29.3066775Z + diskspace_cutoff=70 2024-12-18T07:19:29.3077466Z ++ docker info -f '{{.DockerRootDir}}' 2024-12-18T07:19:29.4086564Z + docker_root_dir=/media/4TB/docker-rootless 2024-12-18T07:19:29.4102375Z ++ df -H --output=pcent /media/4TB/docker-rootless 2024-12-18T07:19:29.4105777Z ++ sed -n 2p 2024-12-18T07:19:29.4108844Z ++ sed s/%// 2024-12-18T07:19:29.4112050Z ++ sed 's/ //' 2024-12-18T07:19:29.4154081Z + diskspace=56 2024-12-18T07:19:29.4155716Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2024-12-18T07:19:29.4157000Z + [[ 56 -ge 70 ]] 2024-12-18T07:19:29.4248163Z Post job cleanup. 2024-12-18T07:19:29.4317405Z Post job cleanup. 2024-12-18T07:19:29.4946676Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-12-18T07:19:29.5624729Z Post job cleanup. 2024-12-18T07:19:29.7187345Z Post job cleanup. 2024-12-18T07:19:29.7274844Z Post job cleanup. 2024-12-18T07:19:29.7344156Z Post job cleanup. 2024-12-18T07:19:29.8341364Z [command]/usr/bin/git version 2024-12-18T07:19:29.8393341Z git version 2.34.1 2024-12-18T07:19:29.8436256Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/f73ed9b8-ab00-45fc-b6c3-fa21c58544f7' before making global git config changes 2024-12-18T07:19:29.8438421Z Adding repository directory to the temporary git global config as a safe directory 2024-12-18T07:19:29.8439965Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2024-12-18T07:19:29.8510854Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-12-18T07:19:29.8573315Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-12-18T07:19:29.9253839Z Entering 'android/libs/fbjni' 2024-12-18T07:19:29.9380777Z Entering 'third_party/FP16' 2024-12-18T07:19:29.9504583Z Entering 'third_party/FXdiv' 2024-12-18T07:19:29.9625226Z Entering 'third_party/NNPACK' 2024-12-18T07:19:29.9747122Z Entering 'third_party/NVTX' 2024-12-18T07:19:29.9870720Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T07:19:29.9995106Z Entering 'third_party/XNNPACK' 2024-12-18T07:19:30.0150150Z Entering 'third_party/benchmark' 2024-12-18T07:19:30.0270508Z Entering 'third_party/composable_kernel' 2024-12-18T07:19:30.0404170Z Entering 'third_party/cpp-httplib' 2024-12-18T07:19:30.0521103Z Entering 'third_party/cpuinfo' 2024-12-18T07:19:30.0639921Z Entering 'third_party/cudnn_frontend' 2024-12-18T07:19:30.0757216Z Entering 'third_party/cutlass' 2024-12-18T07:19:30.0890511Z Entering 'third_party/eigen' 2024-12-18T07:19:30.1010393Z Entering 'third_party/fbgemm' 2024-12-18T07:19:30.1124820Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T07:19:30.1235338Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T07:19:30.1347861Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T07:19:30.1473068Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T07:19:30.1583991Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T07:19:30.1701139Z Entering 'third_party/flatbuffers' 2024-12-18T07:19:30.1823538Z Entering 'third_party/fmt' 2024-12-18T07:19:30.1940930Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T07:19:30.2058189Z Entering 'third_party/gloo' 2024-12-18T07:19:30.2180179Z Entering 'third_party/googletest' 2024-12-18T07:19:30.2299900Z Entering 'third_party/ideep' 2024-12-18T07:19:30.2415520Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T07:19:30.2548004Z Entering 'third_party/ittapi' 2024-12-18T07:19:30.2663985Z Entering 'third_party/kineto' 2024-12-18T07:19:30.2778544Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T07:19:30.2887096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T07:19:30.3003338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T07:19:30.3115580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T07:19:30.3229865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T07:19:30.3340288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T07:19:30.3459774Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T07:19:30.3574530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T07:19:30.3689619Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T07:19:30.3803413Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T07:19:30.3924692Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T07:19:30.4036033Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T07:19:30.4157781Z Entering 'third_party/mimalloc' 2024-12-18T07:19:30.4275573Z Entering 'third_party/nccl/nccl' 2024-12-18T07:19:30.4395183Z Entering 'third_party/nlohmann' 2024-12-18T07:19:30.4516681Z Entering 'third_party/onnx' 2024-12-18T07:19:30.4663336Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T07:19:30.4788127Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T07:19:30.4905352Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T07:19:30.5016501Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T07:19:30.5126359Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T07:19:30.5237939Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T07:19:30.5353540Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T07:19:30.5461945Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T07:19:30.5573593Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T07:19:30.5679080Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T07:19:30.5794604Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T07:19:30.5912652Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T07:19:30.6072777Z Entering 'third_party/pocketfft' 2024-12-18T07:19:30.6191093Z Entering 'third_party/protobuf' 2024-12-18T07:19:30.6311113Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T07:19:30.6422163Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T07:19:30.6541698Z Entering 'third_party/psimd' 2024-12-18T07:19:30.6658223Z Entering 'third_party/pthreadpool' 2024-12-18T07:19:30.6774215Z Entering 'third_party/pybind11' 2024-12-18T07:19:30.6892295Z Entering 'third_party/python-peachpy' 2024-12-18T07:19:30.7012158Z Entering 'third_party/sleef' 2024-12-18T07:19:30.7132459Z Entering 'third_party/tensorpipe' 2024-12-18T07:19:30.7247687Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T07:19:30.7358748Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T07:19:30.7465834Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T07:19:30.7578486Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T07:19:30.7686650Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T07:19:30.7851154Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-12-18T07:19:30.7904356Z http.https://github.com/.extraheader 2024-12-18T07:19:30.7927519Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-12-18T07:19:30.8003908Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-12-18T07:19:30.8677959Z Entering 'android/libs/fbjni' 2024-12-18T07:19:30.8746014Z http.https://github.com/.extraheader 2024-12-18T07:19:30.8840074Z Entering 'third_party/FP16' 2024-12-18T07:19:30.8904101Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9000821Z Entering 'third_party/FXdiv' 2024-12-18T07:19:30.9067915Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9175053Z Entering 'third_party/NNPACK' 2024-12-18T07:19:30.9241095Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9330002Z Entering 'third_party/NVTX' 2024-12-18T07:19:30.9392782Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9491718Z Entering 'third_party/VulkanMemoryAllocator' 2024-12-18T07:19:30.9556146Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9645102Z Entering 'third_party/XNNPACK' 2024-12-18T07:19:30.9707070Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9839809Z Entering 'third_party/benchmark' 2024-12-18T07:19:30.9903663Z http.https://github.com/.extraheader 2024-12-18T07:19:30.9999335Z Entering 'third_party/composable_kernel' 2024-12-18T07:19:31.0067887Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0174304Z Entering 'third_party/cpp-httplib' 2024-12-18T07:19:31.0240395Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0334412Z Entering 'third_party/cpuinfo' 2024-12-18T07:19:31.0397330Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0489900Z Entering 'third_party/cudnn_frontend' 2024-12-18T07:19:31.0559229Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0647059Z Entering 'third_party/cutlass' 2024-12-18T07:19:31.0712034Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0821873Z Entering 'third_party/eigen' 2024-12-18T07:19:31.0887672Z http.https://github.com/.extraheader 2024-12-18T07:19:31.0985203Z Entering 'third_party/fbgemm' 2024-12-18T07:19:31.1051581Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1145026Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-12-18T07:19:31.1205224Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1298313Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-12-18T07:19:31.1358771Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1452172Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-12-18T07:19:31.1512210Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1617873Z Entering 'third_party/fbgemm/third_party/googletest' 2024-12-18T07:19:31.1679324Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1769252Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-12-18T07:19:31.1829314Z http.https://github.com/.extraheader 2024-12-18T07:19:31.1922640Z Entering 'third_party/flatbuffers' 2024-12-18T07:19:31.1988114Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2084545Z Entering 'third_party/fmt' 2024-12-18T07:19:31.2149817Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2243734Z Entering 'third_party/gemmlowp/gemmlowp' 2024-12-18T07:19:31.2307855Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2399413Z Entering 'third_party/gloo' 2024-12-18T07:19:31.2462317Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2556707Z Entering 'third_party/googletest' 2024-12-18T07:19:31.2620181Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2713169Z Entering 'third_party/ideep' 2024-12-18T07:19:31.2778857Z http.https://github.com/.extraheader 2024-12-18T07:19:31.2867720Z Entering 'third_party/ideep/mkl-dnn' 2024-12-18T07:19:31.2929519Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3041349Z Entering 'third_party/ittapi' 2024-12-18T07:19:31.3107026Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3201263Z Entering 'third_party/kineto' 2024-12-18T07:19:31.3266945Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3358819Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-12-18T07:19:31.3420315Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3507343Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-12-18T07:19:31.3569045Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3665588Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-12-18T07:19:31.3727661Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3820237Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-12-18T07:19:31.3880864Z http.https://github.com/.extraheader 2024-12-18T07:19:31.3973441Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-12-18T07:19:31.4034499Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4121323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-12-18T07:19:31.4184457Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4284983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-12-18T07:19:31.4345268Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4440392Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-12-18T07:19:31.4501902Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4594218Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-12-18T07:19:31.4654445Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4749584Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-12-18T07:19:31.4811951Z http.https://github.com/.extraheader 2024-12-18T07:19:31.4915082Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-12-18T07:19:31.4975063Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5063489Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-12-18T07:19:31.5126136Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5227267Z Entering 'third_party/mimalloc' 2024-12-18T07:19:31.5292390Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5386645Z Entering 'third_party/nccl/nccl' 2024-12-18T07:19:31.5452535Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5546602Z Entering 'third_party/nlohmann' 2024-12-18T07:19:31.5609328Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5704549Z Entering 'third_party/onnx' 2024-12-18T07:19:31.5772312Z http.https://github.com/.extraheader 2024-12-18T07:19:31.5897671Z Entering 'third_party/onnx/third_party/pybind11' 2024-12-18T07:19:31.5962202Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6066075Z Entering 'third_party/opentelemetry-cpp' 2024-12-18T07:19:31.6134032Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6229194Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-12-18T07:19:31.6291754Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6384319Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-12-18T07:19:31.6445224Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6538554Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-12-18T07:19:31.6600360Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6689731Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-12-18T07:19:31.6749541Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6844692Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-12-18T07:19:31.6904135Z http.https://github.com/.extraheader 2024-12-18T07:19:31.6995461Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-12-18T07:19:31.7057387Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7150431Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-12-18T07:19:31.7210738Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7297741Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-12-18T07:19:31.7361965Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7459851Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-12-18T07:19:31.7522513Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7623249Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-12-18T07:19:31.7685810Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7829613Z Entering 'third_party/pocketfft' 2024-12-18T07:19:31.7893797Z http.https://github.com/.extraheader 2024-12-18T07:19:31.7990143Z Entering 'third_party/protobuf' 2024-12-18T07:19:31.8053070Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8148746Z Entering 'third_party/protobuf/third_party/benchmark' 2024-12-18T07:19:31.8212909Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8305743Z Entering 'third_party/protobuf/third_party/googletest' 2024-12-18T07:19:31.8367908Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8470619Z Entering 'third_party/psimd' 2024-12-18T07:19:31.8537154Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8636147Z Entering 'third_party/pthreadpool' 2024-12-18T07:19:31.8701420Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8797459Z Entering 'third_party/pybind11' 2024-12-18T07:19:31.8861048Z http.https://github.com/.extraheader 2024-12-18T07:19:31.8956373Z Entering 'third_party/python-peachpy' 2024-12-18T07:19:31.9022080Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9117304Z Entering 'third_party/sleef' 2024-12-18T07:19:31.9180967Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9274957Z Entering 'third_party/tensorpipe' 2024-12-18T07:19:31.9340446Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9432448Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-12-18T07:19:31.9492721Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9585726Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-12-18T07:19:31.9644844Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9733985Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-12-18T07:19:31.9797889Z http.https://github.com/.extraheader 2024-12-18T07:19:31.9887657Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-12-18T07:19:31.9947669Z http.https://github.com/.extraheader 2024-12-18T07:19:32.0032469Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-12-18T07:19:32.0097307Z http.https://github.com/.extraheader 2024-12-18T07:19:32.0511949Z Cleaning up orphan processes